Great advice guys. Thank you. BDK, I did actually whack the foam with a hammer a few times, and it dented in the middle, more of a compression really. It didn't shatter, crack or have any signs of structural damage, other than being compressed.
I had completely forgotten about the recoil pad area, until JWW123 said to add some dowels.
Still in the planning stages, but I think I like where it's going, and it seems do-able.