Well to be honest, I never understood the whole lightweight rifle scene; My 20# rifle got hauled all over the sawtooth mountains.

That aside, Anytime you remove metal, you will affect the strength, and perhaps more importantly; the heat treatment.
Will it be enough rigidity to cause the decline of western civilization? Probably not, but I don't think I would be shooting nuclear loads either.
The hardening or Toughness loss, would cause me more concern than anything else.