That's what I was thinking, barrel warming up and causing vertical drift. If it was an M14 I would diagnose an over torqued barrel or too tight a gas cylinder lock caused by bubba using shims.

For a Savage, it could very well be receiver screw torquing into the stock may need a bit of tweaking.