Velocity swings of 100 fps can have an effect on accuracy at longer ranges. With 1/4" in groups at 100 yards including the 100 fps velocity swings, the observable effect is on drift and drop at longer ranges.

The .204 is not likely a 500 yard, much less a 1,000 yard, rifle. So, crank up your favorite ballistics calculator and find the difference in drop at your nominal "long range" for 50 fps above and 50 fps below your nominal velocity.

With .25 in groups at 100 yards, you might be able to see the difference at 300 yards, which might take you out of "minute of mouse" but there is a high likelihood you will still be within "minute of prairie rat."