If you are worried about it affecting accuracy there may be a fix that doen't affect it.

Back the barrel out a few more thousandths until it loads smoothly then when you load set the bullet seating die to make the rounds have a longer OAL by the same amount you changed in the barrel. This should give you a easier loading rifle while maintaing the accuracy you have now.

I know quite a few smiths who use a fired case with a piece of tape or two on the back to set headspace, I would back the barrel off and set the headspace using this method. Then measure the case with and without the tape on it. The difference should be what you add to the OAL of the loaded bullet to maintain your accuracy. It shouldn't be much and I seriously doubt the accuracy will be affected enough by the difference and I would try your loads now before changing anything.

Dolomite