Doing what you are asking about is done by many, many of the guys who regularly haunt this forum. I have never heard of accuracy problems from doing it. Checking headspace seems to be what is the only thing done to the barrel but rezero may be needed. I do not recall hearing any discution about loss of accuracy by swapping barrels.

El Lobo