Dave is right, I adjust my loads for the change in the reflection caused by the added length of the muzzle brake.
The added length of the muzzle brake changes the harmonics of the barrel, even if the bullet is not touching the muzzle brake as it exits.
That change in harmonics could, and probably would, change the accuracy if you use identical ammo or the same load in both.
I should have explained that.

The muzzle brake itself wouldn't change the potential accuracy of the barrel, but unless the velocity is adjusted to change the exit time and match the harmonics with the muzzle brake, you probably would change the harmonics of the barrel-muzzle brake combination and would change the accuracy.
You might also have seen better accuracy with the muzzle brake if the ammo had been a closer match the harmonics of the barrel + brake than to the barrel alone.

Charlie b is also correct when he points out the sensitivity of individual barrels to bullet weight.
Of my two .308s, one shoots better with match bullets from 155 and 168 grains and the other (with the muzzle-brake) shoots more accurately with match bullets from 175 to 195 grains.
One of my three 6.5mm CMs (with a 24-inch barrel) shoots more accurately with 130 to 140 grain bullets than the other two (with 26-inch barrels) that shoot best with 142 to 147 grain bullets.
All my loads are tuned to the harmonics of each individual barrel, bullet, and caliber, depending upon whether the powder is temperature sensitive or not, also on the temperature that I will be shooting at, usually within about 2 deg F.
I admit I am anal about accuracy, even if I am probably too old to expect much improvement.

Remember, factory ammo is usually loaded close to Pmax - IMO, people buy ammo based on velocity over accuracy - which no manufacturer could hope to match to every barrel anyway.
If your barrel shoots any factory ammo accurately, you probably have a good barrel. All you need to do is find out which bullet and velocity fits your barrel best, with or without a muzzle brake.
All of my rifles shoot my handloads better than factory ammo - usually by at least 0.1 inches at 100 yards with rifles that average 0.4 inches or smaller groups with my hand loads.
My 10 best loads with each rifle usually average 0.05 to 0.1 inches better than the overall average, simply because every barrel has preferences for powders, bullet shapes and bullet weights.