I've read your post a couple of times and was trying to match your experience with FGMM 168 factory ammo with mine.
You obviously have measured your chamber to conclude where the jump for FGMM is.
All of the 168 FGMM SMK ammo that I have measured have an O.A.L. about 0.005 shorter than SAAMI - around 2.795 O.A.L.
I presume Federal does that so bolts will close in chambers that are precisely cut to SAAMI recommended O.A.L.

If your .308 is reading 0.005 off the rifling with FGMM ammo, your chamber should be right at SAAMI, so seating the same bullet back 0.015 off the rifling would get you an O.A.L. of 2.785.

I have only had one Savage barrel - a .308 rebarreled by the factory - that came with a chamber shorter than SAAMI by 0.006. The bolt would not close on FGMM 168s consistently without seating them back another 0.005. That lasted about 200 rounds before the bore eroded a bit.
Most of my Savage barrels have come from the factory with chambers at least 0.020 longer than SAAMI for their calibers.
Most of my Savages like to be at least 0.015 to 0.020 off the rifling. Some like being further off than that.

If you are loading with a O.A.L. shorter than SAAMI with 168 SMKs and hence shorter than FGMM, then I can't imagine why the fps from your reloads are consistently lower than FGMM ammo @ 2650 fps with a 24-inch barrel, especially since you are using IMR4064.
If your barrel is shorter than 24-in, the velocity of both the FGMM and your hand loads will be slower but relatively by the same degree.
FGMM has used IMR4064 powder for years - after they changed from RL-15 maybe 10 years ago.
I heard recently that they changed powders again but I haven't been able to validate that claim. If so, the powder they now use might be different and therefore your results might also be.

Seating the bullet deeper in the case increases the pressure and adds muzzle velocity - 0.010 deeper would increase mv by about 4-5 fps depending on the powder.
Seating the bullet out further decreases the pressure and drops the muzzle velocity - seating 0.010 further out would decrease mv by about 4-5 fps depending on the powder.

All that being said, if you are getting the accuracy you describe, don't sweat it.
You understand your rifle and use that understanding to tailor your loads to improve the accuracy.
If it means adding a few tenths of a grain to get a better result, do it unless you are seeing pressure signs.
However, I have never found the best accuracy at factory ammo velocities.
Velocity sells factory ammo, not accuracy although I think FGMM is the gold standard for .308 factory ammo.
All my .308s shoot much more accurately with the same bullets as factory ammos with my reloads than with factory ammo.
My best loads are all lower than factory claimed velocity by about 50 to 100 fps depending upon the rifle.