wtex,

Those averages were with mostly match bullets in hand loads.
I only shoot factory ammo when I am breaking in the rifle to get an idea of what weights the barrel likes.
I am a nut for accuracy and quickly migrate to hand loading because I get the best accuracy with my hand loads.

With the .308 that likes lighter bullets the factory FPGMM 168s shot in the low 0.7s.
The same factory ammo shot in the mid-0.5s with the .308 that likes heavier bullets.
My oldest .308 averaged in the low 0.8s with the same factory ammo.
Most of the samples were small so I don't consider them statistically significant but they are indicative of what each rifle could do.

As you can see from my previous post, all my hand loads did better than factory, but that could be because I also got more accurate (mostly more consistent) as I got used to each rifle.

Best groups for the factory ammo were better than the factory average for each rifle, but using best groups can be deceiving.
Shooter induced variations can pull a shot back into a group just as easily as they can pull a shot out of a group.

It only takes one inconsistent set-up to destroy a group.
Check your groups to see if you have one outlier and a pretty close grouping for the other 4 shots.
That would indicate to me that I was causing the outlier more than suspecting the rifle.

A 1/8th inch variation in eye relief positioning behind the scope from shot to shot can raise or lower POI by 1/4 inch.

Having your stock off the notch in your shoulder between you shoulder bone and the clavicle can move the bullet about 1/2 inch left or right (mostly right if you stock is on your shoulder bone).

With a light rifle with a relatively short barrel, you need to watch out for muscling the stock when aiming.
It doesn't take much.
Changing the grip on your stock, or changing the tightness of your grip or press into your shoulder can also create variations in POI.

Any or all of the variations I listed can easily cause a sub-1-inch group to grow much larger.
It doesn't take many 1/2 inch variations to get a 2 inch group, especially with a new rifle that you aren't yet comfortable with.
It takes me at least 100 rounds to get myself comfortable behind a new rifle.
And if the stock is a bit different or the weight is different, it takes me longer to figure out what the consistent set up needs to be.
A light, short-barreled hunting rifle takes me the longest to get consistent with because the tendency to muscle the stock is greater.

Also, you didn't say what type of scope you were using.
A low power scope with a broad hunting reticle might be obscuring a large part of you aiming point at 100 yards.
All of my groups were shot with a target reticle (1/8th MOA) and fairly high power scopes (24 to 32X) appropriate for target shooting and load development. They were way too much magnification for hunting.
If you can't be sure you are aiming exactly at the same spot for every shot, you can be adding to the variations that grow the group size.

Give yourself a chance to get comfortable with your rifle and experiment with the action screw torque settings.
You didn't say if your stock was an Accu-stock with the aluminum frame (around 50 in.-lbs. torque) or a plastic stock without aluminum pillars which are very sensitive to torque settings (30 to 35 in.-lbs.).
The HS precision stocks have aluminum frames and V-block pillars and get torqued like an Accu-stock.
On some newer models that have aluminum chassis, the chassis can be torqued in the 60 in.-lbs. range.