We all have found little things that change our intended point of impact unexpectedly even in calm winds.

One thing that makes my heartbeat visible in a high powered scope is too tight a cheek weld.
Once I consistently set up a cheek weld and didn't try to press my cheek too tightly into the stock, I stopped seeing heartbeats.

I also find that not setting the stock in exactly the same position in your shoulder causes horizontal movement in the point of impact.
When my shoulder mount is out of the notch in my arm socket and the rifle recoils, my sight picture is moved to the right and after recoil the aim point is to the right and so is the point of impact.
When I set the stock correctly, the rifle comes straight back and my rifle is aimed at my aim point and the point of impact is at my aim point.

Stock positioning is the cause of the majority of my shooter induced horizontal variation.
It has taken a lot of practice to get my set up correct the majority of times and my group averages, even in winds above 5 mph, are smaller and there is far less variation.