I would bet your trouble is that simple.
When I set one up I typically measure 3-5 times and never have more than a .002 variation but I have to un-stick the bullet every time.
You do not need to pry or hammer on it but it needs to be "firmly planted" in order to get consistent results.