Bolt handle removal

I am trying to remove the teardrop bolt handle on a 1979 "J " series and have dis-assembled the bolt but cannot get the handle to come off. I am the original owner and this gun has no rust anywhere. Removed the barrel yesterday without any problems. What's the trick with the handle?

just a good fit, It will come off. have a vice with wood blocks or something you can put the bolt tube in supported by the handle and tap on the loose screw knocking the tube out of the handle?