I have a couple small tapered drift punches that I use for such stuff. I think they were from a grandfather machinist. One is slightly less than 1/16" at the tip, then 3/32, then 1/8. I would not be surprised if he ground them himself.