It helps if you use a drift punch from the other side. push the drift through the trigger hanger and sear, stop at the spring and work it through the spring's hole. Then pressing the punch back out using the hanger pin with resistance on the drift. You may need to work it around but it should go through easily.
For reference.
Bookmarks