I never bother to anneal. But I do outside trim necks. It depends on your cartridge shoulder angle, load intensity, and maybe even your individual chamber.

If your cases don't requite trimming to length after 15 reloads then annealing probably isn't worth the bother.

I find my primer pockets get loose and I disguard cases before annealing would have been worthwhile.