Is it possible for a wood dowell used to push a patch through to do damage to the bore when run through from the breech?

It is mainly all I use,since I got used to using wood with my black powder cartridge rifle's.....