If you have plastic followers, yes, they should be changed out. A lot of shotguns come with good followers in them already, like the VM and the M2. The nose of the follower can also engage the shell latch at different locations which can result in lifter interference, so it needs to be watched.
Plastic parts, in general are good for some things, but running back and forth in steel tubes is not one of them. When they pick up a piece of sand or grit, it can end up causing some significant reliability issues.