Totem Poles are magical devices which, when worshiped, trigger a effect. This effect will often grant the player access to previously unreachable parts of the world, or cause massive devastation to the enemy. If you right-click on the Totem Pole, it will show you how many of your Followers are needed for the red bar to increase at maximum speed. 

Totem Poles differ from Stone Heads in that they cannot contain a discovery. If a discovery was assigned to a Totem Pole, it would get converted into a stone head instead.

It's also impossible to sink a totem pole.