The fact is, no one "grows" followers. I've never in my time on Medium been asked to follow someone in any manner. (Or asked anyone else to follow me.)

Instead, it's a sign of respect and appreciation. An indication that you have something of value to share, at least to someone. And when that happens, it IS cool. To know someone enjoys and appreciates reading your ideas, and wants access to more of them.

(If you want them to go away, I suppose you could block them. I don't know why that would happen, and I know of no one who has ever made the attempt...or at least related it publicly. But if that follower bothers you, perhaps you can get rid of them fairly easily.)