Can't say as it is an issue for me. I like to play "Spot the Rambler" as I go by. (Rules available on application)
If they are in the distance they are scenery. If they are coming towards me I greet them with a cheery "Good Morning or Good Afternoon" (I switch at precisely 12:00 just to annoy folk). I doff my Tilley to any ladies in the group as well.
If I am overtaking, I speed up a bit to get past quickly, again with the cheery greeting ritual so they know I am behind them and not part of the group catching up. Once they are behind me I can't see them anyway. Occasionally, I will take my time to eavesdrop, there are sometimes some weird conversations going on.