I try not to get too hung up on 'too slow' or 'fast enough' - for me, walking is not a competition. Some people walk fast. Some don't. Everyone is different. Take it at your own comfortable pace.
However, route planning partly relies on knowing your average walking speed on the terrain you'll be walking. Naismith's Rule is helpful I find. It allows one hour for every three miles distance on flat ground, plus one hour for every 2,000 feet ascent. More about this
on UK Hillwalking.
FWIW, on level surfaced roads I average approx 3.75mph (over a five mile distance).
On easy mountain routes I average about 2mph.
On tough and long mountain routes I average approx 1.75mph.
Here is a logged example of the latter taken from Dave's blog:
Scafell Pike from Seathwaite. Our route was Taylorgill Force, Styhead Tarn, Corridor Route, Lingmell Col and Scafell Pike; then back via Broad Crag, Great End, Esk Hause, and Grains Gill to Stockley Bridge and Seathwaite. The GPS gave the distance as 10.8 miles and the cumulative ascent as 1,182 metres (3,790 feet). The moving time was 05:54 giving an average speed of 1.83mph. (Which is fairly close to the Naismith calculation)