I'm another one who doesn't really train for Long Distance Walks, and I find day 3 a bit of a strain but am usually fine after that.
I have never been one for going to the gym, but the year I did, I didn't really find it helped much with walking fitness. I'm not sure why, but guessed it was to do with the fact that what you do in the gym is steady and rhythmic and on smooth surfaces, and the great outdoors isn't like that. The best training for walking is walking really, and as everyone else says, getting used to long days and finding the speed you can keep going all day. Also, finding your daily rhythm. I like to be up and away early but want to know that my dinner and bed isn't far away by about 5pm. A friend I did long walks with when I was young couldn't get going before 11 in the morning but was happy carrying on late in the evening. I could fit in with that 40 years ago but now I really can't, and a beautiful walking partnership ended.