Whatever the reason, that stubborn determination has fired him to a Premier League appearance record and a central role in the world’s most competitive league at 40.
He will hope to show that dogged edge again when Albion face Nottingham Forest in what be Prem appearance 655.
“Some things don't change,” Milner said with a laugh after being asked whether he still wanted to prove people wrong.
“Football's brilliant, isn't it? You always have your doubters.
“No matter what you do or what you achieve, there's always someone who believes you're not very good or someone believes someone's better or you shouldn't be playing or you're not playing in the right position or you have played too many minutes.
“It's fantastic. It's a game of opinions and you're hearing the opinions at the moment.
“Who's the greatest goalkeeper ever? Who's the greatest centre-half ever and things like that.
“I'm not really one to get involved in those conversations, my job is to do my job and let other people talk about it really. But I think it's always a driver.
“You want to be the best and you can't change it. There are people who are always going to doubt you.
“But that's always something that's been at my forefront, to prove it wrong.
“Whether it's if you're not in the team - alright, I'm going to show you, I'm going to be in the team, I want to be the best player in my position.
“I don't know where that came from, whether it was younger or my dad used to play on that.
“He knew what I was like and he used to say, "You don't work hard enough, you're not going to make it".
“He knew what he was doing and it probably comes from that.
“He would never say it in a horrible way or anything like that, but he knew what I was like and he'd be like, ‘Youve got no chance of scoring the goal from there’ and then two shots later it's in the top corner.
“Things like that. He knew how to get the best of me.”
That stubborn element also got Milner through serious hamstring and knee injuries, followed by nerve damage, which ruled him out from the third game last season until the closing moments of the final fixture, when he was sent on at Spurs.
He admitted: “I think most people, the surgeon, the physio, people who knew what I had, probably every single person thought I was finished - at my age being a footballer and what I had.
“So it was probably that same thing that meant I'm still playing now really to be honest, to just prove and not finish your career in a way that wasn't really in your control.
“It still might happen, whenever it happens. I don't know what's going to happen.
“I’m taking it a day at a time, but that was the drive really.
“I wanted to try and prove that I could get back from that because I don't think many people could.
“And then it's one thing getting back then it's, ‘Can you go and play in the Premier League?’
“Which is tough to do at my age anyway. And then, after having the season I had last year, so that's pleasing that I've managed to be able to do that.
“And then you move on to the next challenge after that.”
Milner has always been teetotal and says he is fortunate to have played at a time when sports science was improving rapidly.
James Milner has a senior role at Albion (Image: Richard Parkes)
He added: “I always ask questions. Why are we doing this?
“Or explain it to me so I understand and then I decide if it is the right thing to do.
“Give me the science behind it. Why are we doing this? Why am I doing an ice bath? Where's the proof behind that?
“And then you build up what you think. I went away from doing ice baths in pre-season a few years ago because I wanted the adaptation of the training and I felt better for doing it and now I save ice baths for after the games.
“Some people like ice baths every single day. Everyone's completely different.”
Milner says he will need something to replace the structure of his football career when he hangs up his boots.
But he does not fear retirement and the void that might mean.
He said: “I think it's a tough question to know when's the right time, because I feel like I still can do it now, but do you go to the point where you can't? Is that too long?
“I finished the game at the weekend and some of the boys went, 'You can't retire Millie this year, you've got to go again next year'.
“So I think that's always a tough decision to get the right time.
“I feel like the coaching side I've been exposed to, in terms of a bit last year here and the badges I've done and things like that, interest me at times.
“But Jurgen [Klopp] always said, whenever you finish, you need to have a rest and a break straight away and I think that's pretty appealing at this moment in time.
“As hard as I work in the season, when we have an international break and we have some days off, I look forward to that, go and play golf, spend time with the family or whatever.
“Whenever I finish, I'll probably have a break and a rest, I think, probably because I've been so 100 for so long.
“I look forward to having that break and having time to do what I want, but I think most people say you miss the structure and what you are training for every day.
“You are going in the gym and it's hard, it's for a reason, it's for the next game and things like that.
“When you haven't got that, I think I'd probably look towards doing marathons or something like that to give me something to go for again.”
Just try and tell him he CAN’T do it.