Spence left [Boro](https://www.thenorthernecho.co.uk/sport/football/middlesbrough/) for Tottenham in 2022 on the back of an impressive loan spell with Nottingham Forest, who he helped win promotion.
But the full-back's time in the capital got off to the worst possible start when then-boss Conte said: “Spence is an investment of the club. The club wanted to do it. The club decided to buy him."
Given Conte's stance, it was no surprise that Spence didn't feature in the Italian's first team plans and the 24-year-old spent time out on loan at Rennes, Leeds United and Genoa, before forcing his way into the side under current boss Ange Postecoglou this season.
And looking back, he says Conte's comments had a huge impact on him.
"It wasn't a nice feeling to be honest," [he said on the Rio Ferdinand Presents podcast.](https://www.youtube.com/watch?v=rZwvtuAk4TI)
"I was coming to the club on a high. I was confident, buzzing, I'd had a great season and just won promotion with my club. Then when I came in I ran into a brick wall really.
"It shattered my confidence really. It's not nice to hear.
"My agent at the time said don't worry. But it wasn't a nice thing. I was going into the club like 'what am I doing'.
"Whatever I did that man (Conte) wasn't happy about anything. He's not really a complimenting guy. It gets to the point where you don't need validation, you know if you're doing well or not.
"I think you get to a certain time and you know whether you're doing well or not."
**RECOMMENDED READING:**
Spence was allowed to leave Boro after a turbulent spell at the Riverside.
He was spotted and signed after his Fulham release and broke into the first team under Jonathan Woodgate. He showed signs of his promise but struggled for consistency and one or two issues off the pitch led to Neil Warnock saying he'd allowed Spence to join Forest "because he needed to sort himself out".
Warnock also famously said at the time that Spence could be playing in the Premier League or in non-league five years down the line.
"If I'm honest I didn't take it on," said Spence looking back on Warnock's comment.
"I don't know. I had a lot of conversations with him to be fair, and he always said he'd seen a lot of players and you can be the best. He made that comment, I'm either going to be non league or whatever he said.
"It's a weird one because if I go in and do rubbish and become non league, he was right. If I go on and become the best, he was right. It's a weird comment. He's alright, man, he's alright."
Spence admits his first couple of years at Tottenham were tough and thinks he was "too humble" when he first joined the club.
He said: "All the loan moves and everything. I never really settled and never had the chance to play. Now I'm playing week in week out I feel settled.
"It wasn't easy. I went to Tottenham to play. I felt loved at Genoa, that was nice.
"When I was at Rennes I wasn't really settled. It was a difficult time. There was a lot of stuff not going well off the pitch. I was in a hotel for like three months, it was horrible, the food wasn't good. Once I did get my place it wasn't a proper place.
"I think I was a bit too humble (when he joined Spurs) thinking things are just going to happen by doing nothing or saying nothing. I think that happened in my first year at Tottenham, I think I gave too much respect to the players and surroundings.
"I feel like I gave them too my respect and was a bit too humble."