As soon as it was on and Spence got up, Sandro Tonali swung the corner in before the Spurs man had got back into the box, and Schar - whom Spence was supposed to mark - headed home.
Spurs were adamant Spence should have been allowed time to get into his defensive position and Frank viewed it as a significant error.
Tottenham protested Chris Kavanagh’s decision to no avail
Action Images via Reuters
"I think for me, we conceded two goals because of two mistakes," Frank said.
"Unfortunately, the ref was part of the mistake in the first one. I'm not saying there's things we can't do better, but his boot is off, we need to tie his shoelace and tie his boot.
"And you can clearly see on the footage afterwards that the whole boot is off, so we need to do it. And it's minute 23, it's not like we delayed the game for one minutes and we have one goal up. It's common sense to allow him time to get in. So as soon as he had the boot on, the ref let it go."
Frank admitted his side were to blame for the second goal, when Antonin Kinsky came for a cross and did not get there to allow Nick Woltemade to head into an empty net.
However, he was particularly aggrieved by the opener as he revealed it would have been Spence marking Newcastle's scorer.
"To make it perfect, Djed's man he had to mark was Schar, that scored," Frank said.
"So that's why we were very animated. I think that was the biggest one. I think the rest was fine. There's always small bits you can be irritated about, but it's very difficult to ref a game.
"But this for me was unfortunately a big mistake. And then we made a mistake ourselves on the second one. So small margins decided the game today."