Emery said his side were denied a clear penalty in the first half when Morgan Rogers appeared to be pulled down by Elliot Anderson.
Referee Sam Barratt ignored Villa protests and VAR Brooks did not intervene.
It looked like the decision would not prove pivotal as Villa went ahead in the second half through Jhon Duran, moments after a wonder save from Emiliano Martinez, only for goals from Nikola Milenkovic and Anthony Elanga – in the third minute of time added on – to turn the game around.
Unprompted in his post-match press conference, Emery said: “Now I have watched it, it’s a clear penalty.
“It’s a massive mistake from the VAR. Today the VAR is a big mistake because it was clear, he (Rogers) had an advantage.
“Please. OK, not the referee. I respect the referee and I respect the VAR, but the VAR is there to help the referees.
“We watched it after because everybody is VAR. We are watching it on TV, it’s a clear penalty. Where is the VAR?”
It was the fifth time from six attempts that Villa have failed to win on the back of a midweek Champions League outing as they struggle to cope with the demands.
Emery is not allowing to use that as an excuse.
“Champions League is different, it’s completely different, it’s another competition,” he added.
“Last year we played in the Conference League and we got fourth, it’s not an excuse.”
Forest may be beginning to have their own Champions League dreams after they moved into the top four.
Victory, which came on the back of beating Manchester United at Old Trafford last week, continues their remarkable turnaround from when they survived relegation on the final day of last season.
Boss Nuno Espirito Santo has masterminded that and wants to enjoy a “great moment”, without getting too carried away.
Asked whether they can compete for a top-four spot, he replied: “We are competing against all the teams in the Premier League, it’s not about that, it’s about preparing the next one, focusing on the task ahead of you and realising that you didn’t achieve anything.
“Today is a big moment, let’s enjoy it together, all the fans, players. But we know that Brentford is going to be so tough, just look how many goals they have scored.
“We are now trying to work on that, that is what is most important, game-by-game. I think we should enjoy this because they are special moments. The previous game, today. We want to do it together, us and the fans, today was very special.”