Blues substitute Patrick Roberts struck an 89th-minute equaliser to cancel out Lukas Nmecha's second-half strike and there were no goals in extra-time, but Leeds held their nerve in the shootout.
Leeds quartet Joel Piroe, Dominic Calvert-Lewin, Brenden Aaronson and Longstaff all converted, while Tommy Doyle's effort for Birmingham was saved and Roberts fired his effort over the crossbar.
Leeds, who won 3-1 at Derby in the previous round, will now bid to reach the quarter-finals for the first time since 2002-03.
They were under the cosh in the first half yesterday and Birmingham went close to taking an 11th-minute lead.
Jay Stansfield hammered a rising drive that was brilliant finger-tipped on to the crossbar by Leeds goalkeeper Lucas Perri.
Blues revealed this week chairman Tom Wagner was recovering from a stroke and home fans unfurled a huge 'get well' banner before kick-off, while there was a minute's applause in the 13th minute in tribute to their American owner.
Birmingham swarmed all over Leeds in that opening 20 minutes, as Jhon Solis and Kai Wagner saw shots held by Perri and skipper Christoph Klarer's header was blocked.
And Perri denied Birmingham what would have been a deserved lead just before the break with another full-length save to keep put Demarai Gray's low shot.
Leeds captain Ethan Ampadu replaced the ineffective Facundo Buonanotte at the interval and the visitors took the lead within four minutes of the restart.
Klarer lost out to Nmecha on the edge of the area and the striker crashed home a superb finish for his seventh goal of the season.
Lukas Nmecha celebrates with Ao Tanaka after scoring the opener yesterday. (Image: Jacob King/PA Wire)
Birmingham mounted a spirited response, forcing a series of corners as Stansfield fired wide and August Priske's effort was blocked.
But with Ampadu prominent, Leeds carried more of a threat in the second period, only for Roberts to conjure a late equaliser.
His shot from outside of the box flew into the net via a deflection off James Justin, yet there was still more drama to ome in normal time.
Calvert-Lewin, who had replaced Nmecha, saw his goalbound shot deflected over while Blues hit a post through Ibrahim Osman's low shot, with fellow substitute Kanya Fujimoto's follow-up effort blocked.
Osman spurned another chance at the far post in the first period of extra time, while Piroe, Calvert-Lewin and Ampadu all had chances for Leeds in the second, as a thrilling cup-tie flowed from end to end before being decided on penalties.