Eggheads have taken a look at previously developed techniques that can be used to trick self-driving cars into doing the wrong thing – and found cheap stickers stuck on stop and speed limit signs, at least, are pretty effective.
They also learned of a specific phenomenon where the systems "memorized" signs, so that when they were hidden, the vehicle still assumed in its calculations that the signs would be present in the position where it first detected them, even when they were hidden from view, leading to "lower-than-expected attack success rates" in the wild.
The scientists – from the University of California, Irvine, and Drexel University, in the USA – are following in the footsteps of earlier studies that have shown patterns of light directed at signs can confuse autonomous vehicles, as can carefully placed tinfoil and paint, electrical tape, and stickers on the road, as well as manipulative audio, and no doubt other approaches.
The tech behind the AI algos, object identifiers, image classifiers, and the other sensor information used by self driving vehicles to squire humans around has developed in leaps and bounds, but computer vision still misses some of the trillions of context clues gathered by a human eye attached to a meatbag's brain from years of observed human life and perfected over millennia of evolution. The human brain might reason, for example, when it sees a sticker on a stop sign: "Oh look, that piece of street furniture is the right size and shape to be a stop sign, but there is a sticker on it. Yep, some idiot has defaced it. I'll stop the car at the sign anyway."
But the object detectors and image classifiers used in self driving systems don't work the same way.
For this latest research, the team focused on traffic sign recognition (TSR) systems used by self-driving cars, and devised a way to measure the effectiveness of past adversarial attacks.
Wang said his team's attack vectors were made up of stickers that had swirling, multicolored designs that confuse AI algorithms used by image classifiers and object detectors you might find in a self driving car's TSR system. Some of these can cause a Stop sign to "disappear" according to the detector — either by covering the sign with an adversarial Stop sign poster, or by adding adversarial stickers onto the sign.
Researchers in the Donald Bren School of Information & Computer Sciences at UC Irvine have demonstrated that traffic sign recognition systems in autonomously driven vehicles can be tricked into either seeing nonexistent roadside commands or not seeing actual ones, leading to aberrant and potentially dangerous driving behavior
Examples of the modified stop signs from the study ... Source: Ningfei Wang, UC Irvine – click to enlarge
In the photo above, for example, the top two leftmost signs use the RP2 [PDF] algo, or the Robust Physical Perturbations algorithm, for the attack. RP2 can generate a perturbation that maximizes the probability that an object detector mis-classifies the object. As the position of the stop sign changes relative to the car driving towards it, the grid cells the object is contained in (and the corresponding network weights) change as well and the perturbations have to be applicable to multiple grid cells simultaneously. Crucially, while they are large enough to be visible to the human eye, these malicious stickers can look like graffiti or "subtle lighting artifacts that could be considered benign" – you can see from the illustration that they don't even obscure the letters STOP. To us they look a bit like a tiny cut-and-stick poster for an underground techno club.
"These stickers can be cheaply and easily produced by anyone with access to an open-source programming language such as Python and image processing libraries. Those tools combined with a computer with a graphics card and a color printer are all someone would need to foil TSR systems in autonomous vehicles," says the study's lead author Ningfei Wang, a research scientist at Meta who performed this work as a PhD student in computer science at UC Irvine.
Specifically, the paper used those existing academic attacks and evaluated them against existing commercial systems used in cars currently on the road. They divide the attacks into "hiding" and "appearing" attacks. As the names suggest, the first method tricks the system so it cannot detect a legitimate traffic sign and the second adds information in places where it shouldn't be, triggering false detection.
They found that low-cost methods — primarily, specially designed stickers adhered to stop and speed-limit signs — can indeed (as previously shown) make such signs undetectable to TSRs in some vehicles, while making nonexistent signs appear out of nowhere to others. Such attacks could result in cars ignoring road commands, triggering unintended emergency braking, speeding, and other rule violations.
In a statement this week to mark with the team's work being presented at the Network and Distributed System Security Symposium in California, Alfred Chen, a UC Irvine assistant professor of computer science, acknowledged academics have studied driverless vehicle security for years and have discovered various practical security vulnerabilities in the latest autonomous driving technology.
"But these studies have been limited mostly to academic setups, leaving our understanding of such vulnerabilities in commercial autonomous vehicle systems highly limited. Our study fills this critical gap," he argued.
The team's conference presentation [Arxiv paper], "Revisiting Physical-World Adversarial Attack on Traffic Sign Recognition: A Commercial Systems Perspective," claims this study was the first large-scale measurement of "physical-world adversarial attacks against commercial TSR systems." In other words, if you wondered how well the aforementioned techniques actually work, this paper tries to figure that out.
The write-up states that while the results reveal it is possible for previous trickery to have a 100 percent success rate against certain commercial TSR systems, these results cannot be generalized across all models.
Also, somewhat surprisingly, the study observed "much lower-than-expected attack success rates overall." This is due to a function the researchers observed across all the cars it tested that hadn't previously been accounted for or known about in prior research: "spatial memorization design."
In one test, the boffins show a sign to the vehicle for a short time (sign display time), and later hide the sign and wait for a certain time duration (sign disappearing time). After that, they drive the vehicle past the original sign-display position to measure whether the sign detection result is spatially memorized. They did the same for speed limit signs, finding that all the vehicle models they tested that supported speed limit signs (not all did) only displayed detection results after the vehicle passed the sign (thus preventing the vehicles from speeding before they reached the sign's location if the new speed limit was higher). The cars were all tested at 5 mph (8 kph) on a "rooftop parking structure," for anyone worried about the testers' safety.
The researchers tested five models of publicly available cars with some level of automated driving capabilities (namely, the Tesla Model 3 2023, Toyota Camry 2023, Nissan Sentra 2023, Mazda CX-30 2023, and Hyundai Tucson 2024) and did not identify which models were vulnerable to which attack, for ethical reasons.
Of its spatial memorization design findings, the paper says that the technique is common among today's commercial TSR systems. The boffins found this means, among other things, that "hiding attacks" (making the sign "disappear" for the TSR system) are "theoretically harder (if not equally hard) than appearing attacks" when it comes to fooling the systems. The reason for this is that when a TSR system initially detects a sign and its location, it appears to "remember" it until it has passed the spot where the sign is supposed to be, even if the sign is hidden during parts of that journey.
Conversely, so-called "appearing" attacks are much more successful – spatial memorization means spoofing a fake stop sign is "much easier than we expected," says the study's lead author Ningfei Wang.
This is all useful information on the road to hardening self driving vehicles against these kinds of attacks.
Not only did the study allow the team to responsibly disclose where attacks had been successful to the self-driving car vendors to allow them to make tweaks where necessary, it also meant that the boffins could mathematically model the apparently altered risk profile of attacks with the new information they gleaned. This hopefully means more small improvements on the journey towards making these systems safer overall. ®