How Researchers Hacked AI Robots Into Breaking Traffic Laws—And Worse

Researchers showed vulnerabilities in AI-powered robots being exploited to perform dangerous tasks, including detonating bombs.

Penn Engineering researchers have uncovered critical vulnerabilities in AI-powered robots, exposing ways to manipulate these systems into performing dangerous actions like running red lights or engaging in potentially harmful activities—like detonating bombs.

The research team, led by George Pappas, developed an algorithm called RoboPAIR that achieved a 100% "jailbreak" rate on three different robotic systems: the Unitree Go2 quadruped robot, the Clearpath Robotics Jackal wheeled vehicle, and NVIDIA's Dolphin LLM self-driving simulator.

"Our work shows that, at this moment, large language models are just not safe enough when integrated with the physical world," George Pappas said in a statement shared by EurekAlert.

Alexander Robey, the study's lead author, and his team argue addressing those vulnerabilities requires more than simple software patches, calling for a comprehensive reevaluation of AI integration in physical systems.

Jailbreaking, in the context of AI and robotics, refers to bypassing or circumventing the built-in safety protocols and ethical constraints of an AI system.

It became popular in the early days of iOS, when enthusiasts used to find clever ways to get root access, enabling their phones to do things Apple didn’t approve of, like shooting video or running themes.

When applied to large language models (LLMs) and embodied AI systems, jailbreaking involves manipulating the AI through carefully crafted prompts or inputs that exploit vulnerabilities in the system's programming.

These exploits can cause the AI—be it a machine or software—to disregard its ethical training, ignore safety measures, or perform actions it was explicitly designed not to do.

In the case of AI-powered robots, successful jailbreaking can lead to dangerous real-world consequences, as demonstrated by the Penn Engineering study, where researchers were able to make robots perform unsafe actions like speeding through crosswalks, stomping into humans, detonating explosives, or ignoring traffic lights.

Prior to the study's release, Penn Engineering informed affected companies about the discovered vulnerabilities and is now collaborating with manufacturers to enhance AI safety protocols.

"What is important to underscore here is that systems become safer when you find their weaknesses. This is true for cybersecurity. This is also true for AI safety," Alexander Robey, the paper's first author, wrote.

Researchers have been studying the impact of jailbreaking in a society that is increasingly relying on prompt engineering—which is natural language “coding.”

Notably, the "Bad Robot: Jailbreaking LLM-based Embodied AI in the Physical World" paper discovered three key weaknesses in AI-powered robots:

The "Bad Robot" researchers tested these vulnerabilities using a benchmark of 277 malicious queries, categorized into seven types of potential harm: physical harm, privacy violations, pornography, fraud, illegal activities, hateful conduct, and sabotage. Experiments using a sophisticated robotic arm confirmed that these systems could be manipulated to execute harmful actions. Besides these two, researchers have also studied jailbreaks in software-based interactions, helping new models resist these attacks.

This has become a cat-and-mouse game between researchers and jailbreakers, resulting in more sophisticated prompts and jailbreaking approaches for more sophisticated and powerful models.

It’s an important note because the increasing use of AI in business applications may bring consequences for model developers right now, for example, people have been able to trick AI customer Service bots into giving them extreme discounts, recommending recipes with poisonous food, or make chatbots say offensive things.

But we'd take an AI that refuses to detonate bombs over one that politely declines to generate offensive content any day.

Edited by Sebastian Sinclair

More News

• All
• CT
• Coindesk
• Decrypt
• DLNews