By Lucy Wang
AI systems are becoming more capable, and they are now used for everyday work, learning, coding, and decision-making. But they are still not fully reliable. They can hallucinate, misunderstand context, and sometimes be pushed past the safeguards meant to keep them from giving harmful answers. Today’s language models contain a vast amount of knowledge, much of it useful in the right setting and dangerous in the wrong one. A model could avoid some risks by refusing almost everything, but that would make it far less useful. The harder problem is figuring out how to keep these systems helpful while making them harder to manipulate.
That problem becomes especially difficult in long conversations. A harmful request may not appear all at once. Instead, a user might build toward it gradually, using a series of prompts that seem harmless on their own. For researchers, this raises two connected questions: how can these attacks be stopped, and how can they be evaluated fairly enough to know whether a defense is actually working?
Those questions motivate Project 185, a Berkeley Master of Engineering Capstone project advised by Benjamin Erichson, a staff scientist at Lawrence Berkeley National Laboratory and the International Computer Science Institute (ICSI). The student team, comprised of Huanli Gong, Jing Ting Zheng, Xinkai Zhang, and Yuchen Zhang, is studying how to defend large language models against prompt attacks, in which users try to manipulate a model into producing harmful information. Their work focuses not only on developing stronger defenses but also on creating better ways to test attacks and defenses under consistent conditions.
“Whenever a technology becomes this capable and this widely used so quickly, safety and security become just as important as performance. Our team is interested in what happens when people try to misuse AI systems or get around their safety protections,” said Erichson.
One core problem the project addresses is “jailbreak attacks,” a type of prompt attack meant to bypass the safeguards built into AI systems. LLMs such as Claude and ChatGPT are designed to avoid responding to unsafe requests, such as requests for dangerous instructions or assistance with self-harm. But those safeguards are not perfect. With the right wording, context, or sequence of prompts, jailbreak attacks can sometimes trick LLMs into producing harmful responses.
“A useful way to think about this is as a kind of social engineering for AI,” said Erichson. “With people, social engineering means tricking someone into giving away information or doing something they normally would not do.” In an AI setting, the same idea can appear as a prompt that hides a harmful request inside an ordinary-looking task, such as asking for a fictional scene instead of directly asking for dangerous instructions. “Early attacks like this showed that AI systems could sometimes be manipulated in surprisingly human-like ways,” he said.
Over time, jailbreak attacks have become more subtle. Instead of giving one unsafe prompt that clearly violates a model’s guidelines, a user may introduce harmful intent gradually through a series of prompts that seem harmless on their own. These multi-turn attacks are especially difficult as AI systems become more capable and able to handle longer conversations, giving harmful intent more room to hide inside what otherwise looks like a normal exchange.
Before researchers can defend against multi-turn jailbreaks, they need to understand how these attacks work and how to evaluate them fairly. Existing methods are often tested under different conditions, with different limits on conversation turns, retries, judge models, and prompt-generation strategies. A judge model is another AI system used to evaluate whether a response is unsafe or whether an attack is getting closer to success. When each study uses a different setup, it becomes difficult to know whether one method is actually stronger or whether it simply benefited from more favorable testing conditions.
That kind of evaluation matters because better defenses depend on a better understanding. Researchers need to know which parts of an attack drive success, where current safeguards fail, and what kinds of risky behavior models should learn to recognize. A benchmark can also produce useful data for future safety training, especially for attacks that unfold gradually across a conversation.
To address this need, the team created MT-JailBench, a modular benchmark for evaluating multi-turn jailbreaks under fixed conditions. “Our goal was to build a more standardized benchmark so we can test methods under the same conditions and better understand what is actually driving the performance of both attacks and defenses,” said Erichson.
The framework breaks each attack into separate components, including how prompts are generated, refined, judged, and controlled over the course of a conversation. This allows researchers to compare methods side by side and see which parts of the process contribute most to attack success. The benchmark showed that evaluation choices can substantially change how attacks rank against one another. It also found that prompt generation plays a major role in performance, while other components, such as refinement and flow control, provide more moderate gains.
Those findings also point toward better defenses. If researchers can see how an attack succeeds, they can look for places to interrupt it. One such place is the feedback loop used in advanced multi-turn jailbreaks, where a judge model evaluates the target model’s responses and helps guide the attacker’s next prompt.
To disrupt that loop, the team developed D-Judge, a defense against multi-turn jailbreak attacks that was accepted at the International Conference on Machine Learning (ICML) 2026. D-Judge rewrites the target model’s responses before they reach the judge, preserving the original meaning while making the feedback less useful to the attacker. In that sense, D-Judge works as a counter-AI defense: it uses one AI system to interfere with the way another AI system helps carry out an attack. In experiments, D-Judge significantly reduced the success rate of advanced multi-turn jailbreaks while preserving performance on benign tasks.
Together, MT-JailBench and D-Judge reflect the path from evaluation to defense. MT-JailBench helps researchers understand how multi-turn jailbreaks work, compare attacks under consistent conditions, and gather data that could support future safety training. D-Judge uses that understanding to intervene in the attack process itself. The aim is not to make language models so cautious that they stop being useful, but to make them safer and harder to misuse as more people rely on them for everyday work.