Locating the decisive error in a failed agent trajectory.
When an agentic system fails a task, diagnosing the failure comes down to two questions: which agent or component was responsible (the who) and at which step the error occurred (the when). Together, they identify the decisive error: the earliest action whose correction would have turned the failed trajectory into a successful one.
Automated failure attribution answers these questions with LLMs instead of manual inspection of long execution logs. As agents become more capable, their failures grow subtler and their trajectories longer, making attribution both harder and more valuable. Reliable attribution turns failures into targeted feedback for debugging and improving agentic systems.
We introduce Who&When Pro, a large-scale benchmark for this task. Our warm-start injection pipeline exactly replays a successful prefix and then injects a single error, yielding 12,326 failed trajectories, each labeled with its responsible agent and decisive step. The traces are drawn from 26 source benchmarks in 9 task categories, from coding, STEM, data science, deep search, and embodied tasks to GUI navigation, multimodal search, image QA, and video QA. They are produced by 15 agent frameworks and span 3 modalities (text, image, and video).
A snapshot of what's inside the benchmark.
Run agents on 26 benchmarks across nine task families, recording both successful trajectories and the failures that arise naturally.
Distill the natural failures into 18 error modes spanning planning, tool use, perception, memory, and coordination.
Turn successful runs into 12,326 labeled traces with warm-start injection, explained below.
Warm-start injection is how we keep the labels exact. We take a run that already succeeded and replay it one step at a time, rebuilding the agent's memory and tools, up to a single chosen step. There we swap in one mistake, then hand control back and let the agent finish on its own. Every step before the mistake matches the successful run, so that one swapped step is provably the first thing that went wrong. The which step and which agent labels are then exact by construction, not estimated after the fact.
| Model | Text | Image | Video | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Agent | Step | Error | Joint | Agent | Step | Error | Joint | Agent | Step | Error | Joint | |
All-at-once protocol. Best per column shown bold & tinted, second-best underlined. — = modality not supported by the model. Metrics: Agent = responsible-agent accuracy, Step = decisive-step accuracy, Error = failure-mode macro-F1, Joint = all three correct.
Models locate the decisive step most accurately on text traces, where the strongest systems are right about 7 in 10 times. Accuracy drops on image traces and drops further on video. A longer trace simply gives the model more to read before it can pinpoint the single step that caused the failure.
Identifying the error type is the weakest skill for every model we tested. A failure that actually starts in planning, verification, or coordination is often labeled a reasoning error. The model responds to the symptom it can see instead of tracing the failure back to the action that caused it.
We compared three ways of showing a trace to the model. The all-at-once setting — the complete trace in a single pass — is the most accurate, beating both a step-by-step pass and a binary-search pass that reveal the trace a piece at a time.
@misc{whoandwhenpro2026,
title = {Who\&When Pro: Can LLMs Really Attribute Failures in AI Agents?},
author = {Liu, Jiale and Xi, Huajun and Zhang, Shaokun and Zeng, Yifan and Yue, Tianwei and Wang, Chi and Kang, Jian and Wu, Qingyun and Wang, Huazheng},
year = {2026},
eprint = {2607.09996},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2607.09996}
}