AI Models Nail Only 39.6% of Steps in Attack Chain Reconstruction Test

Security researchers have long hoped that large language model agents could speed up one of the most tedious parts of incident response: piecing together how an attacker moved through a system, step by step, using scattered logs and telemetry. A new diagnostic benchmark called DiagChain puts that hope to the test, and the results suggest the technology still has a long way to go before it can reliably handle attack chain reconstruction on its own.

Developed by a team of researchers including Xuyang Liu, Yibin Han, Zhenwei Zhang, Kai Chang, Zhiwei Xu, Tian Qiu, Weixian Deng, Jiabao Gao, Xiaolin Peng, Hai Wan, and Xibin Zhao, DiagChain isn’t just another accuracy scoreboard. It’s built specifically to show where and why LLM agents fail when they try to reconstruct the ordered sequence of actions an attacker took, based on evidence pulled from system telemetry.

Key takeaways

• DiagChain is a diagnostic benchmark that evaluates LLM agents on evidence-grounded attack chain reconstruction, going beyond simple pass/fail accuracy.
• The benchmark’s MAIN-69 suite includes 69 scenarios spanning multiple operating systems, evidence noise levels, and chain lengths.
• A new method called ECRAG pairs evidence retrieval with an evolving structured representation of the chain being reconstructed.
• Across 6 different LLMs tested, the best configuration succeeded on just 39.6% of 849 reference steps.
• Smaller models struggle to use retrieved evidence at all, while larger models struggle mainly with putting that evidence in the correct order.

Introducing DiagChain: A New Benchmark for Attack Chain Reconstruction

DiagChain exists because most existing benchmarks only look at final outputs or overall accuracy scores, offering little insight into how mistakes actually form during an agent’s reasoning process. That’s a meaningful gap for cybersecurity teams trying to decide whether an AI agent is trustworthy enough to help triage a real intrusion.

… Continue reading the full article at the original source below.