LLM security vulnerabilities may be unfixable, ICML study warns
There may be no such thing as a fully secure large language model. That is the uncomfortable conclusion of a new paper presented at the 2026 International Conference on Machine Learning (ICML), where researchers argue that LLM security vulnerabilities are not just a product of incomplete training or lazy red-teaming — they are baked into the fundamental architecture of how these systems work.
Key takeaways
- A fundamental flaw in how LLMs identify instruction sources makes them inherently and persistently vulnerable to manipulation, according to research presented at ICML in July 2026.
- The attack technique, called chain-of-thought forgery, won OpenAI’s red-teaming hackathon in August 2025 and has since been shown to affect models from OpenAI, Anthropic, Alibaba, and DeepSeek.
- LLMs track instruction sources using role tags, but research shows they actually rely on text style rather than tags — meaning attackers can spoof any role simply by mimicking the right writing style.
- Training and red-teaming that focus on role detection cannot fully close this gap; no list of disallowed instructions is exhaustive.
- Researchers advise organizations to treat all LLM agent outputs as potentially unsafe, especially in sensitive or critical deployments.
Fundamental Flaw in LLM Instruction Source Identification
The core problem is deceptively simple. When an LLM processes text, it needs to know who is talking — is this instruction from a user, the system designer, a tool, or the model’s own internal reasoning? To manage that, chatbots use role tags: text wrapped in labels like
… Continue reading the full article at the original source below.
