Conceptual Reasoning Index: Best AI model scores 73.6, ceiling is 91

Anthropic and independent researchers have introduced a new way to measure something AI models have historically struggled to prove they’re good at: reasoning through problems that have no clean, verifiable answer. The Conceptual Reasoning Index (CRI), unveiled this week, combines three separate benchmarks to score how well large language models handle philosophical arguments, logical consistency, and decision-theory puzzles — the kind of thinking many researchers believe will matter most as AI systems take on a bigger role in managing their own risks.

Key takeaways

• The Conceptual Reasoning Index combines three benchmarks — LMCA, ACCoRD, and DTBench capabilities — into a single 0-to-100 score of AI conceptual reasoning.
• The top-performing model, Opus 5, scored 73.6 on the CRI, well below an estimated ceiling of around 91.
• The LMCA dataset includes 560 position texts and 1,461 expert-rated arguments covering philosophy, decision theory, and AI risk.
• Scores have climbed roughly linearly since late 2024 with no sign of leveling off.
• The project was built in collaboration with Anthropic and researchers Emery Cooper and Caspar Oesterheld.

Introduction to the Conceptual Reasoning Index (CRI)

The CRI exists because some of the most important work AI systems might eventually do — helping humans understand and plan for advanced AI risk — can’t be checked against a clear right answer the way math or coding problems can. Its creators argue that many of the tasks tied to AI risk mitigation require the kind of argumentation used in philosophy and AI futurism rather than empirical verification, and that current training methods, which lean heavily on data with reliable feedback, tend to leave models weaker at exactly this kind of reasoning.

… Continue reading the full article at the original source below.