AI research project limits exposed as agents score 1 out of 6 on papers
Six days, $3,000 in API credits, and full access to GPUs and the open web sounds like a dream setup for any researcher. Give that same setup to an AI agent, though, and something curious happens: it will spend every last token trying to save an idea that should have been scrapped on day two. That is the core finding behind a new look at AI research project limits, and it points to a gap in autonomous AI systems that has nothing to do with intelligence and everything to do with judgment.
Key takeaways
- Researchers handed AI agents six days, $3,000 in API credits, GPU compute, web access, and two real research questions pulled from unpublished NeurIPS submissions.
- The agents ran their own experiments and wrote full papers with no human writing or editing the underlying code.
- Human experts who had studied the same questions for months scored the resulting papers 2 out of 6 and 1 out of 6 โ clear rejections.
- The agents could manage compute, debug code, and respond to reviewer feedback, but they could not recognize when their whole approach had already failed.
- Fixing the problem will likely require stop conditions, budget checkpoints, and confidence tracking built directly into how agents operate.
Testing AI Agents Against Real Research Questions
The experiment was designed to answer a simple question: can an AI agent run a real scientific research project from start to finish? The setup gave each agent everything a junior researcher might ask for, then let the machines work without supervision.
โฆ Continue reading the full article at the original source below.
