AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

Most AI tests only check if an AI can answer a question once. AUTOLAB, a new benchmark introduced by researchers from institutions including Stanford, MIT, and NVIDIA, shifts the goalposts. It tests if an AI can work like a real engineer for hours. It evaluates if an agent can run experiments, fix mistakes, and constantly improve its work until it reaches a goal. This moves us away from "one-shot" reasoning and toward the messy, iterative reality of scientific and engineering progress.

The failure of short-horizon evaluations

Scientific and engineering advancement is rarely the result of a single brilliant insight. Instead, it is a long-horizon, iterative process. An engineer must propose a change, run an experiment, measure the outcome, and refine the artifact based on empirical feedback (data gathered from physical or simulated observation). Current evaluations for frontier Large Language Models (LLMs) are fundamentally ill-equipped to measure this. Existing benchmarks primarily focus on two extremes. They use static, single-turn coding tasks to test immediate knowledge. Or, they use short-trajectory agentic benchmarks that track only a few interaction steps.

These approaches fail to capture the "sustained" nature of real-world R&D. In a professional setting, an engineer might spend six hours debugging a CUDA kernel (a program designed to run on a GPU) or tuning a machine learning model's hyperparameters (settings that control the learning process). A model might provide a syntactically perfect piece of code that is nonetheless functionally suboptimal or inefficient. Current benchmarks do not penalize this lack of persistence. They only care if the first attempt is "correct." As the authors note, this leaves a massive gap in our understanding of whether frontier models can actually handle tasks that play out over hours rather than minutes.

Designing a closed-loop sandbox

To bridge this gap, the authors developed AUTOLAB. This benchmark consists of 36 expert-curated tasks across four distinct domains: system optimization, puzzles and challenges, model development, and CUDA kernel optimization .

Unlike traditional benchmarks, every task in AUTOLAB starts with a codebase that is functional but intentionally suboptimal. The agent's goal is not just to "solve" the task. It must optimize a specific metric—such as runtime, throughput, or model accuracy—within a strict wall-clock budget (the actual elapsed time in the real world).

The mechanism relies on a sophisticated, closed-loop architecture .

The agent operates within a containerized sandbox (a virtualized, isolated environment like Docker) that provides the necessary tools. The workflow follows a rigorous cycle: 1. Inspection: The agent reads the existing codebase and the task instructions. 2. Modification: The agent proposes and implements edits to the code. 3. Execution and Measurement: The agent runs the code in the sandbox to gather empirical data. 4. Iteration: The agent analyzes the results and decides whether to continue refining or to submit the final solution.

Crucially, the evaluation is "hack-resistant." The authors employ sealed verifiers that the agent cannot see. This ensures that models cannot simply find shortcuts in the scoring logic. They also use "anchored relative scoring." This maps raw performance metrics onto a continuous $[0, 1]$ scale. It uses the performance of the initial baseline and a human-written reference solution as anchors. This allows the benchmark to reward partial progress. It provides a granular view of how close an agent gets to an optimal result.

Persistence over initial brilliance

The results of the evaluation reveal a surprising hierarchy of capabilities. Evaluating 17 state-of-the-art models, the authors found that claude-opus-4.6 is the clear leader. It achieved an Average@3 score of 0.68 and a dominance score of 0.93 . A dominance score of 0.93 means it outperformed almost every other model in head-to-head matchups. However, the most significant finding is not which model won, but why they succeeded.

The paper reports that the dominant predictor of success is not the quality of an agent's initial attempt. Instead, success comes from persistence in the iterative loop. In a case study involving the optimization of a Flash Attention kernel, the authors observed wildly different trajectories .

While many models plateaued near the reference solution, claude-opus-4.6 achieved a 42.4× speedup. It did this by performing 44 consecutive feedback-driven iterations over approximately 40 minutes.

Conversely, many powerful models failed due to poor "time awareness." The authors categorize these failures into distinct modes .

Some models, like gpt-5.4, suffer from premature termination. They perform a few quick edits and then simply give up. They do this even when they have hours of budget remaining. Others, particularly certain open-weight models like deepseek-v4-pro, exhibit the opposite problem. They enter exhaustive reasoning chains that consume the entire time budget. They do this without ever actually submitting a finished solution. This suggests that for long-horizon tasks, managing a compute budget is just as vital as raw coding intelligence.

Limits of the autonomous researcher

While AUTOLAB provides a significant leap in evaluation rigor, it is not a universal simulator for all scientific inquiry. The authors are transparent about several limitations. First, the benchmark is strictly focused on executable engineering and machine learning workflows. It measures "measurable auto-research." This refers to work that can be quantified by a runtime or a loss curve. It does not cover the broad, conceptual scientific discovery seen in fields like theoretical physics.

Second, the evaluation is computationally expensive. Because tasks are designed to run for hours, large-scale evaluations require significant infrastructure. Finally, the paper notes that performance is highly sensitive to the "harness." This is the specific software wrapper and system prompts used to facilitate the agent's interaction. The authors demonstrate that changing the harness can shift a model's score by as much as $\Delta = 0.43$. This suggests that much of the "intelligence" seen in current agents might be a byproduct of system engineering.

The verdict: A new requirement for agency

Is the era of the autonomous engineer here? Not quite. The AUTOLAB results suggest that while we have models capable of brilliant one-shot reasoning, we lack agents capable of sustained, disciplined labor. The gap between "knowing how to code" and "knowing how to research" is wide. It is characterized by a fundamental struggle with time management and empirical feedback loops.

However, the benchmark provides a necessary roadmap. To move toward truly capable autonomous agents, research must pivot. We must move from increasing parameter counts to improving "time awareness." Future agents must also improve their ability to navigate long-horizon trajectories. The full benchmark and evaluation harness are available at autolabhq/autolab. This provides the community with the tools to turn "reasoning engines" into actual "research agents."