CEO-Bench: Can Agents Play the Long Game?

Most current AI agent evaluations suffer from a temporal myopia (a tendency to focus only on the immediate future). We have become very good at measuring how well an agent can execute a discrete, short-horizon task. These include fixing a bug, navigating a website, or resolving a customer service ticket. However, we have almost no way to measure how an agent performs when its actions today fundamentally reshape the environment of tomorrow.

The field of AI agency is currently dominated by benchmarks like SWE-bench or WebArena. These focus on "local" task completion. The agent receives a goal, acts for a short time, and receives immediate feedback. Real-world intelligence, particularly in strategic roles, is defined by the ability to navigate long horizons under uncertainty. This involves interpreting noisy signals, managing depleting resources, and adapting to shifting market conditions. Until now, we lacked a sandbox that could test whether an agent can actually "play the long game."

The researchers behind CEO-BENCH have filled this gap with a high-fidelity startup simulation. Instead of fixing a single line of code, the agent operates a fictional company, NovaMind, for 500 simulated days. The results are a sobering reminder of the gap between tactical proficiency and strategic orchestration. Most state-of-the-art models simply cannot survive the simulation without going bankrupt.

The failure of short-horizon metrics

Current agentic evaluations typically treat the environment as a series of isolated episodes. This approach fails to capture the compounding nature of real-world decisions. In a business context, a decision to slash prices might look like a "success" in a short-term metric. Increased sales might appear positive initially. However, if it erodes brand reputation or triggers a price war, it becomes a catastrophic failure in the long term.

As shown in, running a startup requires coordinating a massive web of interdependent moving parts.

Pricing affects growth. Growth stresses infrastructure. Infrastructure reliability affects customer satisfaction. Satisfaction drives reputation. Most existing benchmarks ignore these feedback loops. Because they lack "delayed and coupled consequences," they cannot distinguish between an agent that follows instructions and an agent that steers a system toward a coherent, long-term goal.

A programmable engine for business complexity

To move beyond static tasks, the authors developed a mechanistic simulator (a model based on explicit physical or logical rules). It prioritizes cause-and-effect stability over LLM-generated "vibes." The architecture of CEO-BENCH relies on several key layers:

1. The Pythonic Interface: The agent interacts with the world through novamind_api. This is a programmable Python package. This allows the agent to go beyond simple tool calls. It can write its own scripts to perform complex data analysis or automate multi-step workflows, as seen in .

2. Mechanistic World Rules: The simulator uses explicit mathematical models to drive the environment. It uses Poisson distributions (a mathematical model for counting events occurring at a constant rate) for customer acquisition. It also uses Ornstein–Uhlenbeck processes (a model for mean-reverting stochastic processes) for macroeconomic cycles. This ensures the world behaves consistently across different runs.
3. Partial Observability: The agent does not have access to the "ground truth" of the simulation. It cannot see actual customer satisfaction scores. It cannot see the exact willingness-to-pay parameters of a market segment. Instead, it must act as a true CEO. It must mine a 19-table business database via SQL and monitor a noisy social media feed to infer the latent (hidden) state of the market.
4. The Feedback Loop: Every action has a cost and a potential consequence. Spending on R&D might not yield a quality improvement for weeks. An aggressive marketing campaign might saturate a market segment. This makes future acquisition more expensive.

Survival is the ultimate metric

The authors evaluate a wide array of models. These range from frontier closed-weight models like GPT-5.5 and Claude Opus 4.8 to open-weight models like DeepSeek V4 Pro. The primary metric is "cash on hand" at the end of the 500-day period. Bankruptcy serves as a hard terminal state.

The results, summarized in, reveal a stark hierarchy. Most models—including highly capable ones like Gemini 3 Flash and Claude Sonnet 4.6—fail to complete the simulation. They hit a $0 balance and go bankrupt. Only two models, Claude Opus 4.8 and GPT-5.5, successfully finish the simulation with a balance above the $1M starting capital. Even these leaders struggle with consistency. Neither model consistently turns a profit across all experimental runs.

Furthermore, the paper reports that the best-performing models exhibit qualitatively different behaviors. Stronger agents engage in "sophisticated analytics." For example, they write code to simulate customer cohorts (groups of customers who share common characteristics) for forecasting. They also mine negotiation histories to uncover hidden preferences. Weaker models often default to passive, ineffective cash-preservation strategies.

Constraints of the synthetic boardroom

While CEO-BENCH is a significant leap forward, it is important to recognize its limitations. The simulation is an abstraction. Several limitations persist:

• Numerical vs. Qualitative Change: The authors could not find a reliable way to evaluate qualitative product shifts. Therefore, "product quality" is represented as a single numerical value. In the real world, a CEO makes qualitative pivots. This simplification may miss the nuances of true product innovation.
• Simplified Scope: To keep the simulation computationally and economically feasible, the authors excluded certain dimensions. They left out legal compliance, cybersecurity, and fundraising. A real CEO spends significant cognitive load on these activities.
• The "Simulator Gap": The mechanistic rules are robust, but they are still approximations. There is a risk that an agent might learn to exploit specific mathematical quirks. It might optimize for the simulator's specific math rather than learning generalizable strategic principles.

Verdict: A necessary new yardstick

Is CEO-BENCH ready for prime time? Yes. Is it a perfect proxy for human intelligence? Not yet.

The paper demonstrates a massive blind spot in our current evaluation pipeline. We are building agents that are excellent at "doing" but struggle with "steering." If the goal of agentic AI is to move from digital assistants to autonomous operators, we need to change our metrics. We must stop rewarding agents solely for how many tasks they complete. We should instead reward them for how much value they sustain over time.

For practitioners, the takeaway is clear. If you are developing agents for roles involving resource allocation or long-term planning, traditional benchmarks may give you a false sense of security. Your agent might pass a coding test with flying colors. However, it may collapse the moment it has to manage a budget. CEO-BENCH provides a framework for catching that failure before it happens in production.

Figures from the paper