The Ceiling Is Low
Even the best public models pass only 66.7% of complex, multi-step business workflows. That is not a rounding error — it is a structural limit. Claw-Eval-Live, a new live benchmark, evaluates 13 frontier agents on 105 tasks sourced from public demand signals. The results expose a clear split: local workspace repair is near-solved for top models, but cross-service coordination remains deeply broken. If you are shipping agents that touch CRM, finance, or HR systems, assume they will fail roughly one-third of the time.
The Problem
Standard agent benchmarks suffer from two fatal flaws: staleness and superficial grading. Most suites, like WebArena or SWE-bench, freeze their task sets at publication. As tool ecosystems evolve, the benchmark drifts from reality. More critically, they mostly score the final output. An agent can generate a perfectly formatted report while silently skipping a required database update or editing the wrong file. This "fluent but wrong" failure mode is invisible to output-only graders but catastrophic in production.
Claw-Eval-Live addresses this by decoupling the signal layer (which workflows are relevant) from the snapshot layer (how we grade them). The benchmark constructs tasks from ClawHub Top-500 skills, ensuring the task mix reflects current user demand rather than author preference. Crucially, it grades via action-grounded verification. It inspects tool traces, service audit logs, and post-run workspace states. It does not trust the agent’s narrative; it trusts the observable evidence. This forces evaluation to measure executional closure, not just linguistic plausibility.
How It Works
The benchmark construction follows a rigorous five-stage pipeline, summarized in [Figure 1].
- Signal Collection: The pipeline ingests a timestamped snapshot of ClawHub Top-500 skills. These are not ground truth, but an externally inspectable prior for workflow salience.
- Pattern Clustering: Fragmented skill names are grouped into stable workflow patterns (e.g., "document transformation," "cross-tool coordination"). This removes surface noise while preserving evaluation-relevant distinctions.
- Family Weighting: Patterns are mapped to families, and weights $w_f$ are calculated based on upstream signal mass. This ensures the release proportionally represents high-demand areas.
- Seed Expansion & Screening: Weighted patterns expand into executable candidates with prompts, fixtures, and graders. Pilot runs filter out tasks that are too brittle or non-discriminative.
- Optimization-Based Selection: The final 105-task release is not manually picked. It is selected via a Mixed-Integer Linear Program (MILP) that maximizes the preservation of a pilot model ordering ($m_1 \succ m_2 \dots$) while ensuring coverage across all task families. This turns curation into an auditable constraint satisfaction problem.
Execution happens in controlled environments. Tasks split into two surfaces: service-backed workflows (87 tasks, e.g., CRM queries, email drafting) and workspace repair (18 tasks, e.g., shell commands, file edits). Grading is hybrid. Deterministic checks verify tool calls, entity accuracy, and state changes. Structured LLM judging (using GPT-5.4) is reserved only for semantic dimensions like coherence or completeness, and only when deterministic evidence is insufficient. This minimizes judge-model bias by anchoring the LLM in explicit traces.
Numbers
The leaderboard in Table 3 reveals the current capability ceiling. The top model, Claude Opus 4.6, achieves a 66.7% pass rate (70/105 tasks passed). GPT-5.4 follows at 63.8%. No model crosses the 70% threshold.
The disparity between execution surfaces is stark. Workspace repair tasks yield pass rates $\ge 72.2\%$ across all models, with top models nearing 100%. Service-backed workflows, however, cap at 59.8% for the best model. This confirms that local diagnosis is easy; coordinated, multi-system execution is hard.
Family-level heterogeneity hides in the aggregate. [Figure 4] shows that Development/Terminal tasks are near-ceiling, while HR/People tasks average <22.2%, with several models scoring 0.0%. Management tasks are effectively all-fail. Task discrimination is not uniform; [Figure 5] shows that the most discriminative tasks lie in the middle of the pass-count range. Easy tasks (all-pass) and impossible tasks (all-fail) provide little signal. The real differentiation happens in complex reconciliation and multi-document merge tasks, where partial tool usage collapses the score.
Resource efficiency varies widely. GPT-5.4 is the most efficient among top-tier models, using fewer tokens and less time than Claude Opus 4.6 while maintaining competitive accuracy. Claude Opus 4.6 incurs an estimated \$31.61 in API costs for the full suite, compared to \$6.27 for GPT-5.4.
Practical Implications
The paper demonstrates that action-grounded grading is essential for reliable evaluation. The benchmark’s grading logic resides in task-specific grader.py files. These scripts inherit from an AbstractGrader class that implements deterministic checks on tool dispatches and workspace state. Practitioners can replicate this approach by implementing similar verification steps in their own CI/CD pipelines.
Instead of accepting final text as proof of work, verify the audit logs. The benchmark proves that 66.7% is the current ceiling. Aim for higher determinism in your own grading to squeeze out the remaining gains.
What's Missing
The paper is honest about limitations, but practitioners should note three gaps:
- Judge Bias Risk: The authors use GPT-5.4 for semantic grading, and GPT-5.4 is also an evaluated model. While they mitigate this by grounding judgments in traces, there is inherent bias risk. A model might be rewarded for producing traces that align with the judge’s implicit preferences.
- Snapshot Stability: The benchmark is a time-stamped snapshot. The 105 tasks are fixed for this release, but the upstream signals drift. Quarterly refreshes are planned, but current users cannot rely on this specific task mix for long-term regression testing.
- Limited Family Coverage: Despite 105 tasks, the long tail of specialized workflows is thin. The top four families (PRODAPP, SHELL, HR, SALES) account for 45 tasks. Complex, niche enterprise workflows may not be represented, limiting the benchmark’s generalizability to highly regulated or specialized industries.