Operation-Guided Progressive Human-to-AI Text Transformation Benchmark for Multi-Granularity AI-Text Detection

Most AI-text detection benchmarks operate on a binary assumption. They assume a document is either human-written or AI-generated. They look at a finished piece of text and assign a single label. But real-world workflows are different. People use LLMs to polish, paraphrase, or expand existing drafts. This creates a spectrum of "mixed-authorship" where AI involvement accumulates incrementally.

Current detection research largely ignores this progression. Existing benchmarks focus on static, final outputs. This leaves a blind spot regarding how AI signatures emerge or vanish during a revision cycle. This paper introduces OpAI-Bench to fill that gap. It reveals a counterintuitive reality: adding more AI edits does not always make a document easier to detect.

The failure of binary endpoint detection

The status quo in AI detection is built on static snapshots. Researchers collect human text and AI text. They then train a classifier to find the boundary. This works for distinguishing a raw LLM completion from a Wikipedia article. However, it fails to capture the nuances of co-editing. As shown in, naive approaches treat each version of a document as an independent entity.

This loses the historical context of how the text evolved.

Because existing benchmarks lack intermediate states, they cannot answer critical engineering questions. At what percentage of AI coverage does a detector become reliable? Which specific edit operations—like compressing a paragraph or paraphrasing a sentence—introduce the strongest statistical signals? Most importantly, they miss the "non-monotonic" trap. This refers to a pattern where detection performance does not move in a single direction. A document with 50% AI involvement might actually be harder to classify than one with 10% or 90%.

Building a cumulative revision trajectory

To solve this, the authors designed OpAI-Bench around a versioned trajectory. Instead of generating isolated samples, they take a human-written source ($V_0$) and build a sequence of nine versions ($V_0$ through $V_8$). The key choice is the cumulative construction method. Each version $V_t$ is created by editing the previous version $V_{t-1}$. This ensures the revision history is preserved.

The construction pipeline follows these steps: 1. Deterministic Shuffling: To ensure reproducibility, the authors use a fixed shuffle order for sentences. This order is seeded by the document ID. 2. Incremental Coverage: They increase the "AI coverage" (the fraction of sentences edited by an LLM) from 0% to 100%. Once a sentence is edited, it stays in the "edited" set for all subsequent versions. 3. Operation Diversity: The authors apply five distinct rewrite operations: polish, paraphrase, style rewrite, compress, and expand. 4. Multi-Granularity Provenance: They track authorship at four levels: document, sentence, token (individual words), and span (contiguous groups of characters). This helps locate exactly where the AI influence begins.

Non-monotonicity and the compression trap

The results from OpAI-Bench challenge the assumption that more AI content equals higher detectability. The authors report that detector performance is highly volatile. Specifically, they observe a significant performance dip in intermediate versions. This often happens around $V_4$.

Looking at and, document-level accuracy does not climb steadily.

Instead, it fluctuates based on the domain and the generator. The authors note that the $V_4$ version coincides with the "compression" operation. This version acts as a detection dead zone. In their controlled analysis, the authors found that compression is generally harder to detect than expansion or paraphrasing at the same coverage level.

The sentence-level results are equally telling. While fine-tuning on OpAI-Bench data improves stability, the "non-monotonic" pattern persists. LLM-as-detectors (using models like Claude or Gemini) perform well on early edits. However, they drop sharply during the compression phase. They only partially recover as the document approaches 100% AI coverage. This suggests that "mixed" text creates a statistical profile that mimics high-quality human writing. This effectively masks the AI signature.

Identifying the limits of the benchmark

OpAI-Bench has clear boundaries. First, the revision trajectories are constrained by predefined operations and coverage levels. Real human-AI collaboration is often unpredictable. A user might expand a section and then manually delete parts of it. This would break the cumulative logic used in the benchmark.

Second, there is a risk of "artifactual detection." The benchmark uses controlled XML-style delimiters to track spans. It also ensures sentence-level consistency through automated validation. The resulting text might possess subtle structural regularities. These regularities might not exist in organic human-AI collaboration. A detector trained on these "clean" transitions might struggle with real-world linguistic noise.

Finally, the benchmark is tied to specific domains like essays, news, reports, and abstracts. While this provides breadth, it does not guarantee universal success. A detector's success in recognizing an AI-polished essay may not translate to other formats.

The verdict: Move beyond the binary

If you are building AI detection systems, the takeaway is clear. Stop optimizing only for the endpoints. A detector that is 99% accurate at distinguishing raw LLM output from a textbook is insufficient. It will fail in professional workflows where users perform incremental assistance.

The OpAI-Bench results prove that the "middle ground" is a major challenge. The $V_4$ performance drop shows that current models may struggle with certain edit types. To ship a production-grade detector, you must evaluate it against these trajectories.

Is it worth prototyping this next week? Yes, if your product involves content moderation or academic integrity. The code is available at https://github.com/VILA-Lab/OpAI-Bench. If you rely on a simple binary classifier, test your sets through this framework. You will likely find that your "high accuracy" numbers hide vulnerabilities in mixed-authorship scenarios.

Figures from the paper