Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025

Who Annotates in NLP? A Large-Scale Audit of Human Annotation Reporting (2018–2025)

In the development of NLP systems, human annotation is the empirical foundation. Whether we are building datasets or evaluating model outputs, we rely on human judgments as the ground truth. Researchers often assume this ground truth is a settled fact. However, the validity of that truth depends on who did the work, how they were trained, and how disagreements were resolved.

Historically, the community has lacked a quantitative way to measure how well we document this process. We knew that annotator bias or poor instructions could undermine a study. Yet, we did not know if the field was actually improving. This paper changes that by conducting the first large-scale, task-level audit of human annotation reporting across major NLP venues from 2018 to 2025. The authors found a systemic transparency gap. While we report how many people worked, we fail to report critical methodological details like compensation, training, and socio-demographics.

The Problem

The status quo in NLP research is a "black box" approach to human labor. Most papers report the what (the dataset) and the result (the model performance). They often gloss over the how. When a paper claims a model performs well on a sentiment task, that claim is only as robust as the annotators who defined "sentiment." If those annotators were not trained, were not compensated fairly, or represented a narrow demographic, the entire benchmark might be biased.

Current reporting is heavily skewed toward operational logistics. As shown in, authors are quite good at reporting recruitment strategies (90.4%), expertise levels (86.5%), and total item counts (86.0%).

These are basic logistical facts. However, the details required to assess the validity of the judgment itself are often missing. Critical metrics like annotator training (18.7%), language proficiency (24.0%), and the release of actual annotation guidelines (34.1%) are frequently omitted. This means we can see the scale of an experiment, but we cannot verify its integrity.

How It Works

The authors move away from "paper-level" analysis to "task-level" analysis. A single paper might contain multiple distinct annotation tasks. For example, one task might build a dataset while another evaluates a model. Each task may have different annotators and protocols. Treating the paper as the unit of analysis obscures these vital differences.

The methodology follows a structured pipeline:

1. Taxonomy Development: The researchers built a unified taxonomy covering seven aspects .

These include general description, agreement (how much annotators agreed), workload, recruitment, compensation, socio-demographics, and quality control. 2. Gold Standard Creation: They manually annotated a high-quality set of 41 papers (72 tasks) called ANNOTATEDGOLD. This served as the ground truth for testing automation. 3. LLM-Assisted Extraction: To scale the audit, they developed a pipeline using structured-output APIs. These APIs force Large Language Models (LLMs) to populate the taxonomy precisely. They used "annotation-section chunking" to handle long documents. This involves identifying sections like "Human Evaluation" via regex (regular expression pattern matching) to prevent context loss. 4. Automated Auditing: They applied this pipeline to a massive corpus of 1,603 papers from ACL venues (ANNOTATEDLLM) to perform a meta-analysis of reporting trends.

To quantify completeness, they introduced the REPORTAGE SCORE. This metric calculates the ratio of reported attributes to the set of attributes that should be reported for a specific task type.

Numbers

The most striking result is the discrepancy between different types of research. The authors report that resource creation studies (building datasets) are significantly more transparent than model evaluation studies .

While resource creators document their methodology thoroughly, those performing model evaluations consistently under-report recruitment, compensation, and quality control. This suggests a risk of bias in the benchmarks used to claim model superiority.

Regarding the automation of this audit, the results are robust. The authors report that their best-performing model, Gemini-3.1-Pro, achieved a Krippendorff’s $\alpha$ of 0.606. This is a coefficient used to measure inter-rater agreement. This actually slightly outperforms the human-human agreement ($\alpha = 0.585$) measured on the gold standard [Table 2]. This indicates that LLMs can reliably extract structured metadata from scientific prose at a scale humans cannot match.

Finally, the paper examines the impact of the 2022 ACL Responsible NLP Checklist. The interrupted time-series analysis in shows that reporting quality has improved over time.

However, the checklist did not trigger a sudden, massive spike in transparency. Instead, it seems to have acted as a standardizing force. It caused different venues to converge toward a similar level of reporting quality.

What's Missing

The paper is a thorough audit, but it has gaps. First, the large-scale analysis relies on keyword-based retrieval. While the authors validated this against a random sample and found deviations to be modest, the corpus is inherently biased.

It targets papers that explicitly mention "annotation" or "evaluation." It may miss nuanced studies that use human judgment without those specific keywords.

Second, the taxonomy requires significant interpretation. The authors admit that some categories depend on "informed judgment" when a paper's description is ambiguous. This means the "truth" being measured is partially a reflection of the researcher's interpretation.

Lastly, the paper focuses on reporting rather than actual practice. A low reportage score tells us a paper is opaque. It does not prove the researchers skipped training or failed to pay workers. It only proves they did not document those actions.

Should You Prototype This

If you are building internal tooling to monitor data quality or compliance, the answer is yes. The extraction pipeline and the taxonomy are highly actionable. You do not need to wait for a "perfect" model. The fact that Gemini-3.1-Pro hits human-comparable agreement means you can use this today. You can audit your own internal documentation or vendor reports.

However, do not look for a plug-and-play solution to "fix" your research papers. This is not a tool for generating content. It is a diagnostic tool for measuring transparency. Use the taxonomy as a checklist for your own team. Ensure that when you ship a new benchmark, you are not leaving the most important methodological details in the dark.

Figures from the paper