On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

LLM Annotation Reliability Limited by Internalized Priors and Decision Stickiness

When using AI to label data, the AI's own pre-existing ideas often override the instructions you give it. Even if you provide a better definition or more examples, the AI often refuses to correct its initial mistakes. This is especially true if it is very confident in them.

In the current production landscape, LLMs are increasingly deployed for zero-shot annotation (labeling without prior examples) and "LLM-as-a-judge" tasks. The prevailing engineering assumption is that the user’s prompt acts as a control channel. If a model misses a nuance, you refine the rubric or inject few-shot examples (providing a few completed examples) to steer the behavior. We treat the prompt as the governing logic.

However, this assumes LLMs are blank slates. In reality, they arrive at the task with massive, internalized priors (pre-existing beliefs) shaped by web-scale pre-training and RLHF (Reinforcement Learning from Human Feedback). This paper investigates the friction between these internal concepts and user-provided definitions. It turns out that for many annotation tasks, the model's internal "idea" of a concept like toxicity is much harder to move than we thought.

The failure of prompt-based steering

The status quo in LLM annotation relies on the belief that steerability (the ability to shift model behavior via interventions) is a reliable lever. If a model fails to identify a specific type of hate speech, the standard playbook is to expand the definition in the prompt. Alternatively, you might add gold-standard examples.

As shown in, the researchers probe the limits of this approach.

They look at how familiarity and alignment interact. They find a fundamental breakdown. Prompting is surprisingly good at consolidating answers the model already gets right. However, it is remarkably poor at fixing errors. This isn't just a minor inefficiency. It is a structural limitation of the prompt-based control loop. When a model makes a mistake in a zero-shot setting, that mistake is often "sticky."

Measuring concept alignment via DSF

To quantify this disconnect, the authors move beyond simple text-matching. They introduce Definition-Specific Familiarity (DSF). Most engineers might try to measure if a model "knows" a dataset by checking for text memorization. This means seeing if the model has seen the specific strings in its training set. The authors argue this is the wrong signal for annotation.

Instead, DSF measures the semantic alignment between a model's internalized concept and the target definition. The mechanism works as follows: 1. The model is prompted to explain its own understanding of a concept. 2. This self-description is converted into a vector (a mathematical representation of meaning) using a sentence encoder. 3. The authors calculate the cosine similarity (a measure of the angle between two vectors) between this self-description and the actual task definition.

To prevent the results from being biased by a single mediocre encoder, they use a "consensus DSF." This is the unweighted mean across six diverse sentence encoders. This includes MiniLM and OpenAI’s text-embedding-3-small. This ensures the metric captures genuine conceptual alignment. It avoids capturing mere artifacts of a specific embedding model's quirks.

Stubborn errors and the confidence trap

The empirical results are sobering for anyone building automated labeling pipelines. The authors report an overall "rescue rate" of only 34.8%. This is the fraction of initial errors corrected by prompting. This means nearly two-thirds of zero-shot errors are completely resistant to being fixed. Better prompting or few-shot examples often fail to help.

The most dangerous failure mode is "decision stickiness." The paper finds that the higher the model's initial confidence, the less likely it is to be corrected. As illustrated in, there is an inverted-U relationship.

Rescue probability peaks at moderate confidence (0.6–0.7). However, it collapses to just 20.8% for errors where the model was more than 90% sure it was right.

Even more concerning is the "critical calibration failure" revealed in . When given misaligned or outright incorrect definitions, models do not signal confusion. They follow the wrong instructions with the same high confidence levels they exhibit when following the correct ones. This means you cannot use a model's self-reported confidence as a proxy to detect definition errors. The models cannot distinguish between aligned and misaligned instructions via confidence scores.

Limitations of the study

While the findings are robust, there are gaps that a practitioner should note. First, the study focuses heavily on binary classification (toxic vs. not toxic). It is unclear if these "sticky" decisions manifest differently in multi-class problems. They may also behave differently in complex span-labeling tasks.

Second, the authors cannot disentangle the source of these priors. We don't know if this anchoring comes from raw pre-training data. We also do not know if RLHF has effectively "hard-coded" certain definitions. Finally, the study treats prompting as the primary control channel. It does not explore whether more heavy-handed interventions, like fine-tuning (updating model weights on specific data), could bypass these internalized priors.

The verdict

If you are planning to deploy an LLM-based annotation pipeline, don't bet on the prompt to save you from a bad model-task match.

The evidence suggests that selecting a larger model or spending weeks on prompt engineering is less effective than ensuring conceptual alignment. The authors demonstrate that DSF is a much stronger predictor of performance ($r = +0.41$) than text memorization.

Actionable takeaway: Before running a million-token annotation job, run a small pilot to calculate the DSF for your model-task pair. If the alignment is low, no amount of few-shotting will likely rescue the tail latency of your error rate. Code and analysis pipelines are available; see the paper for the canonical link.

Figures from the paper