CaliPPer: quantifying, predicting and improving AI model performance for binding prediction

CaliPPer: A Post-Hoc Framework to Quantify and Improve AI Binding Prediction Reliability

AI models that predict how antibodies or T-cells bind to targets often fail when they see something new. In immunology and drug discovery, these models accelerate identifying therapeutic candidates. However, their performance on novel epitopes (the specific part of an antigen recognized by the immune system) or antigen variants is notoriously unpredictable. This unpredictability leads to high failure rates in the lab. Every failed binding validation consumes expensive reagents and precious bench time.

Until now, the industry has lacked a way to gauge how much to trust a model's prediction on a specific, unseen target. Some methods estimate aggregate performance on new datasets. However, they typically provide only a single, blunt number for the entire cohort. There has been no effective way to move from "how well does this model work on this dataset?" to "how much should I trust this specific candidate?"

A new paper introduces CaliPPer (Calibration and Prediction of Performance). This framework bridges that gap. It does not require retraining your model. Instead, it acts as a post-hoc (applied after the fact) triage layer. It uses the distance between a new sample and the training data to predict performance and recalibrate confidence scores.

The failure of aggregate-only estimation

Current approaches to estimating model performance on out-of-distribution (OOD) data generally rely on density-ratio methods like PAPE or M-CBPE. These methods re-weight predictions based on how likely they are under the test distribution. However, these techniques suffer from two fatal flaws in biological binding prediction.

First, they assume "support overlap." This is the idea that the training and test data cover the same sequence space. In reality, novel epitopes occupy regions of sequence space the model has never visited. Second, they assume the "covariate-shift-only" condition. This assumes the underlying rules of binding remain constant. For new antigen variants, the mapping from input to output can shift. The authors report that these failures cause density-ratio predictions to collapse toward a single average value. They fail precisely on the novel targets that matter most for drug discovery. Furthermore, existing methods are aggregate-only. They cannot tell a screening team which specific molecules in a library are high-priority.

Anchoring performance on S2DD

CaliPPer solves this by anchoring performance estimation on a biologically informed distance metric called Sample-to-Domain Distance (S2DD). Rather than relying solely on density ratios, CaliPPer treats performance as a function of how far a sample lies from the training distribution in sequence space .

The mechanism works in three distinct layers:

1. Multi-chain S2DD Calculation: Immune receptors involve multiple interacting components. Therefore, a single-chain distance is insufficient. S2DD calculates distances for each chain (such as the peptide, the $\alpha$-chain, and the $\beta$-chain) using modular base metrics. These include Levenshtein edit distance (a measure of string similarity) or BLOSUM substitution scores. It then applies a weighting scheme ($\sigma \cdot C$) that emphasizes chains with concentrated training distributions. This ensures that if a new peptide is highly novel, that signal dominates the final distance score.
2. Performance Degradation Curves: The framework fits a mathematical curve to the relationship between S2DD and observed performance on a labeled calibration set. This characterizes how much a model's reliability drops as samples become more distant from the training data .

3. Bayesian Recalibration: At the finest resolution, CaliPPer performs per-sample triage. It estimates distance-dependent Positive Predictive Value (PPV; the probability that a positive prediction is correct) and Negative Predictive Value (NPV; the probability that a negative prediction is correct). It then uses a closed-form Bayesian update to adjust raw model scores. This process pulls scores toward the prior prevalence when the model becomes uninformative at high distances. This effectively re-ranks candidates based on local reliability rather than raw probability .

Predictable decay and improved discovery

The authors validate CaliPPer across ten models and eight architectures. Their work covers both TCR and BCR domains. The results suggest that the relationship between sequence distance and model failure is remarkably systematic. The paper reports that across all tested models, performance declined predictably with increasing S2DD distance. They achieved distance–performance correlations $|r|$ between 0.80 and 0.92 .

For engineers assessing utility, the authors measure mean absolute errors (MAE) for aggregate performance. These errors for AUROC, AP, or F1 range between 0.008 and 0.070 .

This low error means the framework can reliably forecast how a model will behave on a new dataset before you run experiments. Crucially, the framework does not just predict failure; it mitigates it. The authors demonstrate that distance-aware Bayesian recalibration can improve AUROC by up to +0.20 on unseen epitopes or variants .

The most striking evidence comes from retrospective applications to five published studies. In these real-world scenarios, CaliPPer improved True Discovery Rates (TDR; the fraction of top-ranked candidates that are actually correct) in every instance. For example, in a TCR neoantigen study, the raw model's top-5 candidates yielded zero confirmed hits. After CaliPPer recalibration, the TDR rose from 0/5 to 3/5 .

Implementation constraints and edge cases

While the results are strong, there are clear operational requirements to consider.

First, CaliPPer is not "zero-shot." It requires a labeled calibration cohort. The authors specify that you need at least 30 calibration samples per distance bin. You also need at least 8 samples from the minority class to fit reliable curves. If your validation set is tiny or extremely imbalanced, the curve fitting will be unstable.

Second, the framework relies on the calibration cohort "bracketing" the query. If you are testing a target that is significantly more distant from the training set than anything in your calibration set, reliability decreases. In this case, the model must extrapolate the degradation curve.

Finally, the effectiveness of the recalibration depends on the calibration cohort. It must exhibit a non-flat PPV/NPV profile. If your calibration data is too uniform or stays entirely within a "plateau" region, CaliPPer effectively reduces to standard Platt scaling. This means it will provide no additional re-ranking benefit.

The verdict: A necessary triage layer

CaliPPer is a pragmatic tool for moving from model training to model deployment. It recognizes that "accuracy" is a moving target. That target depends entirely on the distributional distance of the query.

If you run a high-throughput screen where the cost of a false positive is high, this is worth a prototype. The framework is model-agnostic and post-hoc. You can wrap it around your existing inference pipeline without changing your model weights. The modularity of the S2DD distance makes it versatile. It can support everything from Levenshtein to molecular fingerprints for both protein and small-molecule workflows.

The code is reportedly available at https://github.com/jianqingzheng/calipper. If you have a labeled validation set ready, you can implement the distance-aware re-ranking quickly. This can help determine if your model's "top hits" are actually reliable.

Figures from the paper