Technology Shocks, Relative Performance Measures, and Outcomes: Evidence from Classical Chess

When Absolute Gains Vanish Into Relative Metrics

When powerful new AI chess tools became free in 2020, chess players globally began drawing much more often. Paradoxically, even though players' absolute playing strength improved, their official Elo ratings stayed largely the same. This happens because ratings are a relative measure. They are built from results against other rated players rather than an absolute scale of quality. If everyone improves simultaneously, the relative hierarchy remains static. This masks the underlying technological leap.

This paper investigates this disconnect using a massive dataset of 3.9 million classical games. It explores how a massive technological shock—specifically the release of neural-network-based chess engines—can fundamentally alter the outcome distribution of a system without shifting its primary performance metric.

The blind spot in Elo ratings

Standard performance tracking in competitive environments relies on relative ranking. In chess, the Elo system assigns a numerical rating based on the difference between players. A 200-point gap implies a specific expected score (the probability of a win plus half the probability of a draw). However, the system is agnostic to the absolute level of play. As seen in [, Panel B], the distribution of player ratings remained remarkably stable across the Covid interruption.

The median shifted only slightly from 1951 to 1928.

The problem is that the Elo system does not independently determine the win-draw-loss probabilities. It only tracks the expected score. If a technological advancement causes players to convert more wins and losses into draws, the expected score might remain nearly identical. Consequently, the rating system sees no "drift." This happens even though the nature of the competition has changed. This creates a measurement failure. The metric reports stability while the underlying system undergoes a structural phase shift.

Mapping the outcome surface shift

To detect this invisible shift, the authors move away from simple rating tracking. Instead, they model the entire "outcome surface." This is the probability of a White win, Black win, or draw as a function of both average rating and rating difference. They employ penalized tensor-product splines (a flexible mathematical method for fitting smooth curves to data) to estimate these probabilities.

The methodology follows a rigorous progression to isolate the signal from noise: 1. Surface Estimation: They estimate separate draw, White-win, and Black-win surfaces for the pre-Covid and post-Covid periods. 2. Composition Decomposition: Using an Oaxaca–Blinder-style decomposition (a method to separate effects of group composition from actual changes), they separate the change in mean draw rates. They split this into a "composition component" (changes caused by a shift in the rating distribution) and a "residual conditional shift" (actual changes in how players at a given rating perform). 3. Relabeling: They attempt to reconcile the two surfaces. They seek a linear transformation, $f(R) = a + bR$, that maps post-Covid ratings to their pre-Covid equivalents.

As shown in [.2], the post-Covid era exhibits a clear upward shift in draw probabilities.

This shift occurs across the entire grid of average ratings and rating differences.

A 4% jump in draws

The authors report a game-weighted mean increase in draw probability of 4.3 percentage points. This means that in the total pool of games, draws became significantly more common. Crucially, the decomposition in [Table 3.4] reveals that the "composition component" accounts for only 0.002 of this increase. Almost the entire shift is a "residual conditional shift." This means the change is happening at the level of individual matchups, not because different types of players are playing more often.

The relabeling results are particularly striking. The authors find that a linear transformation accounts for more than 90% of the shift in all three outcome surfaces. Specifically, the transformation $f(R) = 559.9 + 0.792R$ effectively maps the new reality onto the old one. This implies that a post-Covid rating of 2000 corresponds to a pre-Covid rating of 2144. Furthermore, [Table 4.1] demonstrates that this "relabeling gap" is non-linear. It is larger for lower-rated players (a 206-point gap at 1700) than for higher-rated players (a 144-point gap at 2000). This suggests that the technological shock provided disproportionately larger absolute gains to lower-rated players.

Unobserved behavioral channels

While the correlation between the timing of the Stockfish 12 release (September 2020) and the draw rate spike is compelling, the paper has clear boundaries. The authors admit they do not observe actual moves, opening choices, or direct engine usage. They can document the effect of the technology, but not the mechanism.

There are three specific gaps that prevent this from being a definitive causal proof: 1. Intensity vs. Quality: The data cannot distinguish between "extensive" use (players using engines more frequently) and "intensive" use (players getting more profound insights from the same amount of engine time). 2. Confounding Behavioral Shifts: The Covid interruption coincided with a massive shift to online play. The authors cannot definitively rule out that changes in risk-taking, stamina, or preparation habits contributed to the increased draw rate. 3. Selection Bias: While the "repeated same-color pairs" sample helps mitigate this, they cannot fully prove that the players who returned to over-the-board play are representative of the pre-Covid population.

Verdict: Watch the outcomes, not the ranks

The paper is a vital warning for anyone managing relative performance systems. This applies to Elo ratings, employee promotion tracks, or supplier benchmarks. If a tool improves the baseline performance of an entire cohort, your relative metrics will report zero progress. Meanwhile, your absolute capabilities skyrocket.

Is this worth a prototype for your own monitoring systems? Yes, if you operate in a domain where "absolute quality" matters more than "relative standing." If you see stagnant rankings despite increasing complexity, stop looking at the leaderboard. Instead, start modeling the distribution of outcomes. Look at error rates, completion types, or success modes. The "relabeling" effect proven here is a mathematical certainty in any shared-improvement environment. Don't let a stable rank hide a fundamental shift in your system's capability.

Figures from the paper