Auditing Differential Visibility of Political Content on TikTok

Auditing TikTok: How Improper Analysis Units Manufacture False Claims of Political Shadow Banning

Researchers recently investigated whether TikTok suppresses political content. This phenomenon is commonly called "shadow banning" (the quiet throttling of a user's reach without notice). While many advocacy groups and creators have alleged that certain political stances face systemic suppression, this study suggests that many existing "proofs" are actually mathematical artifacts. The researchers found that while conventional analyses appeared to show massive reach gaps, these results vanished when the data was analyzed at the correct unit of observation. Instead of suppression, the data revealed a different trend: certain types of political content actually earn significantly higher engagement per view.

The failure of pooled observations

The core problem in current algorithmic auditing is the choice of the "unit of analysis" (the specific level at which researchers group data to draw conclusions). Most existing audits of political visibility rely on "pooled" comparisons. In these designs, researchers take thousands of individual data points and group them by political stance. They then look to see if one side has a lower average reach.

As the authors demonstrate, this approach is fundamentally broken when dealing with time-series data. Because a single video generates multiple hourly snapshots, these observations are highly autocorrelated (meaning the data points are highly dependent on previous values). Treating these near-duplicate snapshots as independent pieces of evidence leads to a massive inflation of statistical significance.

This is illustrated in .

It shows that as the unit of analysis expands from the account level to the "video-hour" level, the $p$-value (the probability that the observed result occurred by chance) collapses toward zero. This creates the illusion of a definitive discovery. In one instance involving Israel/Palestine content, the pooled analysis reported a staggering $p < 10^{-147}$. This number suggests absolute certainty. However, it is actually a product of "pseudoreplication" (counting dependent observations as if they were independent).

A hierarchy of independence

To rectify this, the authors implement a rigorous hierarchical auditing framework. Rather than treating every timestamp as a new piece of information, they prioritize the account as the fundamental unit of independent observation. This is because the "stance" (the political position) is assigned to the account itself, not to individual seconds of video playback.

The methodology follows a strict sequence: 1. Curated Sampling: The researchers select 67 accounts across three topics (U.S. immigration, Trump coverage, and Israel/Palestine). They ensure a stratified mix of follower counts and political stances. 2. Dense Panel Collection: Using a third-party API wrapper, they collect hourly snapshots of reach (views per 1,000 followers) and engagement (likes, shares, saves, and comments) for 2,753 videos. 3. Ensemble Classification: A three-model LLM ensemble (using GPT-4o, Gemini, and Claude Sonnet) filters the videos to identify which are "on-topic." This ensures the sample is not polluted by irrelevant content. 4. Randomization Inference: To avoid relying on large-sample assumptions, the authors use permutation testing. They shuffle the stance labels across the 67 accounts 20,000 times to create a "constructed null" distribution. This allows them to see how often a random arrangement would produce the observed gap.

By anchoring the analysis to the account, the researchers ensure that the statistical power is derived from actual independent entities.

The vanishing reach gap

When the authors apply this corrected lens, the "shadow ban" narrative evaporates. The paper reports that at the account level, every single reach contrast for the three topics was null after multiple-testing correction ($q \approx 0.9$). Specifically, for the Trump and Israel/Palestine topics, the effect sizes ($\delta$, a measure of rank difference) were essentially zero.

However, the study does find a robust, statistically significant signal in a different metric: engagement-per-view. The authors report that "oppositional" content—defined as anti-Trump or pro-Palestine—actually earns significantly more reactions per view than "establishment" content. For example, on the Trump topic, oppositional content showed a Cliff's $\delta$ of $-0.51$ ($q = 0.025$). For Israel/Palestine, it was $-0.64$ ($q = 0.021$).

As shown in, this asymmetry is particularly pronounced in "effortful" reactions like shares and saves.

Comments show little difference. This finding is crucial. It provides an alternative explanation for why certain content might appear suppressed. If one side of a debate has a much more intense, active audience, their content may circulate differently. This is a function of audience mobilization rather than algorithmic demotion.

Confounding and the limits of observation

It is important to recognize what this paper does not prove. The authors state that a null result in an observational study is not a definitive proof of platform neutrality. It is merely an absence of evidence for the specific magnitude of suppression they tested.

There are two major caveats to consider: First, the study is limited by its reliance on "follower-normalized reach" as a proxy for visibility. Because the researchers do not have access to TikTok's internal ranking logs, they cannot perfectly separate algorithmic demotion from organic audience behavior.

Second, significant confounding variables complicate the interpretation of the raw data. As highlighted in, the "pro-Palestine" side in this corpus was significantly larger in terms of median follower count.

They also posted primarily in Arabic. In a poorly designed pooled analysis, these differences in language and audience size can mimic the signal of a shadow ban. The authors mitigate this by showing that even when restricting the analysis to English-only content, the reach gap remains null. Meanwhile, the engagement gap persists .

The verdict: Audit with care

The verdict is clear: The claim of systemic political shadow banning on TikTok is not supported by this data when analyzed using sound statistical principles. The "evidence" cited by many observers is likely a byproduct of improper scaling and the failure to account for autocorrelated data.

If you are an analyst or a policymaker, remember that a high $N$ (number of observations) is a trap if those observations are not independent. A study with 500,000 rows of data is significantly weaker than a study with 50 independent accounts if those rows are just hourly updates of the same few videos. For a visibility audit to be credible, it must report the design effect and analyze at the level of the independent unit—the account. Without this, you aren't measuring platform policy. You are just measuring the heartbeat of a single video.

Figures from the paper