Progressive Disclosure for LLM-Maintained Wiki Knowledge Bases: a Preregistered Ablation

A preliminary pilot study recently upended a fundamental assumption in AI development: the idea that making a digital library easier to browse primarily saves money by avoiding large index loads. When researchers tested this "progressive disclosure" intuition—the practice of providing compact summaries instead of massive datasets—they discovered that the actual mechanism of cost savings was entirely different than expected.

By testing this on a real 709-page markdown wiki, a study by Theodore Cochran at AI for Altruism (A4A) reveals that savings are driven not by avoiding a large index, but by enabling more surgical, targeted interactions with the data.

The failure of the index-loading intuition

In many Retrieval-Augmented Generation (RAG) systems—architectures that provide Large Language Models (LLMs) with external knowledge—engineers try to balance "recall" (finding the right information) against "context length" (the amount of text fed to the model). The common design instinct is to build a dense index that acts as a map. The hope is that a "leaner" access structure, such as a table of contents or per-document abstracts, will allow an agent to navigate the corpus without paying the "tax" of reading a massive, centralized directory.

As shown in the study's setup, the researchers wanted to see if restructuring a corpus for progressive disclosure would yield these promised savings.

However, the pilot revealed a mismatch. Highly capable, tool-using agents often ignore the index entirely. These models use reasoning to infer which files they need and jump straight to them. Consequently, the "savings" intended to be captured by eliminating the index load were never actually there to be saved.

A surgically precise access structure

To move past this mismatch, the authors designed a rigorous ablation study. An ablation is a scientific method where researchers systematically change specific components of a system to see how much each part contributes. The researchers froze a 709-page wiki as immutable git tags. This ensured the actual content of the pages remained byte-identical across all tests. Any change in performance had to be caused by the access structure—the way the agent finds information—rather than the information itself.

The study tested four distinct "arms" or versions of the knowledge base: 1. A0 (Baseline): The original corpus with a full, monolithic index. 2. A1 (State Extraction): A slimmer index where volatile "recent activity" data was moved to a separate file. 3. A2 (Summaries): The addition of a one-line summary for every page in the frontmatter (metadata at the start of a file). 4. A3 (Retrieval Tool): The introduction of a specialized Python tool (wiki_query.py) that returns keyword-ranked "gists" (brief snippets containing titles and summaries).

These arms were crossed with three different "access regimes" to simulate different levels of agent autonomy: a protocol-constrained agent, a free self-routing agent, and a "forced" regime where the entire catalog is preloaded into the prompt.

Evidence of targeted navigation

The results challenge the original hypothesis while confirming the practical utility of the retrofit. Using a Claude Opus 4.8 model to generate answers, the researchers found that the A3 retrieval arm achieved "non-inferior" quality compared to the A0 baseline. Specifically, the quality composite (a score from 0–8) for A3 was 6.32 compared to 6.30 for A0 [Table 1]. This stayed well within the pre-set margin of error ($\delta = 0.5$). While there was a minor dip in correctness for complex aggregation questions, this was offset by better citation validity and less "padding" (unnecessary verbosity).

The most striking findings relate to cost. Even though the agent didn't save money by skipping the index, the A3 arm was significantly cheaper in every regime. As illustrated in, the cost reduction is substantial.

It reached approximately 30% for enforced agents and 34% for free agents. In the forced catalog-preload regime, the saving was a massive 58%.

The authors conclude that the mechanism for these savings is behavioral. The A3 arm leads to more "targeted access." The agent cites fewer pages, dropping from 6.10 to 4.22. It also performs fewer tool turns. The efficiency comes from an agent that knows exactly where to look and stops reading once it has the answer.

Limitations in grading and scope

Despite the clear cost benefits, the study highlights challenges in how we evaluate these systems. The researchers utilized an "LLM-as-a-judge" approach. They employed an OpenAI GPT-5 model to grade the outputs of the Claude Opus 4.8 system. While this cross-family grading is a strong defense against model bias, the study's own reliability audit fell short. The researchers report that their human-rater agreement (measured by Cohen’s $\kappa$) did not meet the pre-registered threshold of 0.60.

This discrepancy arose because the human grader and the AI judge disagreed on subjective dimensions like "citation" and "padding." The human viewed thorough, link-heavy responses as helpful. Conversely, the AI judge penalized them as unnecessary verbosity. Furthermore, the study's findings are limited to a single corpus and a single model pairing. It remains unclear if these patterns hold across vastly different domains or with less sophisticated "weaker" models.

The verdict: restructure for precision

If you are building an LLM agent to navigate a large internal wiki, the verdict is a clear yes. Implement progressive disclosure, but do so to enable better retrieval, not just to shrink your index.

The research demonstrates that adding per-page summaries and a dedicated keyword-ranking tool provides a massive return on investment. It can cut API costs by 30% to 58% without degrading answer quality. The key takeaway for engineers is to focus on the "targeting" aspect of the architecture. Don't just aim to reduce the size of the prompt. Aim to provide the agent with the tools and metadata necessary to minimize its own investigative footprint. Code for the harness and the parity-testing tools used in this study is reportedly available; see the paper for the canonical link.