Skip a Layer or Loop It? Learning Program-of-Layers in LLMs

Instead of running every layer of an AI model for every question, this new method learns to skip unnecessary layers or repeat important ones. This allows the model to work faster on easy tasks. It also lets the model think harder on difficult ones. This happens without needing to be retrained.

Current Large Language Models (LLMs) operate on a principle of rigid uniformity. Whether you ask a model to define "hello" or to solve a complex equation, the model executes the same sequence of transformer layers. This "static depth" approach uses a fixed, non-recurrent order. It assumes that every input requires the same amount of cognitive effort. This is rarely the case in real-world reasoning.

The authors argue that fixed-depth execution captures only a narrow slice of an LLM's actual reasoning capacity. They suggest that the layers within a pretrained model can be viewed as a library of atomic functions. These are modular building blocks. They can be rearranged, skipped, or looped to create a custom "program" for each input.

The inefficiency of static depth

In standard inference (the process of using a trained model to generate predictions), the model follows a predetermined path. It moves from Layer 0 to Layer $D-1$. This creates a mismatch between task complexity and allocated computation. For simple queries, the model likely over-computes. It wastes energy and latency on redundant transformations. For difficult math problems, the model may under-compute. It might fail because it lacks the "latent reasoning" depth required for a correct solution.

Previous attempts to fix this have focused on "early exiting." This involves stopping the computation early if a model is confident. Other methods focus on "layer skipping." These prune unimportant layers. However, the authors point out that these methods are often one-dimensional. They might allow a model to stop sooner. But they rarely allow it to "think harder" by revisiting layers. Existing dynamic-depth methods often make local, layer-by-layer decisions. These decisions lack global coordination. They attempt to navigate a forest by looking only at the next step.

Programming the layers

The researchers propose a framework called Program-of-Layers (POLAR). Rather than treating the model as a monolithic pipe, POLAR treats the pretrained layers as a toolkit. An execution program $\pi$ is a sequence of layer indices. This sequence tells the model which functions to call and in what order .

The mechanism works through three primary stages:

1. Segmentation: The system partitions the continuous stack of layers into discrete "modules" or segments. To keep the search space manageable, the authors bound the segment length to a maximum of four layers ($K_{max}=4$). They note that most effective programs rely on short, contiguous blocks .
2. Operation Assignment: For each segment, the system assigns one of three operations: skip (omit the segment), keep (run the segment once), or repeat (run the segment twice). Combining these two operators—skipping and looping—allows for a richer variety of computational paths.
3. Single-Shot Prediction: To avoid the high cost of searching for the perfect program using Monte-Carlo Tree Search (MCTS, a heuristic search algorithm), the authors train a lightweight "POLAR prediction network." This network takes the input and encodes it. It then directly outputs the optimal segmentation mask and operation labels.

Crucially, the base LLM remains completely frozen. No weights are changed. The system simply changes how the existing weights are accessed during the forward pass.

Evidence of latent reasoning

The authors used MCTS as a diagnostic tool to prove that better programs actually exist. Their findings were striking. For inputs that the standard model solved correctly, 75.5% actually admitted shorter, more efficient programs .

Even for inputs the model got wrong, 36.2% could be corrected by using a different program . These programs were sometimes even shorter than the standard pass.

When moving to the actual POLAR predictor, the performance gains are significant. On the DART-Math benchmark, the authors report that POLAR consistently improves accuracy over standard inference. For the Llama-3.2-3B-Instruct model on the easiest difficulty level (DM-1), the authors report a jump from 40.6% to 46.2% accuracy when using multiple candidate programs [Table 2].

Perhaps most importantly, the authors demonstrate "test-time scaling." As you allow the model to explore more candidate programs (increasing $k$), the accuracy climbs monotonically [Figure 8a]. This proves that providing more "thinking time" through varied execution paths directly translates to better performance. They call this scaling at latent reasoning depth.

Trade-offs and constraints

While the results are impressive, the method has specific limits. The segmentation is bounded by $K_{max}=4$. This means the model cannot easily form very long, complex recurrent loops involving many layers in a single block. They justify this by noting that most successful programs found via MCTS were naturally local and simple . However, this limits the theoretical expressiveness of the "program" the model can run.

Furthermore, the effectiveness of POLAR is tied to the quality of the supervision. The predictor is trained on programs discovered by MCTS. If the search process fails to find the truly optimal programs, the predictor will inherit those limitations. There is also the matter of "out-of-distribution" (OOD, data that differs from the training set) performance. While the authors report that POLAR generalizes well to new subjects like physics or law [Table 3], the degree of transferability in highly specialized domains remains an open question.

Finally, the method relies on the assumption that pretrained layers contain enough functional diversity. If a model's layers are highly specialized for a strict sequential flow, the gains from skipping or looping might diminish.

The verdict

If you want to squeeze more intelligence out of existing, frozen models without retraining, POLAR is a compelling direction. The authors demonstrate that the overhead is remarkably low. It adds only about 3.05 ms of latency [Table 4]. This represents roughly 0.8% of a standard forward pass. Because the predictor itself is tiny (about 2.1 million parameters), it is a negligible fraction of the host model's size.

Is it ready for production? For mathematical reasoning or logic-heavy tasks where accuracy is paramount, the answer is a qualified yes. The ability to scale accuracy by simply increasing the number of execution programs explored is a powerful lever. Code for the project is reportedly available at https://github.com/tianyi-lab/PoLar.

Figures from the paper