Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning

A Leaner Path to Agentic Intelligence

Training AI agents—models that interact with tools, code, and environments—requires Reinforcement Learning (RL). In this regime, an agent learns through trial and error by receiving "rewards" for successful actions. However, as these agents grow more complex, the software needed to train them has become a massive burden.

A subtle but dangerous problem exists in current training stacks: "quiet failure modes." For example, a serving engine might turn text into tokens differently than the trainer does. This "retokenization drift" creates a mismatch between what the model saw during generation and what it sees during training. The system won't crash, but the resulting gradients (the signals used to update model weights) will be biased. This leads to failed experiments that are nearly impossible to debug.

Current industry standards for large-scale RL are often architected for extreme scale. They involve deeply layered stacks of distributed backends, rollout engines (systems that generate agent experiences), and complex configuration registries. While powerful, these systems create a significant "tax" on researchers. Every time a scientist wants to test a new algorithm, they must navigate a labyrinth of disconnected code. This friction slows discovery and makes it difficult to catch subtle bugs.

A new framework from NVIDIA, called Molt, proposes a different philosophy. Instead of building layers of abstraction, Molt focuses on extreme readability and a "single-backend" design. The goal is to create a codebase small enough for a human—or an AI coding assistant—to hold in their head entirely. Crucially, it aims to do this without sacrificing the ability to train massive Mixture-of-Experts (MoE) models at frontier scale.

The Complexity Tax in Agentic RL

The transition to "agentic" workloads—where models perform multi-turn reasoning or execute code—has broken many traditional training pipelines. In a typical hyperscale stack, the system is divided into many moving parts. These include a trainer, a distributed backend, and various "glue" layers that connect them.

The authors of the Molt paper argue that this multi-backend structure is a poor default for research. When a researcher modifies an algorithm, the change often has to be threaded through several different layers of the stack. This isn't just an inconvenience. It increases the cognitive load on the researcher and hides the actual logic of the experiment behind layers of infrastructure.

One Loop, Three Components

Molt replaces the fragmented stack with a single, asynchronous loop designed around three core components: user agents, vLLM rollout engines, and a single FSDP2 policy actor . FSDP2 (Fully Sharded Data Parallelism) is a technique that breaks a large model into pieces and distributes them across many GPUs.

[: The whole system: three components and one loop. Molt composes user agents in plain Python, vLLM rollout engines behind a request router, and a single FSDP2 policy actor on NeMo AutoModel around one Ray asynchronous queue.]

The framework operates on a "token-first" contract. Rather than passing text back and forth, Molt ensures that the exact token IDs and log-probabilities (the mathematical likelihood of a specific word being chosen) generated by the engine are passed directly to the trainer. This ensures that "every trained token is exactly the token that was generated." This eliminates the risk of retokenization drift.

To handle scale, Molt leverages existing, highly optimized libraries. It uses Ray for orchestration, vLLM for high-speed generation, and NVIDIA's AutoModel for distributed training. By composing these instead of forking them, Molt ensures that any upstream improvements flow directly into the framework.

Performance Without the Bloat

Does a leaner codebase come at the cost of speed? The authors tested Molt against a state-of-the-art Megatron-based stack (called "slime") using a 30B parameter multimodal MoE model.

The paper reports that Molt is statistically comparable to the Megatron-based stack in throughput. Specifically, the authors measure a step time of $119.4 \pm 2.3$ seconds for Molt. In contrast, slime recorded $109.5 \pm 10.3$ seconds. While slime was slightly faster on average, the difference falls within the range of normal variability between runs. For a researcher, this means they can gain massive simplicity without losing significant training speed.

Molt achieves this with a much smaller footprint. The Molt RL implementation consists of approximately 8.6K lines of Python code. Competing frameworks like verl and slime use 62K and 25K lines respectively [Table 1]. Furthermore, the authors demonstrate that Molt scales; they successfully ran the same asynchronous loop on a 700B parameter MoE model. This proves that "leanness" does not prevent the system from handling frontier-scale workloads.

Addressing the MoE Consistency Gap

The framework is highly efficient, but the paper notes challenges regarding Mixture-of-Experts (MoE) models. In an MoE model, only a subset of the model's parameters (the "experts") are active for any given token.

The authors report a specific challenge: the rollout engine and the training actor might independently choose different experts for the same token. This occurs due to tiny numerical differences. To solve this, Molt uses "rollout routing replay." The engine tells the trainer exactly which experts were used during generation.

However, the authors note a limitation in their benchmarks. In some tests, this mismatch led to throughput-only measurements rather than effective policy updates. This suggests that while the mechanism helps, full convergence-parity validation for these massive models is still a work in progress.

The Verdict: A New Standard for Research

If you are deploying a production pipeline for a single, static model, existing hyperscale stacks are likely sufficient. However, if you are researching agentic behavior, Molt is a significant step forward.

By prioritizing a "readability-first" design, the authors have created a framework suited for the era of AI-augmented development. The ability to trace an algorithm from a single CLI flag down to a specific tensor is a massive advantage for debugging. With its open-source implementation available at https://github.com/NVIDIA-NeMo/labs-molt, Molt provides a high-performance, low-friction substrate for modern RL research.

Figures from the paper