Frequency-Guided Action Diffusion via Sub-Frequency Manifold Traversal

Robots often move jerkily because they try to copy every tiny mistake or tremor in human demonstrations. This is a fundamental problem in visuomotor policy learning (the process of mapping visual observations to motor actions). Machines attempt to mimic human experts through behavior cloning. Because natural human movement contains high-frequency noise—intermittent jerks, pauses, and jitters—policies trained on raw data inevitably inherit these suboptimal behaviors.

Current state-of-the-art approaches use diffusion-based policies to model complex, multimodal distributions of actions. While powerful, these models treat action generation as a broadband mapping problem. They attempt to predict the entire frequency spectrum of a movement at once. This makes them prone to amplifying high-frequency artifacts during the iterative denoising process (the step-by-step refinement of noise into a signal).

The Problem

In the current paradigm of diffusion-based behavior cloning, a model is trained to map noisy inputs directly to the full-frequency manifold of expert actions. This is problematic because human demonstrations are rarely "clean." They contain high-frequency noise that is temporally entangled with the low-frequency intentionality of the task. When a diffusion model tries to learn this broadband mapping, it struggles to distinguish between global structure and microscopic jitter.

This pathology is specifically acute in diffusion models. During the reverse denoising process, the model can inadvertently amplify high-frequency artifacts. Instead of producing a smooth trajectory, the policy outputs erratic motor commands. This increases mechanical wear and reduces task reliability. Existing smoothing techniques, such as applying a low-pass filter or using temporal ensembling (averaging actions across different time steps), often fail. According to the authors, these methods can degrade success rates by removing necessary fine-grained details or by causing multi-modal action predictions to collapse into a single modality [Table 9, 10].

How It Works

The authors introduce the Frequency Guidance Operator (FGO). It replaces the single broadband objective with a hierarchical, multi-band approach. The core idea is to force the model to navigate through increasingly complex spectral layers.

1. Multi-Band Training: Instead of training on the full action chunk, the authors use a Discrete Cosine Transform (DCT) to apply a low-pass filter $L_f$. This creates a "sub-frequency manifold"—a version of the action containing only frequencies up to a certain cut-off $f$. The model is trained to predict these truncated versions, conditioned on the specific cut-off frequency $f$.
2. k-f Coupled (KFC) Sampling: To ensure the model doesn't waste capacity, the authors implement KFC sampling. High-frequency components are destroyed faster than low-frequency ones during the forward diffusion process. Therefore, the training limits the maximum allowable frequency ($f_{max}$) based on the current noise level $k$. When noise is high, the model only sees low frequencies. As noise decreases, the spectral window expands .

3. Progressive Guidance: During inference, the model does not jump straight to the full-frequency action. Instead, it uses a composite vector field. At each denoising step, it calculates a weighted combination of a "base" prediction and a "fine" prediction. As the denoising progresses, the guidance weight $\omega_k$ shifts the focus from the low-frequency structure to the high-frequency details .

Numbers

The primary benefit of FGO is not just success rate, but the quality of the resulting motion. The authors report significant improvements in action smoothness across 15 robotic tasks.

On the Robosuite "Can" task, the authors measure physical smoothness using JerkRMS (the root mean square of motor jerk). FGO achieves a JerkRMS of $40.79 \pm 0.46 \text{ rad/s}^3$. This is a substantial reduction compared to the $50.87 \pm 1.27 \text{ rad/s}^3$ reported for the DP3 baseline [Table 3]. This lower value means the robot executes much smoother, less violent movements. Regarding success rates, FGO outperforms baselines like DP3 and DiT-Policy across both simulation and real-world benchmarks. In the Robosuite/MimicGen suite, FGO reaches an average success rate of 56.6%. This beats DP3 (52.9%) and FreqPolicy (51.5%) [Table 1].

However, there is a clear trade-off in computational efficiency. The authors note that while training time is only negligibly higher than the baseline, inference latency increases. On the Adroit "Hammer" task, FGO's inference speed is $44.22 \text{ ms}$. The DP3 baseline runs at $39.49 \text{ ms}$ [Table 4]. This extra overhead comes from the multiple forward passes required by the guidance mechanism.

What's Missing

The paper provides a strong empirical foundation, but there are gaps that a production engineer should consider:

• Latency vs. Control Frequency: The authors acknowledge the increased inference latency. However, they do not quantify the impact on high-frequency control loops. In some real-world robotics applications, a jump from $39\text{ms}$ to $44\text{ms}$ might push the system below the required real-time threshold. This could cause instability that offsets the gains in smoothness.
• Precision Trade-offs: There is a risk of "over-smoothing." The authors admit that the guidance can occasionally generate trajectories that lack the precision needed for extremely fine-grained manipulation. They do not provide a formal boundary to prevent this in high-stakes tasks.
• Generalization to Non-Gaussian Noise: The framework relies heavily on the spectral properties of the diffusion process. This assumes isotropic Gaussian noise. It is unclear how FGO would behave in environments where the noise profile is structured or non-stationary.

Should You Prototype This

Yes, but with caveats. If your primary pain point is "jittery" robots that suffer from mechanical fatigue, FGO is a direct solution. The improvement in JerkRMS is concrete. It suggests a real reduction in hardware stress.

However, if you are operating on a tight latency budget, proceed with caution. If your tasks require sub-millimeter precision where every micro-adjustment counts, monitor the results closely. Start by testing the FGO integration on a single task. See if the smoothness gain outweighs the $10\text{--}15\%$ hit to inference speed. Code is reportedly available; see the project website for the canonical link: https://henrywjl.github.io/frequency-guidance-operator/.

Figures from the paper