Orchestra-o1: Omnimodal Agent Orchestration

The Conductor in the Machine

Instead of one giant AI trying to do everything, Orchestra-o1 acts like a conductor in an orchestra. It breaks down complex tasks involving text, images, audio, and video into smaller parts. It then assigns them to specialized "sub-agents" that work in parallel. This helps solve problems faster and more accurately.

The field of LLM-based agents—systems that can reason, plan, and use tools—is moving toward "agent swarms." In these multi-agent systems, a central coordinator manages a team of specialists. This solves problems too large for one model to handle alone. However, most current frameworks are stuck in a unimodal rut. They focus almost exclusively on text or basic vision-language tasks.

Real-world intelligence is omnimodal (involving multiple sensory modes). Humans do not just listen to a lecture. We watch gestures, observe slides, and read handouts simultaneously. Existing agents struggle when these diverse signals must interact. A new study introduces Orchestra-o1. This framework manages heterogeneous (diverse) information flow through sophisticated orchestration.

The bottleneck of native perception

Current approaches to omnimodal intelligence generally fall into two camps. Both face significant hurdles. The first category consists of "native" omnimodal agents. These models attempt to perform perception, reasoning, and tool-use within a single massive neural network. While ambitious, the authors report that even powerful models like Gemini-3-Pro struggle. They achieve only 62.5% accuracy on the OmniGAIA benchmark. This shows that a single model can fail at long-horizon reasoning (planning over many steps) and fine-grained cross-modal understanding.

The second category uses "orchestration." Here, a text-based model acts as a manager. It delegates tasks to specialized sub-agents. While more modular, existing open-source orchestration frameworks are often too rigid. Many rely on linear workflows. These are essentially a single-file line of agents. Such systems fail to capitalize on parallel execution (running multiple tasks at once). As shown in, moving from linear sub-agents to parallel, omnimodal-capable sub-agents represents a fundamental shift in scaling.

Hierarchical control and parallel execution

Orchestra-o1 operates as a hierarchical policy. It separates high-level planning from low-level execution. The "main agent" (the orchestrator) analyzes the task and builds a dependency graph. This graph maps which sub-goals must be completed before others can begin. It functions much like a construction project. You cannot install a roof until the walls are built.

The framework follows a structured loop, as outlined in :

1. Modality-Aware Decomposition: The orchestrator identifies which inputs are relevant to each sub-goal. This includes text, images, audio, or video.
2. Online Specialization: The system performs "cost-aware matching." It assigns easy tasks, like simple web searches, to cheap, fast models. It routes difficult reasoning tasks to high-capacity backends.
3. Parallel Scheduling: The orchestrator identifies "ready" sub-tasks. These are tasks whose prerequisites are already met. It then launches them simultaneously.
4. Iterative Refinement: Sub-agents return summaries and evidence. The orchestrator updates a compressed "context memory" (a condensed record of past information). It then decides whether to launch new tasks or provide a final answer.

By treating orchestration as a parallel scheduling problem, the authors derive a theoretical latency advantage (reduced waiting time). If multiple independent tasks run at once, the total time is determined by the slowest task. This is much faster than running every task one after another.

Scaling accuracy and efficiency

The performance gains reported by the authors are significant. When using GPT-5 as the main agent, Orchestra-o1 achieved 72.8% accuracy on the OmniGAIA benchmark. This is a 10.3% absolute improvement over Gemini-3-Pro. This jump means the orchestrated system handles complex, multi-modal questions much more reliably than a single native model.

The impact on open-source models is also notable. The authors developed a training recipe called DA-GRPO (Decision-Aligned Group Relative Policy Optimization). This was used to train an 8B-parameter model to act as an orchestrator. The authors report that Orchestra-o1-8B increased the previous best open-source accuracy on OmniGAIA from 20.8% to 30.0%. This suggests that small models can become capable managers. They can do this by learning strategic decisions like tool selection and task delegation.

Efficiency is where the architectural choice pays off. The authors measure cost and latency against the AOrchestra baseline. According to, Orchestra-o1 is more accurate and more cost-effective.

The orchestrator explicitly selects the best tool and model for each sub-task. This prevents wasting expensive compute on trivial operations.

Complexity and training gaps

The transition to orchestration is not free. The authors acknowledge that Orchestra-o1 introduces higher system complexity. Developers must maintain sub-agent histories, tool schemas, and asynchronous execution. This requires a more sophisticated software stack than a standard ReAct loop (a framework where a model iteratively thinks, acts, and observes).

Furthermore, the current training regime has a scope limit. The DA-GRPO algorithm optimizes the main agent's decision-making. However, the sub-agent backends remain fixed during training. The "workers" are not yet learning to work better with the "manager." This implies a need for heavy engineering to manage the orchestration overhead. There may also be a performance ceiling until sub-agents can be co-optimized.

The verdict: A blueprint for agent swarms

Orchestra-o1 is an effective blueprint for complex, multi-modal autonomous systems. Moving from "one model to rule them all" to a "managed swarm" is a superior path. This approach handles the messy reality of video, audio, and text effectively.

The framework is ready for serious experimentation. This is especially true for those with the engineering capacity for asynchronous, multi-agent workflows. If you are building systems that interact across multiple senses, these principles are essential. Code and models are available at the links provided in the paper.

Figures from the paper