Reinforcement Learning for Delivery Drone-Based Participatory Sensing in Dynamic Environments

Optimizing the Sky: Can Delivery Drones Also Sense the City?

Researchers have developed a new way for delivery drones to collect urban data—such as air quality or noise levels—while they are already out delivering packages. The system uses two layers of artificial intelligence: one to decide which drone should pick up which sensing task, and another to adjust the drone's speed to handle wind and save battery.

By turning logistics fleets into mobile sensor networks, cities could gain high-resolution environmental data without the cost of dedicated monitoring flights. However, integrating these two missions creates a massive coordination headache.

Making Logistics Productive for Sensing

The core challenge is a problem the authors call SensUAV. In a perfect world, a drone flying from a central depot to a customer would simply follow the shortest path. But if that drone can also deviate slightly from its route to collect air quality data at a specific intersection, the "profit" of the entire system increases.

This is a multi-objective optimization problem. You want to maximize sensing coverage (getting data from diverse locations) and delivery throughput (completing as many orders as possible) while minimizing energy consumption. Think of it like a delivery driver who is also a photographer. They want to take the most beautiful photos possible, but they cannot let the photos make them late for deliveries or run them out of gas.

The difficulty lies in the fact that the environment is not static. Wind disturbances constantly change how much energy a drone uses and how fast it moves. A detour that seems efficient in calm weather might become a battery-draining nightmare in a gusty corridor.

The Complexity of Moving Targets

To understand why this is hard, we have to look at the math. The authors prove that the SensUAV problem is NP-hard (meaning finding the absolute best solution becomes computationally impossible as the number of drones and tasks grows). Instead, we must rely on Reinforcement Learning (RL)—a type of machine learning where an agent learns to make decisions by interacting with an environment and receiving rewards.

Existing RL approaches usually suffer from two specific bottlenecks. First is scalability. As you add more drones, the number of possible combinations of "which drone does which task" explodes. This makes the math too heavy for real-time use. Second is decision heterogeneity. Assigning a task is a "macro" decision (happening every few minutes). Adjusting motor speed to fight wind is a "micro" decision (happening every few seconds).

If you try to force one single AI model to handle both, the system becomes unstable. The "noise" from tiny speed adjustments confuses the high-level logic needed to plan routes. Conversely, if the high-level logic is too slow, the drones cannot react to sudden gusts of wind.

Decoupling Strategy from Motion

To solve this, the authors propose the Two TimeScale Reinforcement Learning (TSRL) framework. As shown in, the framework splits the brain of the system into two cooperating layers.

These layers operate at different speeds.

The upper layer is the Sensing Task Dispatcher. Rather than looking at every possible drone-task pair simultaneously, the dispatcher uses a "task-embedding" approach. It first evaluates which UAVs are suitable for sensing based on their current context. Only then does it select a specific task for that UAV. This sequential decision-making reduces the complexity of the search space.

The lower layer is the Velocity Controller. This layer operates at a much higher frequency. It focuses on the immediate physics of flight. It takes in local observations, such as current wind speed and remaining battery. It then outputs a continuous velocity command. Because this controller is trained first and then "frozen" (kept constant) while the dispatcher learns, the high-level planner sees a stable environment.

The authors report that this separation allows the system to maintain high efficiency. In their experiments, the dispatcher makes a decision. The velocity controller then executes a series of $K$ micro-steps to carry out that decision .

Real-World Gains in Dense Cities

The effectiveness of this dual-layer approach was tested using real-world datasets from Hangzhou and Shanghai. The results suggest that accounting for wind and separating decision scales is a practical necessity.

The study finds that TSRL significantly outperforms existing baselines. In Hangzhou, the framework achieved average system profit improvements of 20.1% over the strongest baseline. In Shanghai, where delivery distances are longer and coordination is harder, the improvement jumped to 46.6%.

One reason for this success is seen in the behavioral analysis. While simpler "greedy" models might prioritize sensing at the expense of deliveries, TSRL finds a middle ground. As illustrated in, the model achieves high delivery profit while simultaneously maximizing sensing coverage.

Furthermore, a case study of a single UAV's trajectory shows that TSRL is capable of identifying multi-stop sensing tours. Other models often leave drones idling or only allow for unplanned sensing.

Crucially, the system is fast enough for production. The authors report an inference latency of approximately 1.3 ms per decision step. This speed makes it viable for real-time deployment in rapidly changing urban skies.

Where the Edges Are

While the results are impressive, the framework is not yet ready for a fully autonomous urban sky. The authors note that their current energy and movement models are simplified. Specifically, they highlight the need for "finer-grained aerodynamic modeling" in future work.

Current models use a simplified formula for power consumption based on velocity and wind. However, real-world aerodynamics involve complex turbulence and rotor interactions. A simple equation might miss these details. Additionally, while the framework handles wind, it does not explicitly account for other unpredictable urban variables. Factors like bird strikes or sudden airspace restrictions would add further complexity to the micro-scale controller.

Figures from the paper