SEAOTTER: Sensor Embedded Autoencoding with One-Time Transcode for Efficient Reconstruction

At a compression ratio of 200:1, the authors report a 7× increase in encoding speed and a 3.5× increase in decoding speed compared to AVIF. They also report an 8% boost in ImageNet top-1 accuracy. This is a significant delta for systems where every millisecond of latency and every bit of bandwidth counts.

In cloud robotics, there is a fundamental mismatch between hardware capabilities. Robots capture massive amounts of high-resolution visual data using low-power sensors. However, they lack the bandwidth to stream that data raw to the cloud. To move data efficiently, engineers typically choose between conventional codecs like JPEG—which are fast but destroy image quality at high compression—or newer standards like AVIF. AVIF offers better rate-distortion trade-offs (the balance between file size and quality). But it demands massive computational resources for encoding. Recent asymmetric autoencoders (AEs) attempt to solve this. They use lightweight encoders and heavy DNN (Deep Neural Network) decoders. But they introduce a new headache: they produce bespoke, non-standard bitstreams. These break existing downstream infrastructure like web browsers, ML data loaders, and hardware accelerators.

The Problem

The core issue is an asymmetry of resources across the entire data lifecycle. As shown in, the sensor, the cloud, and the consumer all have different power and bandwidth budgets.

Current solutions force a compromise. Standard codecs like AVIF provide good quality but are too expensive to run on a battery-constrained robot without custom ASICs (Application-Specific Integrated Circuits). Conversely, specialized neural codecs allow for extremely light on-device encoding. But their decoding process is too heavy for "decode-many" workloads. In a typical robotics pipeline, a single frame might be encoded once. However, it might be read hundreds of times during a training epoch. If every read requires a heavy DNN decoder, the cumulative compute cost becomes prohibitive. Furthermore, these neural formats are "black boxes" to the rest of the stack. You cannot simply drop a raw latent representation into a standard PyTorch DataLoader without building a custom shim.

How It Works

The authors propose SEAOTTER. This framework treats compression as a three-stage pipeline: sensor $\rightarrow$ cloud $\rightarrow$ consumer. The key innovation is a "one-time transcode" in the cloud. This turns a compact neural latent into a standard, highly usable JPEG file.

1. Sensor-Side Encoding: The robot uses a frozen, lightweight FRAPPE encoder ($G_A$). This encoder is designed for extreme efficiency. It costs only 10–100 MACs (Multiply-Accumulate operations) per pixel. It projects the high-resolution input into a compact, quantized $int8$ latent representation. Because the encoder is frozen, the sensor-side compute remains a predictable, hard budget.
2. Cloud-Side Synthesis: Once the latent reaches the cloud, a heavy, fine-tuned FRAPPE decoder ($G_S$) reconstructs the pixels. This decoder is not trained to minimize traditional error metrics like PSNR (Peak Signal-to-Noise Ratio). Instead, it is fine-tuned to maximize the accuracy of downstream machine perception tasks.
3. The JPEG Sandwich: To ensure the reconstructed image survives being turned into a JPEG, the authors implement a "learnable JPEG sandwich" .

This consists of a learned color transform ($F$), a standard JPEG encode using a custom $3 \times 8 \times 8$ quantization matrix ($Q$), and an inverse color transform ($F^{-1}$) at the consumer side.

The color transform $F$ replaces the standard RGB-to-YCbCr conversion. It uses a $3 \times 3$ convolution combined with a softsign companding function (a non-linear mapping to limit dynamic range). This maps pixels into a specialized color space. This space is more resilient to JPEG quantization artifacts. By training the quantization matrices $Q$ alongside the color transform, the system learns which frequencies to preserve for task accuracy.

Numbers

The results suggest that SEAOTTER prioritizes task-relevant information over pixel-perfect reconstruction.

The paper reports that at a compression ratio of 221:1, the SEAOTTER-FT (fine-tuned) variant achieves 69.02% ImageNet top-1 accuracy. This compares to just 56.22% for the vanilla FRAPPE codec [Table 1]. This represents a $+12.80$ percentage point improvement. Even the zero-shot (ZS) version recovers $+4.03$ pp over the base autoencoder.

From a systems perspective, the throughput gains are the headline. The authors measure encoding speeds exceeding 250 MPx/s (Megapixels per second). This is sufficient for 1080p/30fps streaming over Wi-Fi .

On the consumer side, the steady-state decoding is approximately 3.5× faster than AVIF decoding on a CPU. As seen in, SEAOTTER-FT dominates AVIF and WaLLoC across global classification, dense segmentation, and VLM-style (Vision-Language Model) zero-shot tasks.

This proves the transcode makes the data more useful for machine vision.

What's Missing

While the results are impressive, there are several gaps that a production engineer should consider.

First, the paper lacks component-specific ablations. It is unclear how much of the gain comes from the learned color transform versus the learned quantization matrices. If you were to implement this, you wouldn't know which part of the "sandwich" provides the most value.

Second, the scope is limited to the RGB modality. The authors note that depth or infrared signals could use the same architecture. However, they haven't characterized how the learned color transforms would behave with non-standard spectral data. In many robotics contexts, depth-map compression is just as critical as RGB.

Finally, there is no evaluation of human perception. The paper focuses entirely on "machine perception" (CNNs, ViTs, VLMs). Many robotics systems require teleoperation, where a human operator needs to see the feed. Since the learned color space is non-standard, visual artifacts might be problematic for humans.

Should You Prototype This

Yes, if you are building a cloud-robotics data pipeline.

If your bottleneck is uploading high-res imagery from robots to a central server, SEAOTTER offers a win. It lowers the uplink bandwidth requirement while improving the quality of the data your models ingest. The ability to emit standard JPEG files means you won't have to rewrite your entire data engineering stack.

However, if your primary use case is real-time teleoperation, proceed with caution. You will likely need to validate that the "optimized" color space is actually legible to human eyes. The code is available at https://github.com/UT-SysML/seaotter.

Figures from the paper