Benchmarking Composed Image Retrieval for Applied Earth Observation

Benchmarking Composed Image Retrieval for Earth Observation: From Attributes to Disaster Monitoring

Researchers have developed ways to search massive satellite image archives using a combination of a reference picture and a text instruction. An example is asking to "show me this place after a fire." This "composed image retrieval" (CIR) offers a flexible interface. However, it is notoriously difficult to implement for Earth Observation (EO). Most existing methods are designed for consumer photos. They might find a "blue shirt" similar to a pictured one. But they struggle when the goal is to find the exact same geographic location under a different environmental state.

Current state-of-the-art vision-language models (VLMs) excel at matching general concepts. Yet, the transition to specialized satellite imagery is not seamless. It was unclear how well these modern composition techniques transfer to remote sensing. It was also unknown if they could handle high-stakes disaster monitoring. In that setting, preserving scene identity is non-negotiable. This paper bridges that gap. It establishes a formal benchmark and distinguishes between two different tasks: modifying simple attributes and performing identity-preserving change retrieval.

The Problem

Standard remote sensing image retrieval (RSIR) typically relies on a single modality (a single type of input). You either search by an image or by a text string. This creates a bottleneck for operational workflows. An analyst might need to find locations visually similar to a reference region but with a specific change. Examples include a parking lot with lower occupancy [Figure 1(a)] or a facility with a different number of storage tanks [Figure 1(b)].

The status quo breaks down during "change-centric retrieval." In attribute-based search, the goal is to find the same class of object (e.g., a swimming pool) with a new property (e.g., oval shaped). However, in disaster monitoring, the requirement is much stricter. The system must retrieve the exact same scene (the same coordinates and landmarks) but in a different state. This includes states like post-fire [Figure 1(c)] or post-flood [Figure 1(d)]. Current methods often suffer from "geographic drift." This is when a model satisfies the textual modifier (e.g., "burned area") but retrieves a completely different location. The location looks charred, but it fails the core requirement of scene identity.

How It Works

The researchers evaluate various composition strategies. These range from heavy optimization to lightweight, training-free fusion. The architecture of these methods generally follows one of three paths:

1. Textual Inversion: Methods like SEARLE perform test-time optimization. This means they run an optimization loop for every single query. They try to find a "pseudo-token" (a mathematical placeholder) in the text encoder's embedding space that represents the query image. This token is then concatenated with the text modifier. While powerful, this is computationally expensive for production.
2. Memory-Based Surrogates: FreeDom avoids optimization by using a vocabulary of textual anchors. It performs a nearest-neighbor search (finding the closest matches in a dataset) in the text embedding space. It finds words that describe the query image and merges them with the user's modifier. This provides an interpretable "textual surrogate" for the visual input.
3. Score-Based Fusion: These are the most lightweight options for deployment. WeiCom calculates two separate scores. One is for image-to-image similarity. The other is for text-to-image similarity. It combines them using a calibrated formula: $s_{WC}(q, x) = \lambda s'{g}(q, x) + (1 - \lambda) s'(q, x)$. By adjusting $\lambda$ (a weight parameter), engineers can tune the balance between the visual reference and the textual command.

The study also introduces xView2-CIR, a new dataset for testing the "same scene + target state" requirement .

Unlike previous benchmarks, xView2-CIR forces the model to distinguish between a correct post-disaster image and a "false positive." A false positive here is an image that shows the correct disaster but at the wrong location.

Numbers

The performance delta between tasks is significant. On the attribute-heavy PatternCom dataset, FreeDom is the standout performer. It reaches an average mAP (mean Average Precision, a measure of ranking quality) of 49.88% when paired with a SigLIP backbone [Table 1]. This high score indicates it is very effective at matching specific object properties like color or shape.

However, the leaderboard shifts for the identity-preserving xView2-CIR task. The authors find that WeiCom is the most robust for disaster monitoring. It achieves a total mAP of 30.67% on OpenAI CLIP [Table 3]. This lower score reflects the increased difficulty of maintaining geographic identity.

Crucially, the paper shows that "visual query expansion" can be harmful. This technique adds similar images to the query to strengthen it. On xView2-CIR, expanding the visual query from 1 image to 500 images caused the total mAP to drop from 9.59% to 4.51% . This drop happens because adding visually similar images introduces different geographic locations. This actively confuses the model's ability to lock onto a specific scene.

What's Missing

While the benchmark is a major step forward, there are gaps:

• Fine-Grained Localization: The study focuses on image-level retrieval. In real disaster response, an analyst needs to know exactly where the damage is within an image. The paper does not address how composed queries interact with localization tools.
• Temporal Depth: The xView2-CIR dataset is essentially a binary comparison (pre-event vs. post-event). Real EO workflows often require navigating a continuous temporal stream. Analysts need to see when a change occurred over time.
• Dataset Imbalance: The authors admit that xView2-CIR is relatively small and imbalanced [Table 5]. It lacks sufficient data for rare disaster categories like volcanoes or earthquakes. Results for these specific categories should be viewed as indicative rather than definitive.

Should You Prototype This

Yes, but with caution regarding your specific use case.

If your goal is attribute-based exploration (e.g., searching for specific types of land cover), training-free methods like FreeDom or BASIC are excellent candidates. They are scalable and highly interpretable. They offer strong performance for tasks like identifying building shapes or colors.

If you are building a disaster monitoring or change-detection tool, avoid "off-the-shelf" CIR methods that rely on visual expansion. The evidence suggests these will cause unacceptable geographic drift. Instead, focus on calibrated score-fusion methods like WeiCom. They are lightweight and easy to deploy. They are also empirically shown to be more robust at preserving scene identity.

Code and datasets are available at https://github.com/billpsomas/rscir.

Figures from the paper