Project Summary

• Evaluation of existing general-purpose and EO-specialized LLMs on existing EO-related benchmarks and identifying the limitations of existing benchmarks.
• Building an EO-based benchmark to overcome existing gaps.
• Evaluation of existing general-purpose and EO-specialized LLMs on our curated benchmark

Development Tools

• The agentic AI framework was orchestrated using Google's Agent Development Kit (ADK), with tool integration across MCP servers handled via FastMCP.
• Model serving and inference were run through vLLM, with model weights and datasets managed via Hugging Face's transformers, datasets, and accelerate libraries.
• Geospatial and EO data handling relied on xarray, rioxarray, geopandas, and rasterio, with acquisition tools including the CDSE (Sentinel-1/2) API, the CDS API (ERA5), and the EUMETSAT search.
• Semantic search and retrieval were supported by sentence-transformers and a Qdrant vector store, with Neo4j GraphRAG used for graph-based knowledge retrieval, and the overall system exposed via FastAPI/Gradio interfaces.

Development Outputs

• Code: Github Link (will be made public soon)
• Papers: Evaluating EO-Specialized LLM on EO Agentic Benchmarks, GAIA, ECCV 2026, ESA-EO Bench(Preprint - Under Review)

Project Description

Our initial study focuses on evaluating an EO-specialized LLM (EVE) on existing EO benchmarks such as ThinkGeo and Earth-Agent. Our findings can be summarized as follows:

• EVE shows clear, consistent gains in tool orchestration: it ranks best or second-best on instruction-following, tool selection, and tool sequencing across both benchmarks.
• These gains hold even against closed-weight models an order of magnitude larger, as EVE matches or beats them on orchestration metrics.
• Relative to its own backbone (Mistral-3.2-Small), EVE improves by double-digit margins on several planning metrics, indicating the EO-specialized pretraining/fine-tuning is doing real work, not just inheriting backbone capability. However, these orchestration gains do not consistently translate into higher end-to-end task accuracy in the single-agent setting. A model can select and sequence the right tools more reliably, but it does not produce a more correct final answer more often.

To summarize, tool orchestration is not the bottleneck; rather, it is the grounded synthesis of tool outputs. EVE's EO-specific post-training transfers cleanly to planning and orchestration, matching or beating closed models several times its size on exactly the metrics where scale is supposed to matter most. But before concluding that LLMs are broadly unable to synthesize tool outputs in EO agentic systems, we wanted to test whether this is a general failure mode, rather than an artifact of the specific single-agent setting of ThinkGeo and Earth-Agent.

Existing benchmarks can't settle that question on their own because they are largely single-dimensional: most chain tools are limited to imagery retrieval, index calculations, and similar operations within a fixed, sandboxed toolset, rarely extending beyond multispectral data and only occasionally reaching SAR.

Our second study focuses on this identified gap by designing a new benchmark, ESA-EO-Bench, that stresses both text/policy reasoning and visual analysis in equal measure, revealing a different picture from single-agent, imagery-centric benchmarks.

The benchmark is evaluated live against a running Google-ADK-based multi-agent supervisor rather than a sandboxed setting. It spans 294 tasks across 10 independently authenticated MCP servers with real credentials (CDSE, GEE, CDS/ERA5, EUMETSAT, arXiv, GIS, ESA Dashboard, EOSS KG, MOOCs, and VLM reasoning) and 70 unique tools, ranging from satellite imagery acquisition to scientific literature search to knowledge-graph retrieval.

Its design is built around two principles that existing single-agent EO benchmarks don't jointly satisfy:

• Balanced coverage of policy/textual reasoning tasks alongside the visual-analysis tasks that dominate the field, rather than treating text and policy as an afterthought.
• Multi-agent structure, so orchestration quality can be measured when tool use is distributed across collaborating specialist agents, rather than concentrated in a single agent's tool-call trace.

Tasks fall into four evaluation categories, each targeting a distinct failure mode rather than just task difficulty. Sequential Reasoning (111 tasks, 37.8%) tests chaining across 2+ tool calls where one step's output feeds the next: geocode → fetch imagery → inspect is the canonical pattern. Cross-agent Synthesis (84 tasks, 28.6%) requires combining tools from two or more MCP servers/domains into a single solution, directly probing multi-agent handoff that single-agent benchmarks can't exercise. Tool Grounding (70 tasks, 23.8%) isolates a single, direct tool call with concrete arguments, with no chaining, as a baseline for raw tool-selection accuracy. Scope Awareness (29 tasks, 9.9%) is the deliberate negative case: the correct behavior is to call no tool at all, testing whether a model can recognize an out-of-scope request rather than forcing a spurious tool call, a dimension entirely absent from prior imagery-only benchmarks.

Reference-solution length tracks this design directly: most solutions chain 2–4 tool calls, with a long tail of 7–12-step solutions arising from Cross-agent Synthesis tasks that stitch several domains together end-to-end, and Scope Awareness tasks sitting at zero tool calls by construction.

Taken together, ESA-EO-Bench lets us re-test the central question from our first study under conditions the single-agent benchmarks couldn't reach. Whether EO-specialized post-training is genuinely a better orchestrator and reasoner on policy and text as well as imagery, and whether the orchestration gains we observed for EVE persist or erode, once tool use is distributed across collaborating specialist agents instead of concentrated in one. Evaluating EVE and comparably scaled general-purpose LLMs on this benchmark is the next stage of this work. It will determine whether strong single-agent orchestration is a reliable predictor of multi-agent, end-to-end EO task performance, or whether it is a capability that degrades once responsibility for grounded synthesis is split across agents.