Project Description

Abstract

• TerraVerse learns the atmospheric forward map with a generative foundation model. Given a surface-reflectance product and an atmospheric state, the model generates the top-of-atmosphere measurement the instrument would have recorded — a fast, conditionable complement to radiative transfer modeling rather than a replacement for it.
• One checkpoint serves arbitrary instruments. A wavelength-conditioned hypernetwork maps each band's center wavelength and bandwidth to that band's projection weights, so the input and output band sets are requested at inference rather than fixed at training time. This front end accounts for only 0.5 % of the model's parameters.
• Spectral super-resolution is demonstrated. An 11-band multispectral input reconstructs a 224-band hyperspectral target at 92.09 % SSIM, against 92.21 % from the native dense hyperspectral input — effectively no penalty for the sparse input.
• The model operates directly in pixel and band space. Trained as a stochastic bridge with no autoencoder, it carries no reconstruction error floor, and reaches 34.64 dB PSNR / 91.52 % SSIM on native-sensor reconstruction with a fitted radiometric amplitude of 0.993.
• The project curated a large paired hyperspectral corpus of 1,684,227 EnMAP Level-1C / Level-2A observations across 1,331,765 unique locations, with leakage-free spatial and datatake-level splits and harmonized per-sample metadata.
• All models, code, and datasets are released openly under permissive licenses through the ESA-TerraVerse Hugging Face organization.

Detailed description

Overview

TerraVerse is an ESA Φ-lab–funded project that brings together DLR (lead), IBM Research Europe, and VITO to build a generative foundation model that simulates satellite imagery for future missions and for missions with limited public data.

Designing instruments, developing algorithms, and prototyping applications all depend on large volumes of realistic imagery — precisely what new or specialized missions lack. TerraVerse generates realistic, radiometrically calibrated satellite imagery on demand. Because it learns from both real measurements and physics, its outputs are not only visually convincing but physically consistent.

The problem with radiative transfer models

Radiative transfer models (RTMs) are the reference tool for simulating what an instrument would measure, and they remain the gold standard for physical fidelity. They are also constrained in five ways that hold mission preparation back.

Figure 1. Five ways today's radiative transfer models fall short — slow and costly per simulation, idealized assumptions such as plane-parallel atmospheres and Lambertian surfaces, no robust benchmarks, single modality or wavelength range, and fixed inputs — and the capability TerraVerse turns each barrier into.

TerraVerse does not replace radiative transfer physics. It uses it: radiative-transfer look-up tables generated with the DLR in-house model PyDOME supply synthetic Level-1C targets from any Level-2A patch and any sampled atmosphere. Moving that simulation to a GPU produced a speed-up of roughly three orders of magnitude, enabling the generation of synthetic targets in real time during training rather than pre-computing them. The training schedule anneals the synthetic proportion from 80% down to zero over the first third of training, so the model is bootstrapped on physics and then converges on real observations.

Model architecture

TerraVerse is a FLUX-style multimodal diffusion transformer of 356.7 M parameters, trained as a stochastic bridge from EnMAP Level-2A surface reflectance to EnMAP Level-1C top-of-atmosphere radiance.

Figure 2. From a Level-2 scene and target metadata to a calibrated Level-1 prediction. A vision encoder reads the bottom-of-atmosphere scene and its sensor simulations; an MLP encoder turns the conditioning metadata — solar zenith, sensor angle, water vapor — into AdaLN modulation. A dynamic spectral adapter supplies the bridge source, and the FLUX decoder, further conditioned on the target instrument’s spectral response and radiometric accuracy, integrates the trajectory to the predicted Level-1 scene. Training loss is computed against both real and simulated Level-1 targets.

Three design choices distinguish it:

• No autoencoder. The model works directly in pixel and band space, so there is no reconstruction error floor imposed by a latent bottleneck — which matters when the output must be radiometrically faithful rather than merely plausible.
• Latent Bridge Matching. The trajectory starts at the Level-2A observation rather than at random noise, so the source supplies the low-frequency spatial structure and the network learns the atmospheric residual. This makes scene-to-scene translation both faster and more accurate than starting from noise.
• Wavelength-conditioned spectral front end. A hypernetwork maps each band's center wavelength and bandwidth to that band's projection weights. Every band shares parameters, so an arbitrary instrument's band set can be the input, and an arbitrary band set can be requested as the output — including the 22 Level-1C water-vapor bands that have no Level-2A counterpart.

Conditioning covers aerosol optical thickness, water vapor, solar and sensor geometry, and a dimensionless super-resolution factor. Every training sample is presented as some instrument's view through a Gaussian spectral response matrix, drawn from native EnMAP, randomly generated plausible instruments, and nine named instruments, including Sentinel-2, Landsat 8/9 OLI, PlanetScope, SPOT-6, WorldView-3 VNIR, MODIS, and VIIRS.

Datasets

The pre-training corpus draws on globally distributed hyperspectral acquisitions extended with top-of-atmosphere Level-1C products and harmonized metadata, producing 1,684,227 paired L1C/L2A observations over 1,331,765 unique locations from 7,648 datatakes, acquired between June 2022 and August 2025 at 128 × 128 pixels and 30 m resolution, with 224 bands in L1C and 202 in L2A. Each sample carries cloud and per-band validity masks and a metadata record covering acquisition time, solar and viewing geometry, scene water vapor and aerosol optical thickness, and the band-wise gain and offset needed to recover radiance.

Splits prevent leakage at both the spatial and the datatake levels: calibration areas are protected, their associated datatakes are excluded, and a 10 km buffer is enforced around calibration and validation patches.

Results

Verification against the project's protocol gives the following picture.

Spectral super-resolution is the standout result: reconstructing a dense 224-band hyperspectral target from a sparse 11-band multispectral input incurs almost no fidelity loss, which is the clearest evidence that the wavelength-conditioned front end works as intended.

Cross-sensor generalization to real Sentinel-2 data is an active line of development. Zero-shot application — the model having seen only EnMAP pairs and simulated band sets during training — already faithfully transfers spatial structure, and the residual has been localized to a wavelength-dependent amplitude offset rather than to a loss of detail. Fine-tuning, which is the intended use case, has not yet been applied.

Target missions

• Φsat-2 (flying now). An ESA New Space mission carrying an onboard-AI multispectral imager, with limited public data so far. TerraVerse fine-tunes to its characteristics and tests gap-filling against real acquisitions.
• CHIME (planned). The Copernicus Hyperspectral Imaging Mission for the Environment. TerraVerse simulates its imagery ahead of launch, so that algorithms are ready on day one.

Research and Development Tools

• Data processing and dataset construction: Paired L1C/L2A EnMAP data are curated and stored in Zarr for large-scale streaming from object storage, with harmonized sidecar metadata and cloud and validity masks.
• Radiative transfer: PyDOME, the DLR in-house radiative transfer model, generates look-up tables for synthetic Level-1C targets. GPU porting delivered a roughly 1000× speed-up, enabling live synthetic-target generation during training.
• Deep learning framework: PyTorch, with bfloat16 autocast, compiled graphs, and distributed data parallel training. The released checkpoint was trained on eight NVIDIA H100 accelerators for 20 epochs × 10,000 steps, seeing 25.6 M samples.
• Instrument simulation: Gaussian spectral response matrices applied to every training sample, drawing from native EnMAP, randomly generated plausible instruments, and nine named real instruments.
• Evaluation: PSNR, SSIM, and spectral angle mapper against held-out EnMAP validation splits, with radiometric amplitude fitting to detect systematic bias, and cross-sensor evaluation against real Sentinel-2 acquisitions.

Project Outcomes

Models and Weights

TerraVerse 0.1 Base is released under an Apache-2.0 license.

• ESA-TerraVerse Hugging Face organization: https://huggingface.co/ESA-TerraVerse
• IBM–ESA Geospatial Models: https://huggingface.co/ibm-esa-geospatial

Datasets

• EnMAP paired L1C/L2A dataset — 1,684,227 paired top-of-atmosphere / surface-reflectance observations over 1,331,765 unique locations from 7,648 datatakes, with leakage-free splits

Publications

• B. Blumenstiel, J. Jakubik, N. Ait Ali Braham, D. Efremenko, P. Fraccaro, R. Salih Kuzu, S. Sterckx, T. Brunschwiler, "TerraVerse: Simulating Future Satellite Mission Imagery", 2nd ESA–NASA Workshop on AI Foundation Models for Earth Observation, 2026.