Summary of the Project

The ESA OpenSR project (https://opensr.eu/) aimed to bring robust, accountable, and scalable multi-spectral super-resolution techniques to the Earth Observation (EO) community for Sentinel-2 (S2) products.

• OpenSR provides a deep learning framework for super-resolving Sentinel-2 imagery.
• Models enhance resolution while preserving native spectral fidelity and spatial alignment.
• Models are trained on harmonized synthetic data matched to Sentinel-2 spectral characteristics, effectively tackling domain shift challenges.
• The Super-Resolution design focuses on Latent Diffusion Models (LDM) due to their ability to deliver state-of-the-art generative results.
• A Fourier-based low-frequency hard constraint is applied at the final SR layer, to enforce radiometric consistency. This is the first use of this technique in satellite SR.
• A double-step Wald protocol enables SR of 10 m bands but also the 20 m Sentinel-2 bands, achieving an effective resolution of 2.5 m across all Sentinel-2 channels.
• A benchmarking framework has been developed for trustworthy and context-aware assessment of SR model effectiveness, revealing crucial trade-offs, such as that between reconstruction fidelity and perceptual realism.

Project Description

The globally applicable OpenSR models can reconstruct an impressive amount of detail for both the 10m RGB-NIR bands as well as the 20m bands of S2. We have developed a range of models, from lightweight SWIN to large latent diffusion models, enabling the user to select the model based on their specific needs. A special focus is on preserving the actual spectral information; the goal is to keep the actual reflectance values true to the phenomena on the ground. Since downstream applications rely on spectral accuracy, our products are more than just hallucinated enhancement. To convince remote sensing specialists of the viability of SR products, OpenSR models produce confidence metrics along with the models to allow users to gauge the applicability of the SR product to their specific problem. Moreover, all OpenSR models and packages are openly accessible on GitHub creating an ecosystem of tools for the whole SR pipeline.

OpenSR Datasets

The core objective regarding datasets in the OpenSR project is to assemble a vast corpus of multispectral, spatially co-registered Sentinel-2 (S2) and Very High Resolution (VHR) image pairs suitable for training and validating Super-Resolution (SR) models. These pairs must operate within the S2 spectral domain and accurately capture the spatial information required to upscale the 10-meter and 20-meter S2 bands to a target resolution of 2.5 meters. Recognizing the difficulty of acquiring enough high-quality real image pairs, the project employs a sequenced strategy: starting with large volumes of generalized data for pre-training before transitioning to higher-fidelity, domain-specific resources.

The SEN2NAIP dataset is central to the training effort. It consists of cross-sensor pairs using S2 and High-Resolution aerial imagery from the National Agriculture Imagery Program (NAIP) covering the contiguous United States. NAIP was selected due to its extensive coverage and public availability, which facilitates large-scale dataset creation.

The primary function of synthetic Sentinel-2 image generation within the project is to develop a reliable algorithm capable of degrading High-Resolution (HR) aerial imagery, specifically NAIP data (downscaled to 2.5 meters), to accurately mimic the characteristics of a Low-Resolution (LR) Sentinel-2 (S2) image at 10 meters resolution. This systematic process is essential for creating large-scale, radiometrically and spatially consistent training environments required for robust Super-Resolution (SR) model development, circumventing the limitations associated with acquiring concurrent, harmonized cross-sensor data.

The SEN2NAIP collection of SISR and MISR datasets, built using S2 and NAIP image pairs, is available at HuggingFace: https://huggingface.co/datasets/isp-uv-es/SEN2NAIP.

Latent Diffusion SR Design & Implementation

The Super-Resolution (SR) design focuses on Latent Diffusion Models (LDM) due to their ability to deliver state-of-the-art generative results while substantially mitigating the high computational demands of Pixel-Space Diffusion Models (DMs). This approach reduced the computational cost, making large-scale processing of Earth Observation (EO) data feasible by performing the intensive denoising steps in a lower-dimensional latent space.

The LDSR-S2 architecture employs a frozen Autoencoder (AE) and a Denoising UNet. The AE compresses the spatial dimension of the multispectral input (e.g., 4-band 512x512) by a factor of 4. AE training utilizes a weighted composite loss combining the perceptual LPIPS, a GAN component for realistic texture generation, and the Wasserstein Distance (WD) for latent space regularization, ensuring a structured and meaningful latent space. Given that the LPIPS network is pre-trained only for 3-band inputs, the methodology incorporates a band permutation strategy, randomly selecting 3 of the 4 multispectral bands for LPIPS calculation at each training step, a strategy validated to maintain metric accuracy.

The Denoising UNet iteratively refines the noisy image, conditioned on the Low-Resolution (LR) input. A critical adaptation for remote sensing, termed explicit LR encoding, enhances spectral coherence. Instead of conditioning the UNet with the original LR image in RGB-NIR space, the LR image is interpolated, encoded by the AE, and then fed to the UNet. This relieves the UNet of the burden of translating between the spectral domain and the encoded latent space. Inference is performed using DDIM sampling, starting from a noise distribution conditioned on the encoded LR input. The overall Single Image SR (SISR) training strategy for RGB-NIR uses sequential training: starting with a large Computer Vision dataset, followed by fine-tuning on a degraded Sentinel-2 dataset for spectral property adaptation, and concluding with the specialized SEN2NAIP dataset for precise learning.

Uncertainty Estimation in SR models

Uncertainty estimation is a vital objective for promoting trustworthiness and accountability, especially for generative Super-Resolution (SR) models like Latent Diffusion Models (LDMs), which operate on the ill-posed problem of mapping a single Low-Resolution (LR) input to multiple plausible High-Resolution (HR) possibilities. The uncertainty is generally understood in two forms: Aleatoric uncertainty (noise or inherent information loss in the observation) and Epistemic uncertainty (model ignorance or variations between plausible outputs).

The OpenSR project quantifies this uncertainty by exploiting the inherent probabilistic nature of Diffusion Models to generate pixel-level uncertainty maps. This involves running the SR model multiple times (e.g., 10x sampling) for a single LR input to collect a distribution of super-resolved pixel values. Based on these samples, a confidence interval is constructed for each super-resolved pixel. The width parameter is derived directly from the standard deviation of the pixel values across the samples, assuming a Gaussian distribution. A larger interval width signifies higher diversity in sampling and therefore lower confidence, typically occurring in areas with high-frequency content or complex textures where the model is required to be more generative.

Evaluation on the SEN2NAIP validation dataset demonstrated a strong correlation between wider confidence intervals and the presence of high-texture areas, such as houses or fine field lines, where the model is prone to hallucinations. This confirms the practical utility of the uncertainty map in identifying potentially unreliable reconstruction areas. While the generation of these multiple samples significantly increases the computational cost (by a factor of 10 to 20), this pixel-wise uncertainty metric is considered crucial for enabling users to reliably gauge the applicability of the SR product for sensitive downstream tasks, such as agricultural field delineation.

The LDSR-S2 model and weights from the paper “Trustworthy Super-Resolution of Multispectral Sentinel-2 Imagery with Latent Diffusion” are packaged up in the OpenSR-Model repository, making it possible to openly use the latent-diffusion model: https://github.com/ESAOpenSR/opensr-model.

OpenSR SEN2SR Framework

SEN2SR is the OpenSR radiometrically and spatially consistent super-resolution pipeline for all Sentinel-2 bands. Unlike traditional SR methods that risk hallucinating false textures or introducing spectral distortion, SEN2SR is built on a physically grounded and explainable framework, integrating:

• Low-frequency hard constraints (Fourier-based) to enforce realistic spectral structure.
• Explainable AI (xAI) techniques to understand pixel-wise model behavior.
• A Wald-based two-stage refinement for the coarser bands, leveraging the high-res RGB+NIR for guidance.

Most deep learning super-resolution models struggle when transitioning from synthetic training data to real-world satellite imagery. This often leads to undesirable effects such as visual artifacts, spatial misalignments, and spectral hallucinations that compromise the reliability of downstream applications. SEN2SR addresses these challenges through a carefully engineered pipeline. It is trained on SEN2NAIPv2, a harmonized synthetic dataset explicitly designed to mimic the spectral and spatial characteristics of Sentinel-2 imagery, thereby minimizing domain shift. Furthermore, the framework enforces a rigorous super-resolution protocol that constrains the reconstructed low-frequency components of the output to align with the original Sentinel-2 input, ensuring physical and spectral consistency. As a result, SEN2SR delivers robust and artifact-free outputs across a wide range of land cover types – from densely built urban environments to heterogeneous agricultural regions – maintaining both structural fidelity and radiometric realism.

The SR baseline models, developed within the SEN2SR Framework, adhere to a standardized, performance-driven architectural pipeline featuring four main stages: shallow feature extraction, deep feature extraction, image reconstruction, and critically, the final SR hard-constraint stage. Three distinct deep learning architectures were extensively evaluated across various size configurations (Lightweight, Small, Medium, Expanded, and Large) to identify the most robust performers: Convolutional Neural Networks (CNNSR), Swin Transformer V2 (Swin2SR), and Mamba State-Space Model (MambaSR).

A core innovation across all architectures is the low-frequency hard constraint layer, implemented as the final layer. This layer enforces physical consistency, accelerating convergence and restricting the solution space to plausible outcomes. It ensures two conditions are met: (1) reflectance values remain non-negative (via ReLU activation), and (2) the super-resolved output, when degraded back to the LR scale, precisely matches the original LR input. This consistency is achieved in the Fourier domain by combining the original low-frequency content of the LR image with the new high-frequency components generated by the SR network, using a Gaussian filter which was empirically proven to offer the best performance (PSNR) and smoothest transition for this fusion.

Moreover, the OpenSR SEN2SR Framework is designed to enhance both the 10 m RGB+NIR and the 20 m red-edge and SWIR bands to a unified resolution of 2.5 meters, ensuring precise spectral fidelity and geometric alignment across the stack. This is implemented through a two-stage refinement based on the Wald protocol for the coarser bands, which are fused with the high-res RGB+NIR super-resolved bands.

The SEN2SR package provides the entry-point for SR product generation. It can take any SR model and perform patched SR on S2 inputs, both for the RGB-NIR 10m bands to create the high frequencies, as well as for the 20m bands in order to preserve the full spectral information. SEN2SR package is available at the GitHub Repository https://github.com/ESAOpenSR/SEN2SR.

OpenSR-Test: A Protocol for Remote Sensing Image SR Evaluation

To overcome the limitations of standard image quality metrics (such as PSNR, SSIM, and LPIPS) which are poorly suited for the spectral fidelity and structural requirements of Super-Resolution (SR) in the remote sensing (RS) domain, the OpenSR project established OpenSR-Test. This comprehensive benchmark toolbox is tailored specifically for evaluating optical RS imagery SR performance by adapting Wald’s classical protocol for image synthesis.

The proposed protocol defines three fundamental properties that an ideal SR product must satisfy:

1. Consistency Property: The SR product, when theoretically degraded to match the original Low-Resolution (LR) scale, must maintain spectral and spatial fidelity with the original LR input.
2. Synthesis Property: The SR output must demonstrably provide a gain in spatial detail (or high-frequency energy) compared to a simple, naively upsampled LR image.
3. Correctness Property: The SR algorithm must minimize the introduction of hallucinations (unrealistic high-frequency features) and minimize omissions (missing high-frequency features present in the High-Resolution, HR, image).

The benchmark relies on rigorously harmonized cross-sensor datasets (including NAIP, SPOT, and VENµS imagery compared against Sentinel-2) where both spatial alignment (using algorithms like LightGlue and DISK) and radiometric consistency (via histogram matching) have been meticulously verified through quality flags and human inspection.

This robust framework provides a trustworthy and context-aware assessment of SR model effectiveness, revealing crucial trade-offs, such as that between reconstruction fidelity (PSNR) and perceptual realism (LPIPS/CLIP).

The OpenSR-Test toolbox is publicly available at: https://esaopensr.github.io/opensr-test/ and the curated datasets at: https://huggingface.co/datasets/isp-uv-es/opensr-test

Research and Development tools

• The core of the development are latent diffusion models and generative adversarial neural networks, which are used for the super-resolution of Sentinel-2 images.
• All implementations are based on PyTorch.
• Everything is open source and ready to use.

Development Outputs

We have published our Python packages designed to improve the spatial resolution of Sentinel-2 satellite images from the 10 and 20 m bands down to 2.5 meters using a set of neural network models, and our SEN2NAIPv2 dataset, used to train the models.

📦 Dataset: https://huggingface.co/datasets/tacofoundation/SEN2NAIPv2

🦾 Model: https://github.com/ESAOpenSR/opensr-model

💻 Code: https://github.com/ESAOpenSR/SEN2SR

Published Journal Papers

“A Comprehensive Benchmark for Optical Remote Sensing Image Super-Resolution” Cesar Aybar, David Montero, Simon Donike, Freddie Kalaitzis, Luis Gómez-Chova. IEEE Geoscience and Remote Sensing Letters, vol. 21, pp. 1-5, 2024, Art no. 5003105, DOI: 10.1109/LGRS.2024.3401394 (ISSN: 1545-598X)

• Paper: https://ieeexplore.ieee.org/document/10530998
• Website: https://esaopensr.github.io/opensr-test/
• Dataset: https://huggingface.co/datasets/isp-uv-es/opensr-test
• Code: https://github.com/ESAOpenSR/opensr-test

“SEN2NAIP: A large-scale dataset for Sentinel-2 Image Super-Resolution” Cesar Aybar, David Montero, Julio Contreras, Simon Donike, Freddie Kalaitzis, Luis Gómez-Chova. Nature Scientific Data, vol.11, Article number 1389, 18 Dec. 2024, pp.1-13. DOI: 10.1038/s41597-024-04214-y (ISSN: 2052-4463)

• Paper: https://www.nature.com/articles/s41597-024-04214-y
• Dataset: https://doi.org/10.57760/sciencedb.17395
• GitHub: https://github.com/ESAOpenSR/opensr-degradation/
• HuggingFace: https://huggingface.co/datasets/isp-uv-es/SEN2NAIP
• Colab Demo:
  Google Colab

“Trustworthy Super-Resolution of Multispectral Sentinel-2 Imagery with Latent Diffusion” Simon Donike, Cesar Aybar, Luis Gómez-Chova, Freddie Kalaitzis. IEEE Journal of Selected Topics in Applied Remote Sensing, vol. 18, pp. 6940-6952. 2025, DOI: 10.1109/JSTARS.2025.3542220.

• Paper: https://ieeexplore.ieee.org/document/10887321
• Model: https://github.com/ESAOpenSR/opensr-model
• Dataset: https://huggingface.co/datasets/isp-uv-es/SEN2NAIP
• OpenSR-Utils: https://github.com/ESAOpenSR/opensr-utils
• OpenSR-Test: https://github.com/ESAOpenSR/opensr-test
• Colab Demo:
  Google Colab

“Near-Infrared Band Synthesis from Earth Observation Imagery with Learned Location Embeddings and Task-Driven Loss Functions” Simon Donike, Cesar Aybar, Julio Contreras, Luis Gómez-Chova. IEEE Journal of Selected Topics in Applied Remote Sensing, vol. 18, pp. 24319-24330. 2025, DOI: 10.1109/JSTARS.2025.3608941.

• Paper: https://ieeexplore.ieee.org/document/11159252

“A radiometrically and spatially consistent super-resolution framework for Sentinel-2” Cesar Aybar, Julio Contreras, Simon Donike, Enrique Portalés-Julià, Gonzalo Mateo-García, Luis Gómez-Chova. Remote Sensing of Environment, Vol. 334, 1 March 2026, 115222. doi: 10.1016/j.rse.2025.115222 [Preprint available at doi: 10.2139/ssrn.5247739]

• Preprint: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5247739
• Paper: https://www.sciencedirect.com/science/article/pii/S0034425725006261
• Dataset: https://huggingface.co/datasets/tacofoundation/SEN2NAIPv2
• Model: https://github.com/ESAOpenSR/SEN2SR
• Colab Demo:
  Google Colab

Conference Communications

“Super-Resolution of Multispectral Sentinel-2 Imagery with Latent Diffusion Models” Simon Donike, Cesar Luis Aybar Camacho, Luis Gómez-Chova, Freddie Kalaitzis. ESA SUREDOS24 workshop "Super-Resolution and Downscaling for EO and Earth Science", ESRIN, Frascati, Italy, 29-31 May 2024.

“SEN2NAIP: Sentinel-2 Super-Resolution Dataset Using a Realistic Degradation Model” Cesar Luis Aybar Camacho, Simon Donike, Freddie Kalaitzis, Luis Gómez-Chova. ESA SUREDOS24 workshop "Super-Resolution and Downscaling for EO and Earth Science", ESRIN, Frascati, Italy, 29-31 May 2024.

“SuperIX: General presentation, dataset and metrics, toolkits and WebGIS portal presentation” Nicolas Longépé, Luis Gómez Chova, Gunnar Brandt, Alfredo Kalaitzis, Simon Donike, Cesar Luis Aybar Camacho. ESA SUREDOS24 workshop "Super-Resolution and Downscaling for EO and Earth Science", ESRIN, Frascati, Italy, 29-31 May 2024.

"Trustworthy Super-Resolution of Sentinel-2 Products Using Latent Diffusion and Their Applicability to Building Delineation and Flood Detection", Simon Donike, Cesar Aybar, Enrique Portalés-Julià, Luis Gómez-Chova, Freddie Kalaitzis. ESA Living Planet Symposium, Vienna, Austria, 23-27 June 2025.

"Super-resolution of all Sentinel-2 bands to 10 meters using parameter-free attention and cross-correlation embeddings", Julio Contreras, Cesar Aybar, Luis Gómez-Chova, Simon Donike, Freddie Kalaitzis. ESA Living Planet Symposium, Vienna, Austria, 23-27 June 2025.

"TACO: Transparent Access to Cloud-Optimized Spatio-Temporal Datasets" Cesar Aybar, Luis Gómez-Chova, Julio Contreras, Oscar Pellicer, Chen Ma, Gustau Camps-Valls, David Montero, Miguel D. Mahecha, Martin Sudmanns, Dirk Tiede. ESA Living Planet Symposium, Vienna, Austria, 23-27 June 2025.