Project Description

Abstract

• This activity designed, implemented, validated, and exploited Explainable Artificial Intelligence (XAI) methods for Earth Observation (EO) tasks, with an emphasis on hyperspectral images across various applications.
• We showed the kinds of benefits XAI methods might bring to EO (including understanding a model's shortcomings, compressing it for on-board processing, improving its operational capabilities, and more). We built new classical and deep machine learning models for methane detection and estimation of soil parameters from hyperspectral images.
• We built new AI-ready datasets, coupled with the training-test dataset splits for both soil parameters’ estimation from remote sensing images, as well as methane detection. These datasets incorporate a variety of real-world EO scenarios, including image noise contamination and different atmospheric corrections.
• We built meteors – a software package that includes implementations of XAI methods, along with tutorials demonstrating the potential benefits of using XAI for EO.
• We built a roadmap summarizing the most promising research pathways to pursue based on the results of this activity.
• We organized a workshop at ECAI 2025 focusing on XAI for EO. We organized the HYPERVIEW 2 Challenge, which combines the EO analysis task (soil parameters’ estimation from airborne and satellite data – both multi- and hyperspectral) with XAI.

Detailed description

Introduction

In this project, called PINEAPPLE (exPlaINablE Ai for hyPersPectraL imagE analysis), we addressed the important research gap of lack of “trust” in (deep) machine learning algorithms for Earth Observation (EO) by tackling two real-life EO downstream tasks (estimating soil parameters from hyperspectral images and detecting methane in such imagery) using new deep and classic machine learning algorithms empowered by new explainable AI (XAI) techniques. Additionally, we investigated other downstream tasks, including cloud semantic segmentation and hyperspectral unmixing, with the goal of developing trustworthy AI methods and tools for practical EO scenarios. We believe that PINEAPPLE was an important step toward not only “uncovering the magic” behind deep learning algorithms (hence building trust in them in EO downstream tasks) but also showing that XAI techniques could be effectively utilized to improve such data-driven algorithms (both classic and deep machine learning-powered), ultimately leading to better algorithms. Finally, we put special effort into (i) unbiasing the validation of existing and emerging algorithms by ensuring their full reproducibility (both at the algorithm and at the data level), and (ii) understanding and improving the generalization of such algorithms when fundamentally different data was used for testing (e.g., noisy data, data with simulated atmospheric conditions, or data captured in different areas/times, and so forth). In Figure 1, we present a visual summary of the selected algorithms and methods that were explored in PINEAPPLE. At the same time, we wish to emphasize that the approaches developed in this project are generic and readily applicable to other EO downstream tasks and data modalities.

Figure 1: Schematic description of the groups of selected models and explanations explored in this study.

AI-ready datasets

In PINEAPPLE, we aimed to create AI-ready datasets. By an AI-ready dataset, we mean the dataset containing:

• The training-test dataset split that should be followed while developing data-driven algorithms,
• The downstream task (or a set of downstream tasks) that can be targeted using the dataset (e.g., regression, binary classification, multi-class classification, and so forth),
• The quantitative metrics that should always be calculated to compare the algorithms over this dataset.

In addition to the above, the AI-ready dataset may include additional layers of information (alongside the “raw” data and ground truth), such as detailed information about the distribution of data samples, additional ground-truth information, and other details.

Retrieving soil parameters from hyperspectral images

We built upon our previous work stemming from GENESIS (another ESA-supported project), in which we developed the AI-ready dataset HYPERVIEW. HYPERVIEW was used in the IEEE ICIP 2022 HYPERVIEW Challenge and was later summarized in the IEEE GRSM paper (https://ieeexplore.ieee.org/abstract/document/10526314). The original HYPERVIEW dataset included hyperspectral patches with 150 bands, along with the original training/validation/test split and the HYPERVIEW score. Other components rendered below (Figure 2) have been newly introduced in PINEAPPLE, in the HYPERVIEW-PINEAPPLE dataset, and they relate to the data (in green), dataset splits (yellow), and an additional layer of information that might be used in the validation procedures (light orange). These additional layers of information included (i) the in-orbit image simulations (reflecting the Intuition-1 imagery – Intuition-1 is a cube sat developed by KP Labs, Poland, and put into operations on November 11, 2023), (ii) different atmospheric variants simulated in hyperspectral images, (iii) contamination of clean images with data-level parameterized noise of different distributions (Gaussian, Poisson, impulsive), reflecting a variety of on-board scenarios (e.g., thermal-related noise or sensor corruption), and (iv) an anonymized spatial information layer, incorporating the spatial relationships between the parcels, without revealing the information concerning their exact locations.

Figure 2: The HYPERVIEW-PINEAPPLE dataset, and its components that relate to the data (in green), dataset splits (yellow), and an additional layer of information which might be used in the validation procedures (light orange).

Detecting methane in hyper/multispectral images

In PINEAPPLE, we built upon the publicly available hyperspectral image database (AVIRIS-NG sensor) dedicated to methane detection and segmentation (D. R. Thompson et al., “IS-GEO Dataset JPL-CH4-detection-2017-V1.0: A benchmark for methane source detection from imaging spectrometer data,” JPL Document D-101028, 2017), for which we perform a thorough quality check (accompanied with building a co-registration tool for dealing with image/methane enhancement maps’ misalignments), and introduced a new AI-ready dataset based on this airborne data. Additionally, we acquired Sentinel-2 multispectral images for the regions imaged by AVIRIS-NG (however, exact temporal co-registration was not possible due to a mismatch between the data acquisition campaigns). Finally, we developed an algorithm for generating binary ground-truth methane masks from, e.g., methane enhancement maps. We used the data simulators to synthesize in-orbit data acquisition conditions in AVIRIS-NG images. Within PINEAPPLE, we worked on the first, multi-modal dataset for methane detection, incorporating image data (also simulated), textual information (provided by human practitioners and large language models) and land use land cover maps (generated by the TerraMind foundational model: https://arxiv.org/abs/2504.11171) – this dataset can be used not only to build multi-modal methane detection systems, but also to verify model biases and to answer various hypotheses (e.g., is the methane detector biased toward industrial areas, even if they do not contain methane?). The AI-ready methane detection dataset included definitions of quality metrics that must always be calculated when using this dataset.

AI solutions for EO

In PINEAPPLE, we designed and implemented an array of AI models for various EO downstream tasks, particularly for retrieving soil parameters from remote sensing (hyperspectral) data and detecting methane in EO data (both single- and multi-modal). In both soil composition analysis and methane detection, we designed and implemented classical and deep learning algorithms (some of them incorporating additional priors and available expert knowledge concerning the downstream task of interest), also including the recent foundational models (TerraMind). All algorithms were thoroughly verified using the developed AI-ready datasets. It is important to emphasize that within PINEAPPLE, we exploited the feedback loop between the explainable AI and modeling components. The outcome of the XAI analysis could affect the way an AI is built (to make it more robust, compact, optimized, or trusted), and vice versa – AI models and their characteristics were used to design XAI techniques that are appropriate for EO tasks.

XAI techniques for EO

There are many ways to catalog methods for model explanation. The most popular are based on the question “how” a particular technique works. Such an approach distinguishes, for example, the class of model-agnostic methods (that treat the model as a black box without any assumptions about its structure) from model-specific methods, to determine whether a particular explainability technique will work for any model or has certain assumptions or restrictions on the model architecture. Another classification based on the “how” question is the split into local and global techniques for model explanations. Local techniques focus on explaining the behavior of a model for a single observation, while global techniques describe the average behavior of a model for an entire dataset.

In PINEAPPLE, explainability techniques were grouped differently, depending on the answer to the question “what” is being explained, and what aspects of the model we analyze with XAI techniques. Given this question, we will distinguish four classes of explanations. Explanations focused on model performance, most often based on analysis of model residuals. Explanations focused on variable importance for models trained on data after additional processing to extract interpretable variables. Explanations focused on the importance of bands; they will be useful for extracting knowledge related to spectral bands. Explanations focused on the importance of spatial elements, so-called heatmaps – the example outcomes of two selected XAI techniques are presented in Figure 3.

(b) Investigating the spatial importance of super-pixels while predicting soil parameters

Figure 3: Example XAI techniques developed in PINEAPPLE, which can help understand (a) which features are important in classical AI models (the other may be safely pruned without negatively affecting the operational capabilities of the model), and (b) which parts of an input image contain the “important” information.

In PINEAPPLE, we designed and implemented both model-agnostic and model-specific explanation algorithms to interpret machine learning models (both classical and deep) for estimating soil parameters and detecting methane (but, importantly, these methods apply to other downstream tasks in Earth observation). We proved that high-level products developed using XAI techniques may add value to the “end users” (algorithm developers, product end users, and so forth). All techniques and tools were thoroughly verified.

We developed the Meteors package available at https://github.com/xai4space/meteors. It includes practical examples on how to use the developed software and ideas. We believe our efforts are an important step toward deploying XAI techniques for good reasons (e.g., to understand AI models, improve them, optimize them, and effectively compress them). It is also important to mention that we have built a roadmap for XAI in EO. In this roadmap, we showed how XAI techniques may be adopted in practical Earth Observation use cases, what areas of research and development are promising (with respect to developing new XAI methods, building them for “a reason”, targeting new downstream tasks, using XAI methods for compressing, optimizing, and robustifying the models, and so forth). Finally, we wish to emphasize that XAI methods may be of practical utility to various actors, such as AI Model Developers, AI Researchers, EO Researchers, End-Users, and many more – it has been highlighted within PINEAPPLE. Here are some examples on the ESA Φ-lab CIN Website: Example 1, Example 2

Research and Development Tools

• Explainable AI (XAI) methods: meteors (an open-source package for creating explanations of multi- and hyperspectral images) was developed in Python primarily for PyTorch models and was inspired by the Captum library (https://captum.ai/).
• Classical and deep machine learning models for EO: The models developed in this activity were built in Python (with the PyTorch backend) and in MATLAB.
• Data simulators: To synthesize real-world in-orbit acquisition scenarios, we built a software tool (in Python), utilizing the Py6S library (for simulating atmospheric conditions), as well as our noise-contamination scenarios, following various parameterized noise distributions (Gaussian, Poisson, impulsive).
• Experimental reproducibility: All experiments performed within this activity are fully reproducible. While developing methods, tools, and software packages, we used MLflow to run, track, and analyze experiments.
• ESA Φ-lab AI4EO Challenge platform (https://platform.ai4eo.eu/): To host and run the HYPERVIEW 2 Challenge at ECAI 2025, we used the AI4EO platform.

Project Outcomes

• Project webpage: https://xai4space.github.io/
• Code: https://github.com/xai4space/meteors (meteors - an open-source package for creating explanations of hyperspectral and multispectral images)
• Code: https://huggingface.co/KPLabs/LightweightML-BareSoilDetection (Lightweight bare soil detection in hyperspectral images, ready to be deployed on board satellites)
• Graphical introduction to XAI in EO: https://www.dropbox.com/scl/fi/1n82yos6nsbpqgrr2n5gr/Pineapple-graphical-overview-v3-20250103.pdf?rlkey=blwwziqsfbhfthlpnl0gvxage&st=7cu51gr0&dl=0 (the comics titled “Decoding Earth’s Layers: Graphical Introduction to Explainable AI for Earth Observation Application to Hyperspectral Imagers)
• HYPERVIEW 2 Challenge: https://challenges.philab.esa.int/portfolio/easi-workshop-hyperview2/ (Challenge platform used at ECAI 2025: https://challenges.philab.esa.int/portfolio/easi-workshop-hyperview2/, Challenge permanently-open platform: https://platform.ai4eo.eu/hyperview2-permanent)

Publications

• A. M. Wijata, B. Ruszczak, L. Tulczyjew, N. Longépé, J. Nalepa; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, 2026, pp. 868-876, https://openaccess.thecvf.com/content/WACV2026W/GeoCV/html/Wijata_The_Kingdom_is_Naked_Lightweight_Machine_Learning_for_Hyperspectral_Bare_WACVW_2026_paper.html
• A. M. Wijata, N. Longépé, M. W. Przewozniczek, and J. Nalepa, “Machine Learning and Genetic Algorithms: An Intricate Relationship for Locating Methane in Satellite Images,” In Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO '25 Companion). Association for Computing Machinery, New York, NY, USA, 931–934. https://doi.org/10.1145/3712255.3726701, https://dl.acm.org/doi/10.1145/3712255.3726701
• A. M. Wijata, N. Longépé and J. Nalepa, "Tracking the Invisible Enemy: Methane Detection Using Machine Learning Methods," IGARSS 2025 - 2025 IEEE International Geoscience and Remote Sensing Symposium, Brisbane, Australia, 2025, pp. 1233-1237, doi: 10.1109/IGARSS55030.2025.11242425, https://ieeexplore.ieee.org/document/11242425.
• K. Zieba, J. Nalepa and A. M. Wijata, "Detecting Methane in Satellite Images Using Deep Ensembles," IGARSS 2025 - 2025 IEEE International Geoscience and Remote Sensing Symposium, Brisbane, Australia, 2025, pp. 2517-2521, doi: 10.1109/IGARSS55030.2025.11243616, https://ieeexplore.ieee.org/document/11243616
• A. Miroszewski, J. Nalepa and A. M. Wijata, "Light-Cone Feature Selection in Methane Hyperspectral Images," IGARSS 2025 - 2025 IEEE International Geoscience and Remote Sensing Symposium, Brisbane, Australia, 2025, pp. 2640-2644, doi: 10.1109/IGARSS55030.2025.11243470, https://ieeexplore.ieee.org/document/11243470
• H. Baniecki, P. Biecek, T. Kwiecinski, N. Longépé, J. Nalepa, L. Tulczyjew, A. M. Wijata, V. Zaigrajew. Meteors: Open Source Package for Explanations of Remotely-Sensed Images. Living Planet Symposium 2025.
• H. Baniecki, P. Biecek, T. Kwiecinski, N. Longépé, J. Nalepa, L. Tulczyjew, A. M. Wijata, V. Zaigrajew. The Intuition-1 mission: Explaining and improving on-board hyperspectral image analysis using XAI for Earth observation. Living Planet Symposium 2025.
• V. Zaigrajew, H. Baniecki, L. Tulczyjew, A. M. Wijata, J. Nalepa, N. Longépé, P. Biecek. Red Teaming Models for Hyperspectral Image Analysis Using Explainable AI. ICLR 2024 Workshop on Machine Learning for Remote Sensing (ML4RS). https://arxiv.org/abs/2403.08017