Project Summary
- Design the first onboard pipeline for real-time segmentation of Thermal anomalies in Raw Sentinel-2 Images with Deep Learning
- Develop SegTHRawS, the first segmentation dataset for thermal hotspots detection in raw Sentinel-2 images.
- Improve the actual THRawS dataset for thermal hotspots detection in raw Sentinel-2 imagery by adding more than 170 new scenes. These new images improve THRawS’s global coverage, as they cover the missing areas of the original dataset.
- Design and apply DL segmentation models for onboard satellite implementation.
- Implement the developed DL model in CubeSat edge computing hardware
Development Tools
- PyTorch Lightning, as the main software package for the AI code development part
- Gantt Chart, to effectively measure the project progress.
- GitHub, to effectively keep track of the code.
- Overleaf, to write the report.
- THRawS: Dataset of raw Sentinel-2 imagery provided by Dr. Gabriele Meoni and Ir. Roberto Del Prete (ESA Φ-lab)
PyRawS: Python library developed by Dr. Gabriele Meoni and Ir. Roberto Del Prete (ESA Φ-lab) to handle raw Sentinel-2 imagery from the THRawS dataset
Development Outputs
- GitHub repository: Not available yet, but it will be published in https://github.com/Ubotica/SegTHRawS_ext
- Dataset: Not public yet, it will be published in Zenodo. [Link not available yet]
- Thesis report: Embargo of two years according to the university and third parties. It can be requested to the authors.
Project Description
Detecting thermal hotspots has been done for centuries through ground-based approaches, and implemented for aerial and satellite approaches in the last century. Ground- and aerial-based solutions despite their real-time capabilities, present a limited range and have difficulties when monitoring volcanic activity. Earth Observation (EO) satellite-based approaches stand out as potential solutions for global coverage and precise real-time applications. The main limitation is the trade-off between revisit time and spatial resolution, and the existing delay between image acquisition and the thermal event being detected.
Based on these needs, the project started with a thorough review of the literature, where the fundamental concepts for thermal hotspots segmentation in satellite imagery were presented. This study identified a lack of research in onboard Artificial Intelligence (AI) approaches in raw multispectral imagery for thermal hotspots segmentation, which would considerably improve the real-time detection capabilities of current solutions. The main reason for the absence of research is attributed to the lack of available thermal hotspots segmentation datasets on raw multispectral imagery until 2023.
This project aims to overcome these limitations by providing the first real-time end-to-end onboard payload processing pipeline for thermal hotspots Deep Learning (DL) segmentation on raw multispectral imagery. The end-to-end pipeline covers from creating the first dataset for thermal hotspots segmentation in raw multispectral imagery, known as SegTHRawS, until the onboard detection of the thermal event. Furthermore, this pipeline was designed to provide real-time capabilities in edge computing CubeSat hardware. A CubeSat-oriented design positively affects its applicability and potential use in constellations to provide low revisit times and global real-time detection.
The final design of the onboard payload processing pipeline is mainly composed of two parts: image pre-processing and AI segmentation inference. The image pre-processing is executed in the OnBoard Computer (OBC), the Raspberry Pi 3B+, and consists of multi-band co-registration followed by image tiling. The AI segmentation inference is executed in the AI accelerator, the CogniSAT-XE2 board, and includes a DL segmentation network that performs the thermal hotspots detection part of the pipeline.
SegTHRawS, the first segmentation dataset for thermal hotspots detection in raw multispectral imagery, was created to train the segmentation models. A main dataset consisting of globally distributed Sentinel-2 raw images was generated from THRawS, and then a training dataset was derived using a geographical split. This dataset contains a 50/50 proportion of event and non-event images, and the geographically split dataset reserves specific parts of the Earth, known as scenes, for the testing images. This approach improves the generalization capability of the trained model, as those parts of the Earth's surface were not considered during training and validation.
Apart from SegTHRawS, this project has improved the THRawS dataset previously created by Dr. Gabriele Meoni and Ir. Roberto Del Prete. This improvement adds to the actual dataset more than 170 new scenes with thermal events. These new scenes cover the missing areas of the original THRawS and add missing events in the covered areas.
The segmentation masks were created through an automatic labelling process. This consists of using a strict majority voting that combines the output of five thermal hotspots threshold methods: four gathered from the literature and one designed in this project. The four methods from the literature are based on processed satellite imagery, while the one developed in this project is specifically designed for raw multispectral satellite imagery.
The approach followed for the segmentation masks is weakly segmentation labeling. This consists of using an additional class for those pixels for which the class is unsure and avoiding them during model training. Through the majority voting a pixel is classified as an event if four out of five methods agree, a potential event if two or three out of five methods agree, and a non-event for the remaining cases. This approach enables the creation of a coarse segmentation dataset without requiring visual inspection from experts, which would have required considerable time and financial investments.
Concerning the segmentation network, a comprehensive study was carried out during the literature review to find suitable segmentation networks for onboard approaches. From this study, the resultant architectures were U-Net, DeepLabV3+, Attention U-Net, Residual-Attention U-Net, and U-Net 3+, in combination with small encoders such as MobileNetV2, EfficientNet, and MobileOne. In addition to these networks, a set of modified networks based on the U-Net was created for onboard purposes.
These modifications remove the last level of the original architectures and reduce the number of filters per level, to improve the inference time of the final model. These networks reduce the model size from >60 MB to kBs and have five different versions depending on the number of filters per level used.
These networks were trained with the geographically split dataset. The report includes an analysis of the main hyperparameters that influence the training process of a segmentation network. To determine the final model used for the system, the DL models trained were implemented in the CogniSAT-XE2 to measure their inference time in the final hardware. Concerning the final detection and time performance, the final selected model was the ResUNet-S2, one of the modified models created in this project.
Finally, the entire onboard payload processing pipeline system was implemented and tested in the Raspberry Pi 3B+ and the CogniSAT-XE2 to verify the system and subsystem requirements. The results showed the outstanding performance of this model in terms of detection and real-time capabilities concerning state-of-the-art thermal hotspots solutions. Additionally, to demonstrate that the detection performance was caused by model overfitting, the ResUNet-S2 was retrained in a different dataset and obtained better performance than the network in the dataset's original article.
This pipeline allows for real-time detection of thermal hotspots with global coverage in CubeSats. It significantly reduces the delay between image acquisition and thermal event detection, reducing the time performance from hours to minutes. In single-satellite missions, this solution enables real-time detection of the acquired images without global coverage, as a single EO satellite with high spatial resolution cannot cover the entire Earth in real-time.
This solution becomes significantly important when deployed in satellite constellations, that address the revisit time limitation with more satellites.
The main limitation of this project lies in the segmentation masks used for training. The quality of these masks is limited due to the majority voting approach, which combines methods not properly designed for raw multispectral imagery. Some images present bigger segmentation masks than those obtained from visual inspection, which can negatively impact the detection performance of the trained model. Other limitations are the need for an onboard georeferencing system to complete the onboard pipeline, and the limited sensor generalization considered in the design.
Future work of this project may involve the improvement of the time and generalization capabilities of the pipeline. These can be addressed through further code optimization and the inclusion of new imagery during training. Additionally, a potential next step could involve extending the pipeline to cover different spectral bands and sensors and deploying the pipeline for an on-orbit demonstration.