Project Summary

• Goal: build a deep learning pipeline that converts PhiSat-2 Level-0 data, which are uncoregistered, uncalibrated, and noisy, into Level-1C products with geometric/radiometric consistency, suitable for downstream EO tasks.
• Method: first, using phase-correlation to produce L0 products that are co-registered (inter-band); then, using an end-to-end model to perform denoising/destriping and radiometric calibrations needed to obtain the final L1 product.
• Evaluation: evaluated using 2 approaches:
• Standard quality metrics w.r.t. L1C ground truth (MSE, PSNR, SSIM...).
• Performance on downstream tasks. Compare the performance of downstream tasks on the Ground Truth L1 image and the DL-generated L1 image.

Development Tools

• Programming Languages: Python (+ various packages) and PyTorch for development and model training.
• Versioning: GitHub (Private repository).
• Experiment monitoring: Tensorboard.
• Data access: Insula platform to download PhiSat-2’s data (both L0 and L1C).
• Code editor: Visual Studio Code.

Development Outputs

• A set of Python files/scripts containing the whole pipeline, training procedures, evaluation procedures, and all the codebase developed during the project. All of these files are also saved (as a backup) in a private repository on GitHub.
• Intermediate and final slides to present the work and show the advancements.
• Different versions of the used dataset for both training and validation.
• Weights of the models in Torch format.

Project Description

Introduction

When a satellite acquires a RAW image, it is far from clean and usable for any application, task, or analysis. Before actually using EO products, data must be processed to ensure that the final product is clean, co-registered, denoised, and more. The multistep pipeline from raw to processed product is often computationally expensive; in fact, most satellites (with some exceptions) must download raw data to Earth, where it can then be processed on appropriate, powerful hardware. This process can take from hours to several days. In PhiSat-2, the RAW data is labeled as “L0 product”, while the final usable product is labeled “L1 product”.

In this project, we investigate whether the full L0-L1C processing can be learned with deep learning rather than a classical multi-stage pipeline. Our goal is to assess if DL can faithfully model a long, complex chain where multiple tasks, co-registration, denoising, radiometric calibration, etc., are performed and run in parallel by a single model, acknowledging that a single end-to-end model may not capture everything.

If feasible, this approach could offer:

1. Higher image quality: DL has shown the ability to produce high-quality images from noisy, uncalibrated raw inputs (e.g., modern smartphone pipelines).
2. Higher Efficiency: the raw-to-processed pipeline is a very complex and heavyweight pipeline for most of the satellites. Replacing this multistep sequential pipeline with a single DL model or framework could improve overall efficiency.
3. Enabling onboard processing: as mentioned before, the processed data is often computed on the ground (PhiSat-2 is an exception, as it was specifically designed for onboard AI), since it requires many computational resources and auxiliary data. If we move the entire computation on board with an efficient DL framework, we could enable onboard processing for all satellites, and not just for smallsats specifically intended for onboard processing, such as PhiSat2.

Methods and Materials

As our dataset, we are using the PhiSat-2 L0-L1 image pairs dataset, already used in a previous project carried out at the PhiLab on image compression.

The first part of this project focused on an in-depth analysis of the dataset, as high-quality images are important for low-level vision tasks (such as super-resolution, denoising, etc.). Initial analysis of the dataset revealed that many images were very noisy or concentrated in areas with homogeneous textures (clouds, sea, desert, etc.); in such conditions, noise tends to overwhelm the clean signal, which is why an initial cleanup of the dataset was performed by removing these noisy images and also eliminating images where black bands were present (i.e., without valid values).

It should also be kept in mind that there are very high shifts/misalignments between the PhiSat2 spectral bands, up to 40/50 pixels (i.e., up to 250 meters, considering the ground sampling distance of ~5 meters). These shifts render any neural network useless, since computer vision networks, often based on convolutional layers, have small receptive fields (at most 3-9 pixels square) that cannot compensate for such large shifts. Therefore, as an integral part of the pipeline, a coarse co-registration procedure based on phase correlation was implemented before passing the RAW L0 images to the network. The use of this co-registration strategy effectively compensated for these shifts to a large extent, making the neural network usable (as shown in the results).

As for the neural network used to perform the L0-L1 processing task, we chose an architecture that is well-known and proven for image restoration problems: NAFNet (https://arxiv.org/abs/2204.04676). This network, in addition to being an excellent baseline for low-level vision tasks, is also incredibly efficient compared to competing architectures, as it uses very lightweight, simple operations.

The training procedure followed the following approach, which differs greatly from the classic literature: rather than training the model to reconstruct L1C images, which are the final target of the project, the network was trained by feeding it the wavelet transform of L0 images as input and reconstructing the wavelet transform of L1 images. This approach was chosen based on two simple considerations:

1. The computational complexity (i.e., efficiency) of computer vision neural networks is primarily influenced by the number of operations performed in the spatial dimension, i.e., on pixels. As is well known, the wavelet transform of an image provides four subbands of half the size of the original image, with an increased number of channels. This overall reduces the number of operations performed by the network, since the number of spatial values (pixels) is halved.
2. The wavelet transform separates content into a coarse approximation (LL) and oriented details (LH: horizontal, HL: vertical, HH: diagonal). Because PhiSat-2 (and EO imagery in general) can exhibit striping and directionally structured noise, these oriented subbands help the network disentangle stripe artifacts from the true scene signal.

Preliminary Results

We conducted a wide range of experiments to examine how performance varies across different approaches. Below is a legend to help you understand what changes from one setting to another:

• L0 Coregistration: band coregistration based on phase correlation applied to L0 data.
• Cleaned DS: removed from the dataset images with excessive noise, uniform noisy patterns, or missing bands.
• L1 Coregistration: band coregistration based on phase correlation applied to L1 data to correct remaining local misalignments in L1C products.
• Edge Loss: added an edge loss, for training, to the L1 loss (MAE) used as “base” loss to reconstruct loss. Edge loss minimizes the differences between the gradients of the reconstructed L1 data and the L1 GT data, aiming to improve image sharpness and reduce blur.
• Only Flips: during training, applied only horizontal and vertical flips as data augmentations.

The following table shows quantitative results (measured as PSNR and SSIM) of different experiments, using different strategies:

These results show that data quality steps, dataset cleaning, and band co-registration deliver the largest gains. Notably, even nominal L1 products contain residual misalignments; correcting them yields substantial improvements. Edge-focused loss provides a smaller, but consistent boost, and restricting augmentations to flips produced the best scores in this ablation.

Outcome

The outcome of the project is expected to lead to a paper that identifies the main challenges and difficulties of L0-L1 processing using AI, as well as a rigorous quantitative and qualitative analysis of which factors most influence task performance. This is obviously accompanied by a methodology that enables L0-L1 processing via DL models.