Project Summary

• Goal: develop a deep learning raw-to-processed pipeline that reconstructs PhiSat-2 Level-1B products directly from raw Level-0 acquisitions, reducing the CPU bottleneck while preserving image fidelity.
• Method: combine a lightweight learned multispectral band-registration module with a wavelet-domain NAFNet reconstruction backbone. The aligned L0 image is transformed with DWT, reconstructed in wavelet space, and mapped back with IDWT.
• Evaluation: assess L1B reconstruction quality and downstream functional consistency through NDVI reconstruction, using PSNR, SSIM, MS-SSIM, and band-shift errors on OCDataset and PCCDataset.
• Final results: on PCCDataset, the full pipeline reaches 30.49 dB PSNR / 91.23 SSIM for L1B reconstruction and 36.30 dB PSNR / 90.05 SSIM for NDVI reconstruction. On Intel Movidius Myriad 2, it processes a 4096x4096 image in 3.58 minutes, yielding an estimated 3.5x to 6.6x speed-up over the traditional CPU-based preprocessing chain.
• Outcome: a finalized paper manuscript, codebase, trained weights, and benchmarking methodology for onboard L0-to-L1B reconstruction.

Development Tools

• Programming languages and ML framework: Python and PyTorch for model development, training, and evaluation.
• Models and algorithms: custom band-registration network, phase-correlation preprocessing, NAFNet reconstruction backbone, Haar DWT/IDWT wavelet processing, and edge-aware loss.
• Deployment and benchmarking: Intel Movidius Myriad 2 VPU and OpenVINO toolchain for accelerator inference tests.
• Experiment monitoring and development: TensorBoard for monitoring, GitHub private repository for versioning, and Visual Studio Code as a code editor.
• Data access and preparation: Insula/PhiSat-2 data access, paired L0-L1B products, Zenodo dataset releases, and patch-based preprocessing/cleaning scripts.

Development Outputs

• Final paper manuscript: “Raw-to-Processed in Orbit: Fast Onboard Image Reconstruction for EO” published at CVPR Workshop AI4Space.
• Private GitHub codebase containing the complete L0-to-L1B pipeline, training scripts, evaluation scripts, dataset preprocessing routines, and benchmarking utilities.
• Trained model checkpoints for the registration module and the wavelet-domain reconstruction model.
• Final quantitative tables, ablation results, and qualitative reconstructions for L1B and NDVI products.
• Intermediate and final presentation slides summarizing the methodology, deployment results, and future work.

Project Description

Introduction

Onboard Artificial Intelligence is increasingly used in Earth Observation missions to reduce downlink requirements and support low-latency decision-making. However, even when a satellite carries an AI accelerator, raw acquisitions must still be converted into usable products before downstream tasks can be executed. This low-level processing chain includes denoising, radiometric correction, band alignment, and other restoration operations, and it often remains CPU-bound. For resource-constrained CubeSats such as PhiSat-2, this creates a significant bottleneck: the AI accelerator can be available, but the product generation stage still limits the end-to-end latency of the onboard workflow.

The final project addressed the following question: can a learned model reconstruct processed products from raw PhiSat-2 acquisitions with sufficient accuracy and with a runtime compatible with space-qualified AI hardware? The work focused on the L0-to-L1B visual restoration stage. Georeferencing was intentionally kept outside the learned pipeline because it requires ancillary metadata and is better addressed as a separate extension. The objective was therefore not to replace every element of the mission processing chain, but to demonstrate that a lightweight end-to-end deep learning pipeline can approximate the most relevant image-formation and artifact-removal steps.

Compared with the preliminary version of the project, the final work moved from a proof-of-concept reconstruction setup to a complete onboard-oriented pipeline. The final design includes a learned registration stage, a wavelet-domain reconstruction network, an end-to-end training strategy, quantitative evaluation on two dataset variants, downstream NDVI validation, and runtime benchmarking on the Intel Movidius Myriad 2 VPU, the same class of accelerator used onboard PhiSat-2.

Methods and Materials

The experiments rely on paired PhiSat-2 L0-L1B data. The original acquisitions are very large, with a fixed width of 4096 pixels and a height that is a multiple of 4096 pixels. To make training and evaluation tractable, acquisitions were divided into non-overlapping 512x512 patches. A cleaning stage removed samples with invalid pixels, missing content, excessive noise, near-empty regions, or uniform artifacts. This step was important because low-level restoration tasks are highly sensitive to poor target quality. If the target product contains residual artifacts or strong misalignment, the network learns an inconsistent mapping.

Two dataset variants were used in the final experiments. OCDataset contains the original L1B products generated by the standard PhiSat-2 processing chain. PCCDataset is derived from OCDataset by applying additional quality filtering and re-coregistering the L0-L1B pairs using phase correlation, thereby obtaining stronger pixel-wise correspondence between raw and processed products. This distinction is central to the results: the complete learned pipeline performs best on PCCDataset, while the original coregistration conditions in OCDataset introduce a domain gap that mainly affects the upstream band-registration stage.

The final pipeline is composed of two main learned components. First, a band-registration network estimates inter-band shifts for the multispectral L0 input. The model predicts integer horizontal and vertical offsets for each band with respect to a fixed reference band. The estimated shifts are then applied to align the spectral bands before reconstruction. Second, the aligned L0 patch is transformed into the wavelet domain using a single-level 2D Haar discrete wavelet transform. A NAFNet reconstruction backbone operates directly on these wavelet coefficients and predicts the corresponding processed L1B representation. The inverse wavelet transform then maps the output back to image space.

The use of the wavelet domain was retained in the final methodology because it supports both efficiency and artifact separation. Instead of processing the full image directly in pixel space, the wavelet transform splits the signal into subbands at half spatial resolution. This reduces the effective spatial workload of the reconstruction network and separates coarse image content from oriented high-frequency details. This is particularly useful for PhiSat-2 imagery, where vertical striping and Gaussian noise can be more easily isolated in specific subbands than in the original image domain.

Training was performed in stages. The registration module and reconstruction module were first pretrained separately. The complete pipeline was then fine-tuned end-to-end using a joint loss that combines registration accuracy and reconstruction fidelity. The reconstruction objective is based on an L1 loss and an additional edge-aware loss that preserves structural transitions while discouraging stripe-like artifacts. The final augmentation strategy avoids photometric transformations because Earth Observation imagery represents physical measurements. The best reconstruction results were obtained using only horizontal and vertical flips, rather than random rotations plus flips.

Final Results

The final quantitative results confirm that the proposed DL pipeline can reconstruct processed L1B products from raw L0 acquisitions with good fidelity, and that the quality of the coregistration strongly affects the final performance. On OCDataset, the complete end-to-end pipeline reaches 28.05 dB PSNR and 86.00 SSIM for L1B reconstruction. On PCCDataset, where the L0-L1B pairs are better aligned, the performance increases to 30.49 dB PSNR, 91.23 SSIM, and 95.86 MS-SSIM. The same trend is observed on NDVI, which was used as a downstream consistency measure: PCCDataset reaches 36.30 dB PSNR and 90.05 SSIM, compared with 32.76 dB PSNR and 85.27 SSIM on OCDataset.

Table 1 reports the final end-to-end reconstruction performance.

The band-wise analysis showed that registration is the main remaining bottleneck. Band 7 is used as the fixed reference and therefore has zero shift by definition. For the other bands, the registration error increases when the spectral content differs more strongly from the reference, especially in the visible bands. This cross-spectral mismatch propagates to reconstruction quality, explaining why the final results improve significantly when stronger phase-correlation alignment is applied in PCCDataset.

The final ablation studies clarified the contribution of the key training choices. Adding the edge-aware loss improved reconstruction from 31.81 dB PSNR and 95.22 SSIM to 32.25 dB PSNR and 95.40 SSIM. Data augmentation also had a strong impact: using only flips reached 33.87 dB PSNR and 97.07 SSIM, whereas random rotations plus flips achieved 32.25 dB PSNR and 95.40 SSIM. This confirms that preserving the vertical orientation of the stripe noise helps the model learn a more stable destriping prior.

A central result of the collaboration is the deployment-oriented benchmark. The proposed pipeline runs almost entirely on the Intel Movidius Myriad 2 VPU, with only the integer shift application executed on the CPU. For a full 4096x4096 image, the average runtime is 3.58 minutes. The reconstruction module accounts for 2.04 minutes, the registration module for 1.34 minutes, and the remaining time is due to DWT/IDWT, I/O, and runtime overhead. This compares favorably with the traditional PhiSat-2 onboard CPU preprocessing stage, which takes approximately 12.35 to 23.75 minutes depending on the target processing level. Overall, the learned pipeline provides an estimated 3.5x to 6.6x speed-up.

Qualitatively, the reconstructed products preserve the main spatial structures and radiometric patterns of the processed L1B targets. The NDVI evaluation further indicates that the reconstructed images retain functional consistency for a relevant Earth Observation downstream product. The wavelet-domain model also suppresses vertical striping more effectively than an equivalent image-domain variant: column-wise statistics remain closer to the processed L1B reference, while the image-domain model exhibits larger deviations and residual column-dependent drift.

Outcome and Future Developments

The project's outcome is a complete DL-based raw-to-processed pipeline for PhiSat-2, accompanied by a scientific manuscript. The work demonstrates that learned L0-to-L1B reconstruction is a viable route for accelerating onboard product generation, especially when the pipeline is designed around the constraints of space-qualified AI accelerators rather than ground-based hardware.

The collaboration also identified clear future directions. The first is to improve registration accuracy. The current approach estimates shifts relative to a single fixed reference band, but a cascaded or pairwise registration strategy could compare each band against a spectrally closer one, reducing cross-spectral mismatch and improving alignment. The second direction is to further reduce residual denoising artifacts, particularly salt-and-pepper texture in the high-frequency HH wavelet subband. The third direction is to address the consistent blur observed in the blue band through a dedicated deblurring component. Finally, extending the learned processing chain toward fully georeferenced products would require incorporating ancillary orbit and attitude metadata, potentially through a multimodal formulation.

Overall, the final results show that the project successfully moved beyond a preliminary feasibility analysis. It produced a deployable, benchmarked, and scientifically documented pipeline that reconstructs usable L1B products from raw PhiSat-2 data with substantial latency reduction while maintaining sufficient fidelity for downstream EO analysis, such as NDVI estimation.