Project Summary

• Developed Easy LandCover (EZLC): 10,000 real ΦSat-2 patches for four-class semantic segmentation.
• Developed a sensor-aware robustness framework covering noise, blur, haze, radiometric gain error, and band misalignment.
• Extended PyNAS with multi-objective optimization for performance, robustness, and efficiency.
• Improved robustness of ΦSatNet.
• Investigated label-free self-distillation for improving encoder robustness without semantic labels.
• Analyzed the relationship between representation robustness and downstream task robustness.

Development Tools

Python – main development language.

PyTorch / PyTorch Lightning – model training and experimentation.

PyNAS – Neural Architecture Search framework.

NSGA-II-style optimization – multi-objective architecture selection.

ΦSatNet – compact pretrained model for ΦSat-2.

TerraMind, ResNet-18/FPN, ViT – comparison models.

ESA WorldCover – source for land-cover supervision.

ΦSat-2 and Sentinel-2 imagery – Earth observation data.

OpenCV / NumPy / SciPy – image and numerical processing.

CKA, MMD, cosine similarity, MSE, t-SNE – representation analysis.

ESA SpaceHPC / NVIDIA GPUs – model training and NAS experiments.

GitHub and Hugging Face – version control and public distribution.

Development Outputs

• Easy LandCover (EZLC) Dataset

https://huggingface.co/datasets/ivarvdspoel/ez_landcover

• ΦSatNet Robustness Experiments

https://github.com/ivarvdspoel/fm_robustness_eo

• Robust Neural Architecture Search

https://github.com/ivarvdspoel/nas_robustness_eo

• EZLC Dataset Construction

https://github.com/ivarvdspoel/ezlc_construction

• Master’s Thesis: Optimizing Onboard Earth Observation AI for Robustness against Sensor Degradations
• Upcoming outputs: additional downstream-task evaluation, improved self-distillation, larger robustness-aware NAS experiments, and hardware-specific evaluation.
• Publications: no peer-reviewed publication from the collaboration yet.

Project Description

This project was conducted as a Master's thesis in collaboration between Leiden University and the Φ-lab, focusing on robust neural networks for ESA's ΦSat-2 mission. The research addressed an important challenge for AI deployed directly on board Earth observation satellites: models are generally developed using well-processed imagery, whereas images encountered during operation can be affected by sensor characteristics, atmospheric conditions, calibration errors, and imperfections in onboard preprocessing. The objective of the project was to investigate how these data distribution shifts affect compact onboard AI models and how to improve their robustness.

ΦSat-2 demonstrates the use of artificial intelligence directly onboard an Earth observation satellite. Instead of transmitting all acquired imagery to the ground before processing, onboard AI can identify relevant observations and produce useful information closer to the moment of acquisition. This can reduce downlink requirements and shorten response times for applications such as disaster response, wildfire monitoring, maritime monitoring, and land-cover analysis. At the same time, onboard deployment imposes strict constraints on model size, memory consumption, computational cost, and inference latency. The project, therefore, considered robustness alongside the requirement for compact and efficient models.

The first part of the project concerned the lack of labeled downstream tasks for real ΦSat-2 imagery. Existing work within the ΦSat-2 project provided aligned real ΦSat-2 and Sentinel-2 observations, along with associated metadata. This thesis used these aligned products to transfer labels from ESA WorldCover to the ΦSat-2 image grid.

Using this, the project developed Easy LandCover (EZLC), a semantic-segmentation dataset consisting of 10,000 real ΦSat-2 image patches. ESA WorldCover classes were aggregated into four broader classes: background, vegetation, built-up surfaces, and water. Patch selection was designed to avoid strongly homogeneous scenes and to approximately balance the aggregate number of pixels belonging to each class. This resulted in a practical downstream task for training and evaluating models on real ΦSat-2 imagery rather than relying only on simulated observations.

In the figure above, a sample from EZLC is perturbed using brightness, and 4 different models produce 4 different semantic segmentation labels.

The next part of the project developed a sensor-aware robustness evaluation framework. Five controlled perturbations were selected from the literature to represent changes that can occur within the sensor, acquisition environment, or onboard processing chain. These consist of band-dependent sensor noise, optical blur, atmospheric haze, radiometric gain error, and residual band co-registration error. The ΦSat-2 sensor properties informed their formulations and strengths, as well as empirical measurements from the acquisitions, sensor simulation work, and existing Earth observation robustness literature.

Each perturbation modifies the input image while leaving its underlying semantic label unchanged. This enables a controlled comparison between an original acquisition and a degraded version of the same scene. Models were evaluated not only on their performance but also on prediction consistency: how similar the models' predictions remain before and after degradation. Different perturbation strengths were additionally evaluated to determine how models fail as conditions become progressively more severe.

Two approaches for improving onboard robustness were investigated. The first focused on compact task-specific neural networks using Neural Architecture Search. The project built on ESA's PyNAS framework and changed the search strategy with multi-objective optimization, with objectives of clean segmentation performance, prediction consistency under perturbations, and computational efficiency.

In the figure above, a PhiSat-2 Search is illustrated, and the pareto-dominant models are presented.

This multi-objective formulation is particularly useful for onboard AI because the architecture with the highest accuracy is not necessarily the best architecture for deployment. Similarly, prediction consistency alone is insufficient: a model can produce nearly identical predictions under clean and perturbed conditions while consistently producing incorrect predictions. Considering performance, robustness, and efficiency together therefore provides a more realistic way of selecting models for resource-constrained onboard use.

After the search, the highest clean-performing robustness-aware NAS architecture was chosen. This results in a model which is extremely compact, with fewer than 0.1 million parameters. Compared with the clean-trained NAS model, mean perturbed mIoU increased from 0.48 to 0.54, while mean prediction consistency increased from 0.78 to 0.89. Clean mIoU decreased only moderately, from 0.60 to 0.57. These results show that a very small task-specific network can be made substantially less sensitive to the expected input degradations.

The second model-development direction investigated ΦSatNet, a compact pretrained model intended for ΦSat-2 imagery. Unlike task-specific NAS, ΦSatNet is a foundation mode: a shared pretrained encoder learns representations of Earth observation imagery, after which task-specific decoders can use these representations for applications such as segmentation. The research question was therefore not only whether ΦSatNet could be made robust on the EZLC task, but whether robustness could be introduced directly into the reusable pretrained encoder.

In the figure above, the loss of the adapted encoder with self-distillation is shown.

Two adaptation strategies were evaluated: the first used supervised perturbation-aware fine-tuning. A clean-trained segmentation decoder was kept frozen while the ΦSatNet encoder was adapted, exposing the model to the sensor-aware perturbations. Keeping the decoder fixed meant that improvements had to originate from changes to the encoder representation rather than from the task head adapting to the perturbations. This approach increased mean perturbed mIoU from approximately 0.47 to 0.54 and prediction consistency from 0.81 to 0.91, while maintaining a clean mIoU of approximately 0.57.

The second approach investigated whether robustness could be introduced without semantic labels during adaptation. A multi-layer self-distillation method was developed in which a frozen, clean ΦSatNet encoder served as the teacher. The teacher processed the original observation, while a student encoder received a perturbed version of it. The student was trained to produce internal representations resembling those the teacher produced for the clean image. Because the objective operates on representations rather than semantic-segmentation labels, the approach could potentially be applied when labeled data are unavailable.

Self-distillation produced a smaller but measurable improvement. Mean perturbed mIoU increased from 0.47 to 0.48, and prediction consistency increased from 0.81 to 0.84, while clean mIoU decreased from 0.57 to 0.55. Although supervised adaptation was clearly more effective when labels were available, the self-distillation results demonstrate that some robustness can be introduced directly into a pretrained representation without using semantic labels during the robustness-adaptation stage.

The project additionally investigated what happens inside the ΦSatNet encoder when it is adapted for robustness. Clean and perturbed representations were compared across multiple encoder stages using Mean-Squared Error, cosine similarity, Centered Kernel Alignment, and Maximum Mean Discrepancy, together with low-dimensional visualization. An important outcome was that representation invariance is not equivalent to downstream robustness: self-distillation generally made clean and perturbed representations more similar, but this did not always translate into improved semantic-segmentation performance. Supervised adaptation improves task performance even when representation-similarity metrics showed weaker improvements. This indicates that representation metrics are useful diagnostics, but should not replace direct downstream evaluation.

In the figure above, the performance of perturbed images (y-axis) and the clean performance (x-axis) are illustrated. Model sizes are shown by the size of the dot (on a log scale), with a regression line indicating expected robustness relative to clean strength.

Overall, the collaboration resulted in a complete workflow for studying robustness of compact AI on real ΦSat-2 imagery: a real-image downstream dataset, sensor-aware perturbation methods, robustness-aware architecture search, supervised and label-free foundation-model adaptation, and a set of evaluation methods for both predictions and learned representations. Sensor-aware perturbation training substantially improved both compact NAS models and ΦSatNet, while label-free self-distillation provides a promising direction when semantic supervision is unavailable.  The central finding is that robustness should be explicitly considered during the design and adaptation of onboard AI models, rather than inferred solely from clean-image performance.