Project Summary

• Custom Triplet Dataset: Built a co-registered multi-source dataset (Sentinel-2, simulated $\Phi$sat-2, and real in-orbit $\Phi$sat-2) to serve as the foundation for pretraining, domain adaptation, and distillation experiments.
• Compact Model Pretraining: Pretrained an efficient Geospatial Foundation Model from scratch (~200k parameters) using Masked Image Modeling (MiM), specifically architected to meet strict onboard satellite compute constraints.
• Downstream & N-Shot Evaluation: Systematically evaluated the pretrained model across multiple Earth Observation tasks (LULC, Floods, Clouds, etc.) using a rigorously stratified N-shot learning protocol to measure scaling laws.
• Domain Adaptation: Developed and evaluated adaptation strategies to bridge the distributional gap between simulated pretraining data and real in-orbit telemetry, successfully testing the adapted model under N-shot constraints.
• Knowledge Distillation Investigation: Currently investigating advanced cross-architecture Knowledge Distillation techniques to further improve upon the from-scratch MiM baseline performance.

Development Tools

• PyTorch & PyTorch Lightning: The core deep learning frameworks used for model architecture design, distributed training, and structuring the complex multi-headed evaluation modules.
• Torchmetrics & Scikit-Learn: Utilized for comprehensive downstream task evaluation (mIoU, Macro/Micro F1-scores) and calculating Domain Gap metrics (training LinearSVCs for Proxy A-Distance).
• Rasterio & GDAL: Essential geospatial libraries used for creating the triplets dataset.
• LightGlue: Employed for the advanced feature-matching and co-registration pipeline between Sentinel-2 and real $\Phi$sat-2 imagery.
• Pandas, Matplotlib & Seaborn: Used extensively for dataset stratification (histogram processing), tracking scaling laws, and generating visualizations.

Development Outputs

• Project Repo : $\Phi$sat-2 Onboard Foundation Model: Pretraining & Adaptation Pipeline
• Dataset: $\Phi$sat-2 Multi-Sensor Triplets Dataset (Sentinel-2, Simulated, Real)
• Publication (In Preparation)

Project Description

1. Context and Motivation

The deployment of Artificial Intelligence directly onboard Earth Observation (EO) satellites, such as ESA's $\Phi$sat-2 mission, represents a paradigm shift in space-based remote sensing. Onboard processing enables real-time decision-making (e.g., disaster response, cloud masking) and drastically reduces downlink bandwidth requirements. However, satellite hardware imposes severe power and memory constraints. The primary objective of this project was to develop a highly compact Geospatial Foundation Model (~200k parameters) capable of achieving competitive performance across various EO tasks while navigating the critical "domain gap" between simulated data used for training on Earth and real telemetry acquired in orbit.

2. Data Engineering: The Triplet Dataset

To systematically enable pretraining, domain adaptation, and distillation, we first constructed a custom, highly curated dataset. This dataset is structured as "triplets," containing:

• Sentinel-2 reference imagery.
• Simulated $\Phi$sat-2 imagery (generated pre-launch).
• Real in-orbit $\Phi$sat-2 imagery.

A major technical challenge was the sub-pixel co-registration of these heterogeneous sources. Using feature-matching algorithms, we aligned the imagery and fetched corresponding ground-truth masks (e.g., ESA WorldCover).

3. Foundation Model Pretraining and N-Shot Evaluation

With the dataset established, we pretrained our compact CNN from scratch. To maximize the extraction of semantic features without relying on vast amounts of labeled data, we utilized Masked Image Modeling (MiM) as our core self-supervised learning paradigm. Once pretrained, the model was evaluated on a comprehensive suite of downstream tasks: Land Use/Land Cover (LULC) segmentation, Flood mapping, Cloud detection, Burned areas, and Regression tasks (Roads, Buildings). To accurately assess the robustness of the learned features, we developed a rigorous N-shot learning pipeline.

4. Domain Adaptation: From Simulation to Orbit

A model trained exclusively on simulated data inevitably suffers a performance drop when exposed to real in-orbit telemetry due to sensor noise, atmospheric differences, and optical distortions. We developed and implemented latent-space Domain Adaptation strategies to align the feature distributions of the simulated and real $\Phi$sat-2 domains, thereby also aligning them with the N-shot pipeline.

5. Ongoing Investigation: Knowledge Distillation (KD)

While the from-scratch MiM pretraining established a baseline, we are currently investigating Knowledge Distillation (KD) to improve the compact model's performance further. The objective is to transfer deep semantic representations from massive, state-of-the-art Foundation Models (such as large Vision Transformers or ResNets) into our restricted onboard CNN.