Project Summary

• Semantic Segmentation and Classification of Wetland Vegetation for Flood Mitigation. Developed a supervised segmentation model from scratch, utilizing self-supervised learning to classify wetland vegetation from high-resolution multispectral satellite imagery, reducing reliance on annotated data. Created a dataset comprising Sentinel-2 imagery with corresponding Dynamic World labels, facilitating future research, and created a dataset of high-resolution Pleiades NEO imagery with manually annotated labels.
• Benchmarking Foundation Models for SAR-based Vessel and Flood Detection. Evaluated the TerraMind foundation model, developed by IBM and ESA, on SAR-based flood and vessel detection tasks. Reformulated vessel detection from bounding boxes to georeferenced binary segmentation maps to enable segmentation-based training. Benchmarked TerraMind’s performance against a U-Net baseline, and implemented a binary version of Seesaw Loss to better detect small vessels by addressing class imbalance and scale variation. Task-specific evaluation metrics, including relaxed F1 and SIOU, were introduced to more accurately assess segmentation quality for small object detection.
• Enhancing Small Vessel Detection in SAR Imagery via Regression-based Semantic Segmentation. Explored a regression-based approach to improve small vessel detection by generating scale-aware regression maps from binary segmentation masks. Trained a modified U-Net on these maps and evaluated the results using task-specific metrics. Preliminary results demonstrated promising sensitivity to small vessels and highlight the potential of regression-based segmentation for advancing EO applications.

Development Tools

• Data Collection and Annotation: Sentinel-2 Dynamic World labels via Google Earth Engine. Manual annotation via Roboflow. Satellite image retrieval: Google Earth Engine (medium-resolution), Satellietdataportaal (high-resolution). Zenodo for publicly sharing the Sentinel-2 dataset.
• Geospatial Processing and Visualization: Rasterio and GDAL for raster handling. QGIS and ArcGIS for satellite imagery visualization and GIS workflows.
• Programming and Machine Learning: Python with libraries: PyTorch, PyTorch Lightning, TensorFlow, scikit-learn. CUDA for deep learning acceleration. HPC cluster: Snellius (Dutch national supercomputer).
• Experiment Tracking and Version Control: Weights & Biases (WandB) and TensorBoard for model evaluation and hyperparameter tuning. GitHub for version control and code sharing.

Development Outputs

• Implementation of adapted binary Seesaw Loss: https://github.com/ESA-PhiLab/SegSeeSawLoss
• MSc thesis on segmentation and classification of wetland vegetation and corresponding code and instructions: https://github.com/Evameijling/WetlandSemanticSegmentation
• Dataset with Sentinel-2 imagery and Dynamic World labels: https://doi.org/10.5281/zenodo.15125549
• Annotated high-resolution dataset from Pleiades NEO imagery (available upon request)
• MSc thesis summarized and submitted for NCCV 2025 conference (22–23 May 2025, Utrecht, The Netherlands): https://staff.fnwi.uva.nl/a.visser/publications/SegmentationClassificationBiesboschWetlands.pdf [E. Gmelich Meijling, R. Del Prete, and A. Visser, "Supervised and self-supervised land-cover segmentation & classification of the Biesbosch wetlands," presented at NCCV 2025, Utrecht, The Netherlands, May 2025.]
• Although the benchmarking code remains private due to data- and code-sharing restrictions, the code for the regression approach to small vessel detection is under development and will be made publicly available at: https://github.com/Evameijling/

Project Description

This collaboration focused on advancing artificial intelligence methods for Earth observation through three complementary research directions, conducted during the master's thesis and further developed during a three-month visiting researcher period at ESA Φ-lab. The primary objective was to develop a semantic segmentation framework for classifying wetland vegetation in the Biesbosch floodplain in the Netherlands, using medium- and high-resolution optical satellite imagery. The classification of wetland vegetation is a key step in estimating vegetation roughness, which is directly related to water retention and flood dynamics. Monitoring such ecological characteristics is essential for informing flood mitigation strategies and nature-based water management.

A major challenge addressed in this work was the scarcity of annotated training data, particularly for very high-resolution (VHR) imagery such as Pleiades NEO. To reduce reliance on manual labels, a self-supervised pretraining approach was adopted. An autoencoder was trained to reconstruct the original imagery, and its encoder weights were transferred to initialize a U-Net segmentation model. This strategy enabled more efficient feature extraction and improved generalization in scenarios with few labeled examples.

Experiments were conducted on two datasets: (1) Sentinel-2 imagery at 10-meter resolution, paired with Dynamic World labels, and (2) manually labeled Pleiades NEO imagery at 0.3-meter resolution. Figure 1 illustrates the qualitative improvements in segmentation accuracy when using a pretrained model compared to a model trained from scratch on medium-resolution data and labels.

Controlled evaluations compared segmentation accuracy across models trained from scratch and models pretrained with the autoencoder. Segmentation performance on Pleiades NEO data improved from 60.35% to 88.23% after pretraining, confirming the benefit of self-supervised representation learning for fine-grained, high-resolution classification tasks.

The segmentation outcomes were visualized and compared for both resolutions, as illustrated in Figure 2. These examples highlight the increased spatial detail captured in high-resolution imagery, particularly in capturing thin, fragmented vegetation and small water features. In addition, the classification masks allow interpretation of ecological transitions, such as reed-to-water zones, which are critical for estimating flow resistance.

To support further research, a curated Sentinel-2 dataset with Dynamic World labels was published via Zenodo (DOI: 10.5281/zenodo.15125549). A manually annotated dataset of VHR Pleiades NEO imagery, focusing on the Biesbosch region, was also created and is available upon request. All processing, training, and evaluation code is documented and available on GitHub.

The outcomes of this research were documented in a master's thesis and summarized in a peer-reviewed paper submitted to the NCCV 2025 conference in Utrecht, The Netherlands. The work forms the basis for potential operational deployment by conservation partners and contributes to open AI pipelines for vegetation monitoring in flood-prone ecosystems.

In addition to the primary focus on wetland segmentation, the scope of the visiting research period expanded to include the evaluation of foundation models for Earth observation tasks. Specifically, the TerraMind geospatial foundation model, developed jointly by ESA and IBM, was benchmarked for small vessel and flood detection in Sentinel-1 SAR data. TerraMind was originally designed to handle multimodal inputs and token-based spatial predictions. To enable meaningful comparison with task-specific models, the original bounding box annotations from the GRD SAR dataset were transformed into dense binary segmentation masks (see Figure 3), aligning the task with TerraMind’s pretraining tasks. This reframing also allowed for consistent training and evaluation alongside a U-Net baseline.

To further improve detection of small and sparsely distributed targets, a custom binary version of Seesaw Loss was implemented. This loss function penalizes dominant background predictions and dynamically adjusts the importance of hard versus easy examples, making it well-suited for class-imbalanced problems. Experiments demonstrated that Seesaw Loss improved performance under relaxed metrics, such as the Relaxed F1 and Scale-adaptive IoU (SIoU), which are more appropriate for assessing small object detection in SAR. However, across standard metrics like Dice and IoU, Seesaw Loss did not consistently outperform simpler alternatives. These findings provided insight into the practical trade-offs of using advanced loss formulations with foundation models and highlighted the continued advantage of domain-specific architectures when dealing with fine-grained SAR tasks.

The benchmarking also revealed inconsistencies in parts of the training data, particularly in the GRD annotations provided. As part of the work, a thorough quality check was conducted, resulting in an internal report that documented duplicate files, malformed bounding boxes, and misaligned labels. These observations informed dataset filtering and provided recommendations for future annotation efforts.

The final strand of the research explored a regression-based approach for improving small vessel detection in SAR imagery. Traditional binary segmentation often struggles with small or faint targets, which may be overlooked due to thresholding effects or class imbalance during training. To address this, a continuous representation was proposed: instead of classifying each pixel as vessel or background, regression heatmaps were generated from existing binary masks. These heatmaps encoded both the spatial location and approximate scale of each object using Gaussian-like activation profiles, as illustrated in Figure 4.

A modified U-Net architecture was adapted to regress these scale-aware heatmaps directly from SAR inputs. The network learned to predict these soft activation regions that peaked near vessel centers and faded outward, allowing small vessels to stand out more clearly during inference. A two-step postprocessing method was introduced to extract final object masks: low-confidence regions were filtered via thresholding, and the remaining activation areas were segmented using a watershed algorithm initialized from local maxima. This process allowed for spatially consistent detections while avoiding over-segmentation or loss of fine structures (see Figure 5).

Initial results demonstrated that this approach produced high-quality activation maps, with improved sensitivity to small vessels that might be missed by binary segmentation. While the HRSID-dataset used in the experiments lacked a mix of large and small vessels within the same scene, limiting full-scale benchmarking, the early findings suggest that regression-based outputs may offer an effective alternative for small object detection in radar imagery. The codebase is currently being finalized and will be published in a dedicated GitHub repository

The outputs of the collaboration included regular progress presentations, a submitted conference paper, curated datasets, open-source code, and a final technical report documenting the findings. The research supported ongoing initiatives at ESA Φ-lab in the field of AI4EO, particularly in foundation model evaluation and maritime detection using radar data. It also laid the groundwork for future studies on SAR-based regression approaches, benchmarking methodologies, and the operational integration of wetland vegetation classification tools by external partners.