Spiking Neural Networks for Land Cover and Land Use Classification

• Spiking Neural Networks for Land Cover and Land Use Classification
• About this project
• Requirements
• Installation
• Usage
• Training a VGG16-based model on EuroSAT RGB and UC Merced datasets
• Training a spiking model
• Energy consumption estimation
• License
• Contributors
• Contact
• Acknowledgements

About this project

This project investigates the feasibility of deploying spiking neural networks (SNN) in land cover and land use classification tasks. In particular, a VGG-16 -based artificial neural network (ANN) classifier is converted into an SNN using KerasSpiking. After fine-tuning of the converted model, we compare the accuracy depreciation as well as a potential improvement in the energy consumption on selected hardware platforms.

Requirements

This project uses Python 3.6, and requires the following third-party libraries:

• KerasSpiking 0.2.0
• NumPy 1.19.5
• Matplotlib 3.3.4
• TensorFlow 2.4.1
• TensorFlow Datasets 4.2.0
• Tensorflow I/O 0.17.0

No issues related to using different versions of the other libraries was encountered. Nonetheless, recommend creating a separate environment for this project and installing the versions of the packages specified above. [NumPy] (https://numpy.org/) can be acquired by installing the Anaconda distribution of Python.

Alternatively, all the required libraries can be installed using pip:

pip install numpy==1.19.5 matplotlib==3.3.4 tensorflow==2.4.1 \\
tensorflow-io==0.17.0 tensorflow-datasets==4.2.0

In case there are issues with running the TensorFlow files on a GPu-equipped workstation, we recommend trying TensorFlow nightly:

pip install tf-nightly tensorflow-nightly-io

Installation

To install this project either download it as a .zip file and extract it into a desired directory or clone it via the terminal or console command:

• using the HTTPS

git clone <https://github.com/AndrzejKucik/SNN4Space.git>

• or SSH

git clone git@github.com:AndrzejKucik/SNN4Space.git

Usage

Change the current working directory to be the SNN4Spacedirectory.

Training a VGG16-based model on EuroSAT RGB and UC Merced datasets

In the first part of the project we download either the EuroSAT: Land Use and Land Cover Classification with Sentinel-2 Dataset (10 classes, 27000 examples) or the UC Merced Land Use Dataset (21 classes, 100 examples each). We slice it into the training, validation, and test sets using ratios 80%-10%-10% (the UC Merced data examples are ordered according to version 2.0.0 in TensorFlow Datasets ). We augment the training set using random dihedral group transformation, random crop, random brightness change, random contrast change, random hue change, random saturation change. We use a modified version of the VGG-16 network trained on the ImageNet dataset (parameters from the Keras-TensorFlow) version to construct a classifier for this dataset. We replace the max pooling layers with average pooling layers, we remove the head of the network (all the layers following the last pooling layers) and replace it with a global pooling layer, and a dense classifier without bias. We resize the UC Merced images to (224, 224, 3) shape (to be compatible with the usual VGG-16 input size).

The model is trained using the RMSprop optimizer, using early stopping and reducing the learning rate on a plateau (by a factor of 10) if there is no significant improvement in the validation loss after 100 and 50 consecutive epochs respectively. Optionally, L₂ and L₁ regularization is applied to convolutional kernels and biases, respectively.

To train the network run:

python train_models.py [-ds dataset] [-s seed] [-e epochs] [-bs batch_size] \\
[-drpt dropout] [-kl2 kernel_l2] [-bl1 bias_l1] [-lz lower_zoom] \\
[-uz upper_zoom] [-mbd max_brightness_delta] [-mhd max_hue_delta] \\
[-lc lower_contrast] [-uc upper_contrast] [-ls lower_saturation] \\
[-us upper_saturation]

where the optional arguments are:

• dataset - chosen dataset; either eurosat, or ucm; one can also add either prewitt or sobel, then the (normalised) Prewitt or Sobel transforms are applied to the input images, and also, optionally mask, then the original images with those pixels, for which the Prewitt or Sobel transform are zero, masked out are used as the input; if sq is added, then the transforms are squared (or, equivalently, the square root in the Prewitt or Sobel transforms is not applied); so for example it can be eurosat_prewitt_sq_mask ot ucm_sobel etc.
• seed - global random seed,
• epochs - number of training epochs,
• batch_size - training batch size (per a replica),
• learning_ratelearning rate,
• kernel_l2 - regularization L₂ parameter for the convolutional kernels,
• bl1 bias_l1 - regularization L₁ parameter for the convolutional biases,
• lower_zoom - augmentation parameter; lower bound for a random zoom factor; must be positive,
• upper_zoom - augmentation parameter; upper bound for a random zoom factor; must be bigger than lower_zoom.
• max_brightness_delta - augmentation parameter; maximum brightness delta; must be a non-negative float,
• max_hue_delta - augmentation parameter; maximum hue delta; must be in the interval \[0, 0.5\],
• lower_contrast - augmentation parameter; lower bound for a random contrast factor; must be positive,
• upper_contrast - augmentation parameter; upper bound for a random contrast factor must be bigger than lower_contrast
• lower_saturation - augmentation parameter; lower bound for a random saturation factor; must be positive,
• upper_saturation - augmentation parameter; upper bound for a random saturation factor; must be bigger than lower_saturation.

The default values of these parameters (apart from the batch size) are the ones that empirically gave us the best test accuracy performance (91.43%) on UC Merced and (95.07%) on EuroSat.

The trained model is saved to

./models/vgg16/{dataset}/s_{seed}_e_{epochs}_bs_{batch_size}_lr_
{learning_rate}_kl2_{kernel_l2}_bl1_{bias_l1}_lz_
{lower_zoom}_uz_{upper_zoom}_mbd_{max_brightness_delta}_mhd_{max_hue_delta}
_lc_{lower_contrast}_uc_{upper_contrast}_ls_{lower_saturation}_us_
{upper_saturation}_acc_{accuracy}.h5

where each {name}is replaced by the corresponding value of name from the above list and accuracy represents the accuracy of the model on the test set.

Note that the obtained test set accuracy might be different each time ( sometimes even failing to converge), even if identical parameters are used, because the global random seed does not seem to affect the shuffling processes or data augmentation parameters.

Training a spiking model

A VGG-16 -based classifier trained on the EuroSat or UC Merced datasets can be converted into a spiking neural network and trained using KerasSpiking module by running

python train_spiking_models.py -md model_path -ds dataset [-s seed] \\
[-i iterate] [-e epochs] [-bs batch_size] [-lr learning_rate] [-t timesteps] \\
[-dt dt] [-l2 l2] [-lhz lower_hz] [-uhz upper_hz] [-tau tau] [-lz lower_zoom] \\
[-uz upper_zoom] [-mbd max_brightness_delta] [-mhd max_hue_delta] \\
[-lc lower_contrast] [-uc upper_contrast] [-ls lower_saturation] \\
[-us upper_saturation]

where

• model_path - path of a valid .h5 model which we obtain after running train_models.py
• iterate - boolean determining whether the training should be performed iteratively, doubling the number of timesteps, and halving the batch size, the number of epochs, and the learning rate at each iteration,
• timesteps - number of the simulation timesteps,
• dt - temporal resolution of timesteps; it is decreased during the training until it reaches the value of 1 ms
• l2 - weight penalty for L₂ activity regularization of the spikes,
• lower_hz - lower frequency threshold for spiking rate regularization,
• upper_hz - upper frequency threshold for spiking rate regularization.

The rest of the options are as defined the in the previous subsection above, however, epochs, batch_size, and learning_rate are always the starting values if the training is done iteratively, and timesteps is the approximate number of target simulation timesteps (the actual number of timesteps is always a power of 2), while the starting number of timesteps is 1.

Before the training, the local average pooling layers are removed, and the preceding convolutions have their strides set to 2, and their weights appropriately adjusted for consistency. The ReLU activation functions are swapped with spiking activations followed by a low-pass filter

The trained model's weights are saved to

./models/spiking_wgg16/{dataset}/s_{seed}_e_{epochs}_bs_{batch_size}_ls_
{learning_rate}_t_{timesteps}_dt_{dt}_l2_{l2}_lhz_{lower_hz}_uhz_{upper_hz}_lz_
{lower_zoom}_uz_{upper_zoom}_mbd_{max_brightness_delta}_mhd_{max_hue_delta}_lc_
{lower_contrast}_uc_{upper_contrast}_ls_{lower_saturation}_us_
{upper_saturation}_acc_{accuracy}.h5

The trained spiking model can be fine-tuned for a specific number of time steps and fixed temporal dimension by running:

python fine_tune_snn.py -wp weights_path -ds dataset [-s seed] [-e epochs] \\
[-bs batch_size] [-lr learning_rate] [-t timesteps] [-dt dt] [-l2 l2] \\
[-lhz lower_hz] [-uhz upper_hz] [-tau tau] [-lz lower_zoom] [-uz upper_zoom] \\
[-mbd max_brightness_delta] [-mhd max_hue_delta] [-lc lower_contrast] \\
[-uc upper_contrast] [-ls lower_saturation] [-us upper_saturation]

where weights_path is a path to a valid .h5 file with pre-trained weights (as output by train_spiking_models. py), and the rest of the parameters are as above.

Energy consumption estimation

To compare the estimated energy consumption of the ANN and SNN (both with local pooling layers replaced by doubly-strided convolutions) on representative hardware (standard and neuromorphic), run:

python energy_estimation.py -wp weights_path -ds dataset -bs batch_size \\
-t timesteps -dt dt -v verbose

where verbose is the verbosity mode, and the remaining options are as in the previous section.

License

Property of the European Space Agency. Distributed under MIT licence. See LICENSE for more information.

Contributors

• Andrzej Kucik
• Gabriele Meoni

Contact

E-mail: andrzej.kucik@esa.int gabriele.meoni@esa.int

Project Link: https://github.com/AndrzejKucik/SNN4Space

Acknowledgements

• ESA Advanced Concepts Team
• ESA Φ-Lab