Applying Rotation-Equivariant Deep Learning to Cloud and Road Segmentation in Satellite and Aerial Imagery

Abstract

Satellite and aerial images have many applications – images of clouds and roads are of particular relevance to this thesis. Satellite images of clouds are frequently used for climate monitoring, weather tracking, satellite instrument calibration, and on-orbit autonomy; satellite and aerial images of roads are frequently used for mapping flooded areas, predicting illegal logging, traffic monitoring, and route planning.

Cloud detection in satellite imagery is key for autonomously taking and downlinking cloud-free images of a target region as well as studying cloud-climate interactions and calibrating microwave radiometers. Many existing state-of-the-art cloud detection algorithms require multispectral inputs and sometimes confuse clouds with snow, ice, or cold water. We propose deep learning models trained on visible-spectrum, long-wave infrared (LWIR), and short-wave infrared (SWIR) imagery for on-orbit cloud detection. Rotation-equivariant deep learning models are equivariant to rotations, meaning that when an input to the model is rotated, the model output will be equivalently rotated. We compare rotation-equivariant deep learning models to non-equivariant models, and also present comparisons to rule-based methods for cloud segmentation. Additionally, we compare models trained on visible-spectrum (VIS), LWIR, and SWIR imagery to models trained on only VIS and LWIR, on only VIS and SWIR, and on only VIS imagery and make recommendations for imaging bands to prioritize during instrument selection for resource-constrained missions. We find that augmenting VIS imagery with SWIR imagery is most useful for missions where false positives (non-cloud pixels misidentified as cloud) are extremely costly, and we find that augmenting with LWIR imagery is most useful for missions where false negatives (cloud pixels misidentified as non-cloud) are extremely costly.

A secondary focus of this thesis is evaluating rotation-equivariant deep learning models on the road detection domain. Road detection in satellite and aerial imagery can map safe evacuation routes from areas affected by natural disaster or predict deforestation by identifying roads constructed for the purpose of illegal logging. We present the results of rotation-equivariant and non-equivariant models on road segmentation of aerial imagery, and make recommendations for integrating rotation-equivariance into current state-of-the-art road detection algorithms.

We find that our C₈-equivariant dense U-Net, a rotation-equivariant deep learning model, outperforms our other deep learning models on both cloud and road segmentation, and also outperforms rule-based algorithms on cloud segmentation. The C₈-equivariant dense U-Net achieves an F₁ score of 0.9806 on the cloud segmentation dataset when evaluated with a 2 pixel buffer at the cloud boundaries, and achieves an F₁ score of 0.9342 on the road segmentation dataset when evaluated with a 4 pixel buffer at the road boundaries.