A Deep Learning Ensemble Framework for Off-Nadir Geocentric Pose Prediction

Roughly 6,800 natural disasters occur annually, and this alarming number continues to grow due to climate change. Effective methods to improve natural disaster response include change detection, map alignment, and vision-aided navigation to allow for the time-efficient delivery of life-saving aid. Current software functions optimally only on near-nadir images taken around ninety degrees above ground level. The inability to generalize to oblique images increases the need to compute an image's geocentric pose, which is its spatial orientation with respect to gravity. This Deep Learning investigation presents three convolutional models to predict geocentric pose using 5,923 nadir and off-nadir red, green, and blue (RGB) satellite images of cities worldwide. Prior to the ensemble, an autoencoder is assembled to condense the 256 x 256 x 3 images to 32 x 32 x 16 latent space representations, demonstrating the ability to learn useful features from the data. The first model in the ensemble is a U-Net Fully Convolutional Network (FCN) with skip connections used to predict each image's corresponding pixel-level above-ground elevation mask. This FCN achieves a median absolute deviation of 0.335 meters and an R2 of 0.865 on test data. Afterward, the elevation masks are concatenated with the RGB images to form four-channel inputs fed into the second model in the ensemble, which predicts each image's rotation angle and magnification scale, the components of its geocentric pose. This Deep Convolutional Model achieves an R2 of 0.943 on test data, significantly outperforming previous models designed by researchers. In addition to achieving superior model performance, outlier removal was performed through supervised interpolation, and the usefulness of data features was gauged through a sensitivity analysis of elevation masks, to target future avenues of feature engineering. The high-accuracy software built in this study contributes to mapping and navigation procedures to accelerate disaster relief and save human lives.