This paper aims to develop an accurate 3D geometry representation of satellite images using satellite-ground image pairs. Our focus is on the challenging problem of 3D-aware ground-views synthesis from a satellite image. We draw inspiration from the density field representation used in volumetric neural rendering and propose a new approach, called Sat2Density. Our method utilizes the properties of ground-view panoramas for the sky and non-sky regions to learn faithful density fields of 3D scenes in a geometric perspective. Unlike other methods that require extra depth information during training, our Sat2Density can automatically learn accurate and faithful 3D geometry via density representation without depth supervision. This advancement significantly improves the ground-view panorama synthesis task. Additionally, our study provides a new geometric perspective to understand the relationship between satellite and ground-view images in 3D space.
Combining the two strategies we proposed, namely the non-sky opacity loss and illumination injection, has eliminated the biggest obstacle to learning geometry from satellite-ground image pairs. Besides, cat depth with the initial panorama to sent to RenderNet helps the rendered image more faithful to the geometry.
Please refer to 4:48 - 5: 04 in the method video, which shows the illumination controlled video.
To our knowledge, Sat2Density represents the first successful attempt to learn precise 3D geometry from satellite-ground image pairs, significantly advancing the recognition of satellite-ground tasks from a geometric perspective.
To better understand our method, we recommend the reader read Geometry Guided Street-View Panorama Synthesis[1] , Sat2Vid , DirectVoxGo , and GANCraft.
Besides, some co-current works are also recommended: Behind the Scenes, Persistent Nature , and SatelliteSfM.
@article{Sat2Density,
author = {Qian, Ming and Xiong, Jincheng and Xia, Gui-Song and Xue, Nan},
title = {Sat2Density: Faithful Density Learning from Satellite-Ground Image Pairs},
booktitle = {IEEE/CVF International Conference on Computer Vision (ICCV)},
year = {2023},
}