Exploring the capability of text-to-image diffusion models with structural edge guidance for multi-spectral satellite image inpainting

Czerkawski, Mikolaj and Tachtatzis, Christos (2024) Exploring the capability of text-to-image diffusion models with structural edge guidance for multi-spectral satellite image inpainting. IEEE Geoscience and Remote Sensing Letters, 21. 5001905. ISSN 1545-598X (https://doi.org/10.1109/LGRS.2024.3370212)

[thumbnail of Czerkawski-Tachtatzis-IEEE-GRSL-2024-Exploring-the-capability-of-text-to-image-diffusion]
Preview
 Text. Filename: Czerkawski-Tachtatzis-IEEE-GRSL-2024-Exploring-the-capability-of-text-to-image-diffusion.pdf
Accepted Author Manuscript
License: Strathprints license 1.0
Download (6MB)| Preview

Abstract

The letter investigates the utility of text-to-image inpainting models for satellite image data. Two technical challenges of injecting structural guiding signals into the generative process as well as translating the inpainted RGB pixels to a wider set of MSI bands are addressed by introducing a novel inpainting framework based on StableDiffusion and ControlNet as well as a novel method for RGB-to-MSI translation. The results on a wider set of data suggest that the inpainting synthesized via StableDiffusion suffers from undesired artifacts and that a simple alternative of self-supervised internal inpainting achieves a higher quality of synthesis.

ORCID iDs

Czerkawski, Mikolaj ORCID logoORCID: https://orcid.org/0000-0002-0927-0416 and Tachtatzis, Christos ORCID logoORCID: https://orcid.org/0000-0001-9150-6805;
CORE (COnnecting REpositories)