Multi-modal convolutional parameterisation network for guided image inverse problems

Czerkawski, Mikolaj and Upadhyay, Priti and Davison, Christopher and Atkinson, Robert and Michie, Craig and Andonovic, Ivan and Macdonald, Malcolm and Cardona, Javier and Tachtatzis, Christos (2024) Multi-modal convolutional parameterisation network for guided image inverse problems. Journal of Imaging, 10 (3). 69. ISSN 2313-433X (https://doi.org/10.3390/jimaging10030069)

[thumbnail of Czerkawski-etal-JI-2024-Multi-modal-convolutional-parameterisation-network-for-guided-image-inverse-problems]
Preview
 Text. Filename: Czerkawski-etal-JI-2024-Multi-modal-convolutional-parameterisation-network-for-guided-image-inverse-problems.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo
Download (21MB)| Preview

Abstract

There are several image inverse tasks, such as inpainting or super-resolution, which can be solved using deep internal learning, a paradigm that involves employing deep neural networks to find a solution by learning from the sample itself rather than a dataset. For example, Deep Image Prior is a technique based on fitting a convolutional neural network to output the known parts of the image (such as non-inpainted regions or a low-resolution version of the image). However, this approach is not well adjusted for samples composed of multiple modalities. In some domains, such as satellite image processing, accommodating multi-modal representations could be beneficial or even essential. In this work, Multi-Modal Convolutional Parameterisation Network (MCPN) is proposed, where a convolutional neural network approximates shared information between multiple modes by combining a core shared network with modality-specific head networks. The results demonstrate that these approaches can significantly outperform the single-mode adoption of a convolutional parameterisation network on guided image inverse problems of inpainting and super-resolution.

ORCID iDs

Czerkawski, Mikolaj ORCID logoORCID: https://orcid.org/0000-0002-0927-0416, Upadhyay, Priti ORCID logoORCID: https://orcid.org/0000-0002-6212-5314, Davison, Christopher ORCID logoORCID: https://orcid.org/0000-0002-9450-1791, Atkinson, Robert ORCID logoORCID: https://orcid.org/0000-0002-6206-2229, Michie, Craig ORCID logoORCID: https://orcid.org/0000-0001-5132-4572, Andonovic, Ivan ORCID logoORCID: https://orcid.org/0000-0001-9093-5245, Macdonald, Malcolm ORCID logoORCID: https://orcid.org/0000-0003-4499-4281, Cardona, Javier ORCID logoORCID: https://orcid.org/0000-0002-9284-1899 and Tachtatzis, Christos ORCID logoORCID: https://orcid.org/0000-0001-9150-6805;
CORE (COnnecting REpositories)