Automatically generating natural language descriptions of images by a deep hierarchical framework

Tools

Huo, Lin and Bai, Lin and Zhou, Shang Ming (2022) Automatically generating natural language descriptions of images by a deep hierarchical framework. IEEE Transactions on Cybernetics, 52 (8). 7441 - 7452. ISSN 2168-2275 (https://doi.org/10.1109/TCYB.2020.3041595)

[thumbnail of Huo-etal-IEEETC-2021-Automatically-generating-natural-language-descriptions-of-images-by-a-deep-hierarchical-framework]
Preview
 Text. Filename: Huo_etal_IEEETC_2021_Automatically_generating_natural_language_descriptions_of_images_by_a_deep_hierarchical_framework.pdf
Accepted Author Manuscript
Download (2MB)| Preview

Abstract

Automatically generating an accurate and meaningful description of an image is very challenging. However, the recent scheme of generating an image caption by maximizing the likelihood of target sentences lacks the capacity of recognizing the human-object interaction (HOI) and semantic relationship between HOIs and scenes, which are the essential parts of an image caption. This article proposes a novel two-phase framework to generate an image caption by addressing the above challenges: 1) a hybrid deep learning and 2) an image description generation. In the hybrid deep-learning phase, a novel factored three-way interaction machine was proposed to learn the relational features of the human-object pairs hierarchically. In this way, the image recognition problem is transformed into a latent structured labeling task. In the image description generation phase, a lexicalized probabilistic context-free tree growing scheme is innovatively integrated with a description generator to transform the descriptions generation task into a syntactic-tree generation process. Extensively comparing state-of-the-art image captioning methods on benchmark datasets, we demonstrated that our proposed framework outperformed the existing captioning methods in different ways, such as significantly improving the performance of the HOI and relationships between HOIs and scenes (RHIS) predictions, and quality of generated image captions in a semantically and structurally coherent manner.

CORE (COnnecting REpositories)