Conditional face image manipulation detection : combining algorithm and human examiner decisions

Ibsen, Mathias and Nichols, Robert and Rathgeb, Christian and Robertson, David J. and Davis, Josh P. and Løvåsdal, Frøy and Raja, Kiran and Jenkins, Ryan E. and Busch, Christoph; (2024) Conditional face image manipulation detection : combining algorithm and human examiner decisions. In: Proceedings of the 2024 ACM Workshop on Information Hiding and Multimedia Security. Association for Computing Machinery, ESP, pp. 41-46. ISBN 9798400706370 (https://doi.org/10.1145/3658664.3659649)

[thumbnail of Ibsen-etal-IH&MMSec-2024-Conditional-face-image-manipulation-detection]
Preview
 Text. Filename: Ibsen-etal-IH_MMSec-2024-Conditional-face-image-manipulation-detection.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo
Download (10MB)| Preview

Abstract

It has been shown that digitally manipulated face images can pose a security threat to automated authentication systems (e.g., when such systems are used for border control). In such scenarios, a malicious actor can, in many countries, apply for an identity document using a manipulated face image, which can then be used to gain fraudulent access to a system. Research has shown that humans and algorithms struggle to detect digitally manipulated face images, especially when the type of manipulation is unknown or when evaluated across multiple types of manipulations. In this work, we consider the detection performance of algorithms and humans on datasets consisting of retouched, face swapped and morphed images. Specifically, we investigate the joint performance of algorithms and humans in a differential detection scenario where both a trusted and suspected image are presented simultaneously. To this end, we propose a conditional face image manipulation detection approach where the human decision is only considered when the algorithm is unsure about the decision outcome. The results show that the automated algorithm performs better than the human detectors and that combining the decisions of algorithms and humans, in general, leads to an increased detection performance. To our knowledge, this is the first study to explore the joint detection performance of algorithms and humans in a differential face manipulation detection scenario and when using a variety of face image manipulations.

ORCID iDs

Ibsen, Mathias, Nichols, Robert, Rathgeb, Christian, Robertson, David J. ORCID logoORCID: https://orcid.org/0000-0002-8393-951X, Davis, Josh P., Løvåsdal, Frøy, Raja, Kiran, Jenkins, Ryan E. and Busch, Christoph;
CORE (COnnecting REpositories)