Understanding user engagement with cross-platform social media content created by humans versus AI : an evaluation of ChatGPT in content marketing

Tools

Aldous, Kholoud and Salminen, Joni and Farooq, Ali and Jung, Soon-Gyo and Jansen, Bernard (2025) Understanding user engagement with cross-platform social media content created by humans versus AI : an evaluation of ChatGPT in content marketing. ACM Transactions on the Web. ISSN 1559-1131 (In Press) (https://doi.org/10.1145/3756014)

[thumbnail of 3756014] Text. Filename: 3756014.pdf
Accepted Author Manuscript
Restricted to Repository staff only until 1 January 2099.
Download (2MB) | Request a copy

Abstract

Even though generative artificial intelligence (GenAI) is increasingly integrated into user-facing technologies like social media, its impact on content marketing remains unverified. Early evidence suggests that language models (LLMs) can generate content that rivals human-created content (HCC) in terms of appeal. However, the question of adapting such content for various social media platforms remains unanswered. This study examines the effectiveness of an LLM, GPT-4, in customizing cross-platform content for Facebook, Instagram, and X. A total of 892 participants evaluated 30 pairs of AI-created content (ACC) and HCC. The findings reveal that ACC was preferred by users, delivered stronger calls to action, and elicited more user engagement than HCC, especially on Facebook, with a less pronounced effect for shorter posts on X and Instagram. We further generated six data-driven user personas of the 892 participants, illustrating the differences between those who preferred ACC or HCC on the three platforms. The results indicate that GPT-4 can adapt content to platform-specific requirements and maintain high perceived quality, making LLMs applicable for cross-platform content creation for user engagement. Findings contribute to understanding user engagement with AI-generated content across platforms. We also discuss the role of LLMs in content creation, including their ethical implications.

ORCID iDs

Aldous, Kholoud, Salminen, Joni, Farooq, Ali ORCID logoORCID: https://orcid.org/0000-0003-4864-3155, Jung, Soon-Gyo and Jansen, Bernard;
CORE (COnnecting REpositories)