Insights into the accuracy of social scientists' forecasts of societal change

Grossmann, Igor and Rotella, Amanda and Hutcherson, Cendri A. and Sharpinskyi, Konstantyn and Varnum, Michael E.W. and Achter, Sebastian and Dhami, Mandeep K. and Guo, Xinqi Evie and Kara-Yakoubian, Mane and Mandel, David R. and Raes, Louis and Tay, Louis and Vie, Aymeric and Wagner, Lisa and Adamkovic, Matus and Arami, Arash and Arriaga, Patrícia and Bandara, Kasun and Baník, Gabriel and Bartoš, František and Baskin, Ernest and Bergmeir, Christoph and Białek, Michał and Børsting, Caroline K. and Browne, Dillon T. and Caruso, Eugene M. and Chen, Rong and Chie, Bin Tzong and Chopik, William J. and Collins, Robert N. and Cong, Chin Wen and Conway, Lucian G. and Davis, Matthew and Day, Martin V. and Dhaliwal, Nathan A. and Durham, Justin D. and Dziekan, Martyna and Elbaek, Christian T. and Shuman, Eric and Fabrykant, Marharyta and Firat, Mustafa and Fong, Geoffrey T. and Frimer, Jeremy A. and Gallegos, Jonathan M. and Goldberg, Simon B. and Gollwitzer, Anton and Goyal, Julia and Graf-Vlachy, Lorenz and Gronlund, Scott D. and Tse, Dwight C.K., The Forecasting Collaborative (2023) Insights into the accuracy of social scientists' forecasts of societal change. Nature Human Behaviour, 7 (4). pp. 484-501. ISSN 2397-3374 (https://doi.org/10.1038/s41562-022-01517-1)

[thumbnail of Grossman-etal-NHB-2023-Insights-into-the-accuracy-of-social-scientists-forecasts-of-societal-change]
Preview
 Text. Filename: Grossman_etal_NHB_2023_Insights_into_the_accuracy_of_social_scientists_forecasts_of_societal_change.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo
Download (1MB)| Preview

Abstract

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing the accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender–career and racial bias. After we provided them with historical trend data on the relevant domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N = 86 teams and 359 forecasts), with an opportunity to update forecasts on the basis of new data six months later (Tournament 2; N = 120 teams and 546 forecasts). Benchmarking forecasting accuracy revealed that social scientists’ forecasts were on average no more accurate than those of simple statistical models (historical means, random walks or linear regressions) or the aggregate forecasts of a sample from the general public (N = 802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models and based predictions on prior data.

ORCID iDs

Grossmann, Igor, Rotella, Amanda, Hutcherson, Cendri A., Sharpinskyi, Konstantyn, Varnum, Michael E.W., Achter, Sebastian, Dhami, Mandeep K., Guo, Xinqi Evie, Kara-Yakoubian, Mane, Mandel, David R., Raes, Louis, Tay, Louis, Vie, Aymeric, Wagner, Lisa, Adamkovic, Matus, Arami, Arash, Arriaga, Patrícia, Bandara, Kasun, Baník, Gabriel, Bartoš, František, Baskin, Ernest, Bergmeir, Christoph, Białek, Michał, Børsting, Caroline K., Browne, Dillon T., Caruso, Eugene M., Chen, Rong, Chie, Bin Tzong, Chopik, William J., Collins, Robert N., Cong, Chin Wen, Conway, Lucian G., Davis, Matthew, Day, Martin V., Dhaliwal, Nathan A., Durham, Justin D., Dziekan, Martyna, Elbaek, Christian T., Shuman, Eric, Fabrykant, Marharyta, Firat, Mustafa, Fong, Geoffrey T., Frimer, Jeremy A., Gallegos, Jonathan M., Goldberg, Simon B., Gollwitzer, Anton, Goyal, Julia, Graf-Vlachy, Lorenz, Gronlund, Scott D. and Tse, Dwight C.K. ORCID logoORCID: https://orcid.org/0000-0003-2725-1849;
CORE (COnnecting REpositories)