Same data, different conclusions : radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Schweinsberg, Martin and Feldman, Michael and Staub, Nicola and van den Akker, Olmo R. and van Aert, Robbie C.M. and van Assen, Marcel A.L.M. and Liu, Yang and Althoff, Tim and Heer, Jeffrey and Kale, Alex and Mohamed, Zainab and Amireh, Hashem and Venkatesh Prasad, Vaishali and Bernstein, Abraham and Robinson, Emily and Snellman, Kaisa and Amy Sommer, S. and Otner, Sarah M.G. and Robinson, David and Madan, Nikhil and Silberzahn, Raphael and Goldstein, Pavel and Tierney, Warren and Murase, Toshio and Mandl, Benjamin and Viganola, Domenico and Strobl, Carolin and Schaumans, Catherine B.C. and Kelchtermans, Stijn and Naseeb, Chan and Mason Garrison, S. and Yarkoni, Tal and Richard Chan, C.S. and Adie, Prestone and Alaburda, Paulius and Albers, Casper and Alspaugh, Sara and Alstott, Jeff and Nelson, Andrew A. and Ariño de la Rubia, Eduardo and Arzi, Adbi and Bahník, Štěpán and Baik, Jason and Winther Balling, Laura and Banker, Sachin and A.A. Baranger, David and Barr, Dale J. and Barros-Rivera, Brenda and Bauer, Matt and Blaise, Enuh and Boelen, Lisa and Bohle Carbonell, Katerina and Briers, Robert A. and Burkhard, Oliver and Canela, Miguel Angel and Castrillo, Laura and Catlett, Timothy and Chen, Olivia and Clark, Michael and Cohn, Brent and Coppock, Alex and Cugueró-Escofet, Natàlia and Curran, Paul G. and Cyrus-Lai, Wilson and Dai, David and Valentino Dalla Riva, Giulio and Danielsson, Henrik and Russo, Rosaria de F.S.M. and de Silva, Niko and Derungs, Curdin and Dondelinger, Frank and Duarte de Souza, Carolina and Tyson Dube, B. and Dubova, Marina and Mark Dunn, Ben and Adriaan Edelsbrunner, Peter and Finley, Sara and Fox, Nick and Gnambs, Timo and Gong, Yuanyuan and Grand, Erin and Greenawalt, Brandon and Han, Dan and Hanel, Paul H.P. and Hong, Antony B. and Hood, David and Hsueh, Justin and Huang, Lilian and Hui, Kent N. and Hultman, Keith A. and Javaid, Azka and Ji Jiang, Lily and Jong, Jonathan and Kamdar, Jash and Kane, David and Kappler, Gregor and Kaszubowski, Erikson and Kavanagh, Christopher M. and Khabsa, Madian and Kleinberg, Bennett and Kouros, Jens and Krause, Heather and Krypotos, Angelos Miltiadis and Lavbič, Dejan and Ling Lee, Rui and Leffel, Timothy and Yang Lim, Wei and Liverani, Silvia and Loh, Bianca and Lønsmann, Dorte and Wei Low, Jia and Lu, Alton and MacDonald, Kyle and Madan, Christopher R. and Hjorth Madsen, Lasse and Maimone, Christina and Mangold, Alexandra and Marshall, Adrienne and Ester Matskewich, Helena and Mavon, Kimia and McLain, Katherine L. and McNamara, Amelia A. and McNeill, Mhairi and Mertens, Ulf and Miller, David and Moore, Ben and Moore, Andrew and Nantz, Eric and Nasrullah, Ziauddin and Nejkovic, Valentina and Nell, Colleen S. and Arthur Nelson, Andrew and Nilsonne, Gustav and Nolan, Rory and O'Brien, Christopher E. and O'Neill, Patrick and O'Shea, Kieran and Olita, Toto and Otterbacher, Jahna and Palsetia, Diana and Pereira, Bianca and Pozdniakov, Ivan and Protzko, John and Reyt, Jean-Nicolas and Riddle, Travis and (Akmal) Ridhwan Omar Ali, Amal and Ropovik, Ivan and Rosenberg, Joshua M. and Rothen, Stephane and Schulte-Mecklenbeck, Michael and Sharma, Nirek and Shotwell, Gordon and Skarzynski, Martin and Stedden, William and Stodden, Victoria and Stoffel, Martin A. and Stoltzman, Scott and Subbaiah, Subashini and Tatman, Rachael and Thibodeau, Paul H. and Tomkins, Sabina and Valdivia, Ana and Druijff-van de Woestijne, Gerrieke B. and Viana, Laura and Villesèche, Florence and Duncan Wadsworth, W. and Wanders, Florian and Watts, Krista and Wells, Jason D. and Whelpley, Christopher E. and Won, Andy and Wu, Lawrence and Yip, Arthur and Youngflesh, Casey and Yu, Ju-Chi and Zandian, Arash and Zhang, Leilei and Zibman, Chava and Luis Uhlmann, Eric (2021) Same data, different conclusions : radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis. Organizational Behavior and Human Decision Processes, 165. pp. 228-249. ISSN 0749-5978 (https://doi.org/10.1016/j.obhdp.2021.02.003)

[thumbnail of Schweinsberg-etal-OBHDP-2021-radical-dispersion-in-empirical-results-when-independent-analysts-operationalize-and-test-the-same-hypothesis]
Preview
 Text. Filename: Schweinsberg-etal-OBHDP-2021-radical-dispersion-in-empirical-results-when-independent-analysts-operationalize-and-test-the-same-hypothesis.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo
Download (3MB)| Preview

Abstract

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.

ORCID iDs

Schweinsberg, Martin, Feldman, Michael, Staub, Nicola, van den Akker, Olmo R., van Aert, Robbie C.M., van Assen, Marcel A.L.M., Liu, Yang, Althoff, Tim, Heer, Jeffrey, Kale, Alex, Mohamed, Zainab, Amireh, Hashem, Venkatesh Prasad, Vaishali, Bernstein, Abraham, Robinson, Emily, Snellman, Kaisa, Amy Sommer, S., Otner, Sarah M.G., Robinson, David, Madan, Nikhil, Silberzahn, Raphael, Goldstein, Pavel, Tierney, Warren, Murase, Toshio, Mandl, Benjamin, Viganola, Domenico, Strobl, Carolin, Schaumans, Catherine B.C., Kelchtermans, Stijn, Naseeb, Chan, Mason Garrison, S., Yarkoni, Tal, Richard Chan, C.S., Adie, Prestone, Alaburda, Paulius, Albers, Casper, Alspaugh, Sara, Alstott, Jeff, Nelson, Andrew A., Ariño de la Rubia, Eduardo, Arzi, Adbi, Bahník, Štěpán, Baik, Jason, Winther Balling, Laura, Banker, Sachin, A.A. Baranger, David, Barr, Dale J., Barros-Rivera, Brenda, Bauer, Matt, Blaise, Enuh, Boelen, Lisa, Bohle Carbonell, Katerina, Briers, Robert A., Burkhard, Oliver, Canela, Miguel Angel, Castrillo, Laura, Catlett, Timothy, Chen, Olivia, Clark, Michael, Cohn, Brent, Coppock, Alex, Cugueró-Escofet, Natàlia, Curran, Paul G., Cyrus-Lai, Wilson, Dai, David, Valentino Dalla Riva, Giulio, Danielsson, Henrik, Russo, Rosaria de F.S.M., de Silva, Niko, Derungs, Curdin, Dondelinger, Frank, Duarte de Souza, Carolina, Tyson Dube, B., Dubova, Marina, Mark Dunn, Ben, Adriaan Edelsbrunner, Peter, Finley, Sara, Fox, Nick, Gnambs, Timo, Gong, Yuanyuan, Grand, Erin, Greenawalt, Brandon, Han, Dan, Hanel, Paul H.P., Hong, Antony B., Hood, David, Hsueh, Justin, Huang, Lilian, Hui, Kent N., Hultman, Keith A., Javaid, Azka, Ji Jiang, Lily, Jong, Jonathan, Kamdar, Jash, Kane, David, Kappler, Gregor, Kaszubowski, Erikson, Kavanagh, Christopher M., Khabsa, Madian, Kleinberg, Bennett, Kouros, Jens, Krause, Heather, Krypotos, Angelos Miltiadis, Lavbič, Dejan, Ling Lee, Rui, Leffel, Timothy, Yang Lim, Wei, Liverani, Silvia, Loh, Bianca, Lønsmann, Dorte, Wei Low, Jia, Lu, Alton, MacDonald, Kyle, Madan, Christopher R., Hjorth Madsen, Lasse, Maimone, Christina, Mangold, Alexandra, Marshall, Adrienne, Ester Matskewich, Helena, Mavon, Kimia, McLain, Katherine L., McNamara, Amelia A., McNeill, Mhairi, Mertens, Ulf, Miller, David, Moore, Ben, Moore, Andrew, Nantz, Eric, Nasrullah, Ziauddin, Nejkovic, Valentina, Nell, Colleen S., Arthur Nelson, Andrew, Nilsonne, Gustav, Nolan, Rory, O'Brien, Christopher E., O'Neill, Patrick, O'Shea, Kieran ORCID logoORCID: https://orcid.org/0000-0001-7587-8537, Olita, Toto, Otterbacher, Jahna, Palsetia, Diana, Pereira, Bianca, Pozdniakov, Ivan, Protzko, John, Reyt, Jean-Nicolas, Riddle, Travis, (Akmal) Ridhwan Omar Ali, Amal, Ropovik, Ivan, Rosenberg, Joshua M., Rothen, Stephane, Schulte-Mecklenbeck, Michael, Sharma, Nirek, Shotwell, Gordon, Skarzynski, Martin, Stedden, William, Stodden, Victoria, Stoffel, Martin A., Stoltzman, Scott, Subbaiah, Subashini, Tatman, Rachael, Thibodeau, Paul H., Tomkins, Sabina, Valdivia, Ana, Druijff-van de Woestijne, Gerrieke B., Viana, Laura, Villesèche, Florence, Duncan Wadsworth, W., Wanders, Florian, Watts, Krista, Wells, Jason D., Whelpley, Christopher E., Won, Andy, Wu, Lawrence, Yip, Arthur, Youngflesh, Casey, Yu, Ju-Chi, Zandian, Arash, Zhang, Leilei, Zibman, Chava and Luis Uhlmann, Eric;
CORE (COnnecting REpositories)