Portal de Periódicos da CAPES

Artigo Acesso aberto Produção Nacional Revisado por pares

BIBTEX RIS

Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

2018; SAGE Publishing; Volume: 1; Issue: 3 Linguagem: Inglês

10.1177/2515245917747646

2515-2467

Raphael Silberzahn, Eric Luis Uhlmann, Daniel P. Martin, Pasquale Anselmi, Frederik Aust, Eli Awtrey, Štěpán Bahník, Feng Bai, Colin Bannard, Evelina Bonnier, Rickard Carlsson, Felix Cheung, Garret Christensen, Russ Clay, Maureen A. Craig, Anna Dalla Rosa, Lammertjan Dam, Mathew H. Evans, Ismael Flores Cervantes, Nathan M. Fong, Monica Gamez-Djokic, Andreas Glenz, Shauna Gordon-McKeon, Timothy Heaton, Karin Hederos, Moritz Heene, Alicia Hofelich Mohr, Fabia Högden, Kent Ngan‐Cheung Hui, Magnus Johannesson, Jonathan Kalodimos, Erikson Kaszubowski, Deanna M. Kennedy, Ryan Lei, Thomas Lindsay, Silvia Liverani, Christopher R. Madan, Daniel C. Molden, Eric Molleman, Richard D. Morey, Laetitia B. Mulder, B. R. Nijstad, Nolan Pope, Bryson Pope, Jason M. Prenoveau, Floor Rink, Egidio Robusto, Hadiya Roderique, Anna Sandberg, Elmar Schlüter, Felix D. Schönbrodt, Martin F. Sherman, S. Amy Sommer, Kristin Lee Sotak, Seth M. Spain, Christoph Spörlein, Tom Stafford, Luca Stefanutti, Susanne Täuber, Johannes Ullrich, Michelangelo Vianello, Eric–Jan Wagenmakers, Maciej Witkowiak, Sangsuk Yoon, Brian A. Nosek,

Behavioral Health and Interventions

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 ( Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.