Portal de Periódicos da CAPES

Conducting Psychology Student Research Via the Mechanical Turk Crowdsourcing Service

2013; Volume: 15; Issue: 2 Linguagem: Inglês

1527-7143

John A. Bates, Brian A. Lanza,

Statistics Education and Methodologies

Conducting research as a psychology student can be an exciting and valuable experience. Whether that research is a component of research methods coursework, a capstone project, theses, or independent study, it often represents the most significant part of a student's undergraduate educational experience. Unfortunately, the subject pool for such research often is restricted to the population of undergraduate psychology majors attending a university. This can make the attainment of an appropriate sample size very difficult at smaller campuses, severely limiting both power and generalizability of results. Even at larger schools, the time required to solicit participants, implement treatments, and collect data may discourage instructors from seriously considering the incorporation of student-designed and conducted studies into their research methods classes, encouraging them instead to rely on artificially constructed and consequently less meaningful data sets as learning tools. Mechanical Turk (MTurk) is a relatively new, on-line, participant solicitation and data collection tool that may be especially attractive for use in psychological research methods and statistics coursework, undergraduate and graduate independent studies, and undergraduate capstone projects (Buhrmester, Kwang & Gosling, 2011). MTurk was created by Internet retail giant Amazon as an in-house categorization service to assist in the identification and deletion of duplicate product web pages (Pontin, 2007). With MTurk's implementation, however, Amazon inadvertently also had created an infrastructure for an online, nearly world-wide labor market, wherein anonymous workers can be solicited to perform a variety of relatively short human intelligence tasks, such as audio transcription or image tagging, on their home computers for relatively low pay (Mason & Suri, 2012). Since its public release in 2005, MTurk has been adapted by social scientists to other purposes, including research projects in psychology, political science, sociology, and economics. For example, Horton, Rand and Zeckhauser (2011) replicated three classic experiments--the prisoner's dilemma game, the prisoner's dilemma game with the addition of primes, and a Tversky-Kahneman framing experiment--using large MTurk worker pools. Marge, Banerjee, and Rudnicky (2010) successfully investigated the ability of MTurk workers to transcribe various audio recordings accurately, and Eriksson and Simpson (2010) utilized MTurk to examine gender and culture differences in risk preferences among their international sample of workers. Finally, whereas most MTurk research has been asynchronous, in that workers complete their tasks at a time of their own choosing, Suri and Watts (2011) successfully created a virtual waiting room in which their workers were able to meet to receive the instruction to begin playing a public goods resource game at the same time. Despite MTurk's demonstrated utility as a means of collecting large samples of research participants very quickly, it reasonably may be questioned whether MTurk worker responses are as representative and reliable as those provided by samples of research participants obtained via more traditional methods. After investigating this issue, Horton et al. (2011) and Marge et al. (2010) have reported that the quality of MTurk data obtained in their studies rivaled the quality of conventionally obtained data. Paolacci, Chandler, and Ipierotis (2010) noted few differences between traditional and MTurk sample characteristics, and Cokely, Galesic, Schulz, Ghazal, and Garcia-Retamero (2012), Gardner, Brown, and Boice (2012), and Johnson and Borden (2012) all have reported their satisfaction with the validities of data they obtained via MTurk. Nevertheless, it should be understood that demographic characteristics and motives of potential MTurk workers may change rapidly with increasingly international internet accessibility, potentially reducing the quality of results obtained from future MTurk-based research. …