Crowdsourcing Samples in Cognitive Science
2017; Elsevier BV; Volume: 21; Issue: 10 Linguagem: Inglês
10.1016/j.tics.2017.06.007
ISSN1879-307X
AutoresNeil Stewart, Jesse Chandler, Gabriele Paolacci,
Tópico(s)Decision-Making and Behavioral Economics
ResumoIn the next few years we estimate nearly half of all cognitive science research articles will involve samples of participants from Amazon Mechanical Turk and other crowdsourcing platforms. We review the technical aspects of programming for the web and the resources available to experimenters. Crowdsourcing of participants offers a ready and very different complement to the traditional college student samples, and much is now known about the reproducibility of findings with crowdsourced samples. The population which we are sampling from is surprisingly small and highly experienced in cognitive science experiments, and this non-naïveté affects responses to frequently used measures. The larger sample sizes that crowdsourcing affords bode well for addressing aspects of the replication crisis, but a possible tragedy of the commons looms now that cognitive scientists increasingly share the same participants. Crowdsourcing data collection from research participants recruited from online labor markets is now common in cognitive science. We review who is in the crowd and who can be reached by the average laboratory. We discuss reproducibility and review some recent methodological innovations for online experiments. We consider the design of research studies and arising ethical issues. We review how to code experiments for the web, what is known about video and audio presentation, and the measurement of reaction times. We close with comments about the high levels of experience of many participants and an emerging tragedy of the commons. Crowdsourcing data collection from research participants recruited from online labor markets is now common in cognitive science. We review who is in the crowd and who can be reached by the average laboratory. We discuss reproducibility and review some recent methodological innovations for online experiments. We consider the design of research studies and arising ethical issues. We review how to code experiments for the web, what is known about video and audio presentation, and the measurement of reaction times. We close with comments about the high levels of experience of many participants and an emerging tragedy of the commons. Online labor markets have become enormously popular as a source of research participants in the cognitive sciences. These marketplaces match researchers with participants who complete research studies in exchange for money. Originally, these marketplaces fulfilled the demand for large-scale data generation, cleaning, and transformation jobs that require human intelligence. However, one such marketplace, Amazon Mechanical Turk (MTurk), became a popular source of convenience samples for cognitive science research (MTurk terms are listed in Box 1). Based upon searches of the full article text, we find that in 2017, 24% of articles in Cognition, 29% of articles in Cognitive Psychology, 31% of articles in Cognitive Science, and 11% of articles in Journal of Experimental Psychology: Learning, Memory, and Cognition mention MTurk or another marketplace, all up from 5% or fewer five years ago.Box 1MTurk TermsApproval/rejection: once a worker completes a HIT, a requester can choose whether to approve the HIT (and compensate the worker with the reward) or reject the HIT (and not compensate the worker).Block: a requester can 'block' workers and disqualify them from any future task they post. Workers are banned from MTurk after an unspecified number of blocks.Command line tools: a set of instructions that can be input in Python to send instructions to MTurk via its application programming interface (API) [90Mueller P. Chandler J. Emailing workers using Python.SSRN eLibrary. 2012; (Published online July 5, 2012. https://ssrn.com/abstract=2100601)https://doi.org/10.2139/ssrn.2100601Crossref Google Scholar].HIT approval ratio: the ratio between the number of approved tasks and the number of total tasks completed by a worker in her history. Until a worker completes 100 HITs, this is set at a 100% approval ratio on MTurk.Human intelligence task (HIT): a task posted on MTurk by a requester for completion by a worker.Qualifications: requirements that a requester sets for workers to be eligible to complete a given HIT. Some qualifications are assigned by Amazon and are available to all requesters. Requesters can also create their own qualifications.Requesters: people or companies who post HITs on MTurk for workers to complete.Reward: the compensation promised to workers who successfully complete a HIT.Turkopticon: a website where workers rate requesters based on several criteria.TurkPrime: one among the websites that augments or automates the use of several MTurk features [77Litman L. et al.TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences.Behav. Res. Methods. 2016; 49: 433-442https://doi.org/10.3758/s13428-016-0727-zCrossref Scopus (1038) Google Scholar].Worker file: a comma-separated values (CSV) file downloadable by the requester with a list of all workers who have completed at least one task for the requester.Workers: people who subscribe to MTurk to complete HITs in exchange for compensation. Approval/rejection: once a worker completes a HIT, a requester can choose whether to approve the HIT (and compensate the worker with the reward) or reject the HIT (and not compensate the worker). Block: a requester can 'block' workers and disqualify them from any future task they post. Workers are banned from MTurk after an unspecified number of blocks. Command line tools: a set of instructions that can be input in Python to send instructions to MTurk via its application programming interface (API) [90Mueller P. Chandler J. Emailing workers using Python.SSRN eLibrary. 2012; (Published online July 5, 2012. https://ssrn.com/abstract=2100601)https://doi.org/10.2139/ssrn.2100601Crossref Google Scholar]. HIT approval ratio: the ratio between the number of approved tasks and the number of total tasks completed by a worker in her history. Until a worker completes 100 HITs, this is set at a 100% approval ratio on MTurk. Human intelligence task (HIT): a task posted on MTurk by a requester for completion by a worker. Qualifications: requirements that a requester sets for workers to be eligible to complete a given HIT. Some qualifications are assigned by Amazon and are available to all requesters. Requesters can also create their own qualifications. Requesters: people or companies who post HITs on MTurk for workers to complete. Reward: the compensation promised to workers who successfully complete a HIT. Turkopticon: a website where workers rate requesters based on several criteria. TurkPrime: one among the websites that augments or automates the use of several MTurk features [77Litman L. et al.TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences.Behav. Res. Methods. 2016; 49: 433-442https://doi.org/10.3758/s13428-016-0727-zCrossref Scopus (1038) Google Scholar]. Worker file: a comma-separated values (CSV) file downloadable by the requester with a list of all workers who have completed at least one task for the requester. Workers: people who subscribe to MTurk to complete HITs in exchange for compensation. Although researchers have used the internet to collect convenience samples for decades [1Gosling S.D. Mason W. Internet research in psychology.Annu. Rev. Psychol. 2015; 66: 877-902https://doi.org/10.1146/ annurev-psych-010814-015321Crossref PubMed Google Scholar], MTurk allowed an exponential-like growth of online experimentation by solving several logistical problems inherent to recruiting research participants online. Marketplaces like MTurk aggregate working opportunities and workers, ensuring that there is a critical mass of available participants, which increases the speed at which data can be collected. Moreover, these marketplaces incorporate reputation systems that incentivize conscientious participation, as well as a secure payment infrastructure that simplifies participant compensation. Although marketing research companies provide similar services, they are often prohibitively expensive [2Berinsky A.J. et al.Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk.Polit. Anal. 2012; 20: 351-368https://doi.org/10.1093/pan/mpr057Crossref Scopus (2611) Google Scholar, 3Mullinix K.J. et al.The generalizability of survey experiments.J. Exp. Polit. Psychol. 2016; 2: 109-138https://doi.org/10.1017/XPS.2015.19Crossref Scopus (637) Google Scholar]. Online labor markets allow participants to be recruited directly for a fraction of the cost. Academic interest in MTurk began in computer science, which used crowdsourcing to perform human intelligence tasks (HITs) such as transcription, developing training data sets, validating machine learning algorithms, and to engage in human factors research [4Kittur A. et al.Crowdsourcing user studies with Mechanical Turk.in: Czerwinski M. Proceedings of the SIGCHI Conference on Human factors in Computing systems. ACM, 2008: 453-456Crossref Scopus (1266) Google Scholar]. From there, MTurk diffused to subdisciplines of psychology such as decision making [5Paolacci G. et al.Running experiments on Amazon Mechanical Turk.Judgm. Decis. Mak. 2010; 5: 411-419http://journal.sjdm.org/10/10630a/jdm10630a.pdfGoogle Scholar] and personality and social psychology [6Buhrmester M. et al.Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data?.Perspectives On Psychol. Sci. 2011; 6: 3-5https://doi.org/10.1177/1745691610393980Crossref PubMed Scopus (7661) Google Scholar]. Use of MTurk then radiated out to other social science disciplines such as economics [7Horton J.J. et al.The online laboratory: Conducting experiments in a real labor market.Exp. Econ. 2011; 14: 399-425https://doi.org/10.1007/s10683-011-9273-9Crossref Scopus (928) Google Scholar], political science [2Berinsky A.J. et al.Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk.Polit. Anal. 2012; 20: 351-368https://doi.org/10.1093/pan/mpr057Crossref Scopus (2611) Google Scholar], and sociology [8Shank D.B. Using crowdsourcing websites for sociological research: The case of Amazon Mechanical Turk.Am. Sociol. 2016; 47: 47-55https://doi.org/10.1007/s12108-015-9266-9Crossref Scopus (113) Google Scholar], and to applied fields such as clinical science [9Shapiro D.N. et al.Using Mechanical Turk to study clinical populations.Clinical Psychol. Sci. 2013; 1: 213-220https://doi.org/10.1177/2167702612469015Crossref Scopus (846) Google Scholar], marketing [10Goodman J.K. Paolacci G. Crowdsourcing consumer research.J. Consum. Res. 2017; 44: 196-210https://doi.org/10.1093/jcr/ucx047Crossref Scopus (298) Google Scholar], accounting [11Bentley J.W. Challenges with Amazon Mechanical Turk research in accounting.SSRN eLibrary. 2017; (Published online February 28, 2017. https://ssrn.com/abstract=2924876)Google Scholar], and management [12Stritch J.M. et al.The opportunities and limitations of using Mechanical Turk (Mturk) in public administration and management scholarship.Int. Public. Manag. J. 2017; (Published online January 19, 2017)https://doi.org/10.1080/10967494.2016.1276493Crossref Scopus (93) Google Scholar]. Across fields, research using online labor markets has typically taken the form of short, cross-sectional surveys conducted across the entire crowd population or on a specific geographic subpopulation (e.g., US residents). However, online labor markets are fairly flexible, and allow creative and sophisticated research methods (Box 2) limited only by the imaginations of researchers and their willingness to learn basic computer programming (Box 3). Accurate timing allows a wide range of paradigms from cognitive science to be implemented online (Box 4). Recently, several new tools (e.g., TurkPrime) and competitor marketplaces have developed more advanced sampling features while lowering technical barriers substantially.Box 2Innovative MTurk ResearchInfant AttentionParents have been recruited on MTurk to record the gaze patterns of their infants using a webcam while the infants views video clips [78Scott K. Schulz L. Lookit (part 1): A new online platform for developmental research.Open Mind. 2017; 1: 4-14https://doi.org/10.1162/opmi_ a_00002Crossref Google Scholar, 79Tran M. et al.Online recruitment and testing of infants with Mechanical Turk.J. Exp. Child Psychol. 2017; 156: 168-178https://doi.org/10.1016/j.jecp.2016.12.003Crossref PubMed Scopus (25) Google Scholar]. Some participants were excluded because of technical issues (e.g., the webcam recording became desynchronized from the video) or because the webcam recording was of insufficient quality to view the eyes of the infant. However, it was possible to identify the features of videos that attracted infant attention, including singing, camera zooms, and faces.Economic GamesMany economic games have been run on MTurk, including social dilemma games and the prisoner's dilemma [69Rand D.G. et al.Social heuristics shape intuitive cooperation.Nat. Commun. 2014; 5: e3677https://doi.org/10.1038/ ncomms4677Crossref PubMed Google Scholar]. The innovation here was to have remotely located participants playing live against one another, where one person's decision affects the outcome that another receives, and vice versa. Software platforms to implement economic games include Lioness [80Arechar A. et al.Conducting interactive experiments online.Exp. Econ. 2017; (Published online May 9, 2017)https://doi.org/10.1007/s10683-017-9527-2Crossref PubMed Scopus (126) Google Scholar] and NodeGame [81Balietti S. nodeGame: Real-time, synchronous, online experiments in the browser.Behav. Res. Methods. 2016; (Published online November 18, 2016)https://doi.org/10.3758/s13428-016-0824-zCrossref PubMed Scopus (18) Google Scholar].Crowd CreativitySeveral researchers have assembled groups of workers to engage in iterative creative tasks [82Yu L. Nickerson J.V. Cooks or cobblers? Crowd creativity through combination.Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 2011: 1393-1402Google Scholar]. More recently, workers have collaboratively written short stories, and changes in the task structure influence the end-results [83Kim, J. et al. (2016) Mechanical novel: Crowdsourcing complex work through reflection and revision. Comput. Res. Repository. http://dx.doi.org/10.1145/2998181.2998196.Google Scholar]. The innovation here was using microtasks to decompose tasks into smaller subtasks and explore how changes to the structure of these tasks changes group output.Transactive CrowdsSeveral researchers have looked at how crowds can be used to supplement or replace individual cognitive abilities. For example, MTurk workers have provided cognitive reappraisals in response to the negative thoughts of other workers [84Morris R.R. Picard R. Crowd-powered positive psychological interventions.J. Positive Psychol. 2014; 9: 509-516https://doi.org/10.1080/ 17439760.2014.913671Crossref Google Scholar], and an app has been developed to allow people with visual impairments to upload images and receive near real-time descriptions of their contents [85Bigham J.P. et al.VizWiz: Nearly real-time answers to visual questions.in: Perlin K. Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology. ACM, New York2010: 333-342Crossref Scopus (495) Google Scholar]. The innovation here was to allow workers to engage in tasks in contexts that are unsupervised, have little structure, and occur in near real-time.Workers as Field ResearchersMeier et al. [86Meier A. et al.Usability of residential thermostats: Preliminary investigations.Build. Environ. 2011; 46: 1891-1898https://doi.org/10.1016/j.buildenv.2011.03.009Crossref Scopus (62) Google Scholar] asked participants take pictures of their thermostats and upload them to determine whether they were set correctly. The innovation here was using workers to collect field data about their local environment and relay it back to the researchers.Mechanical DiaryMechanical Turk allows researchers to recontact workers multiple times, allowing longitudinal research. Boynton and Richman [87Boynton M.H. Richman L.S. An online diary study of alcohol use using Amazon's Mechanical Turk.Drug Alcohol Rev. 2014; 33: 456-461https://doi.org/10.1111/dar.12163Crossref PubMed Scopus (55) Google Scholar] used this functionality to conduct a 2 week daily diary study of alcohol consumption.Time of DayIt is also easier to test workers at different times of day (e.g., 03:00 h), without the disruption of a visit to the laboratory. Using judgments from a line bisection task, Dorrian et al. [88Dorrian J. et al.Morningness/eveningness and the synchrony effect for spatial attention.Accident Anal. Prev. 2017; 99: 401-405https://doi.org/10.1016/j.aap.2015.11.012Crossref PubMed Scopus (7) Google Scholar] found that spatial attention moved rightwards in peak compared to off-peak times of day, but only for morning types not evening types.Crowds as Research AssistantsOnline labor markets were originally created to facilitate human computation tasks such as content coding for which academics have traditionally relied upon research assistants. Crowds often produce responses that are equivalent or superior to the judgments of experts. For example, Benoit et al. [89Benoit K. et al.Crowd-sourced text analysis: Reproducible and agile production of political data.Am. Polit. Sci. Rev. 2016; 110: 278-295https://doi.org/10.1017/S0003055416000058Crossref Scopus (121) Google Scholar] used MTurk to rate the ideology of statements from political texts: they found that 15 workers produce ratings of equivalent quality to five political science PhD students and faculty. Importantly, crowds can return data exceptionally quickly: workers were able to content code 22 000 sentences in under 5 h for $360.Box 3Coding for CrowdsourcingMTurk is accessible through a graphical user interface (GUI) that can perform most of platform functions either directly or through downloading, modifying and reuploading worker files. MTurk can also be accessed by command line tools that can simplify tasks such as contacting workers in bulk or assigning variable bonuses. More recently, TurkPrime has developed an alternative GUI that offers an extended set of features.It is possible to code and field simple survey experiments entirely within the MTurk platform using HTML5, but functionality is limited. Most researchers create a HIT with a link to an externally hosted survey and a text box in which the worker can paste a confirmation code upon completing the study. Simple surveys might be conducted using a variety of online platforms, such as Google forms or www.surveymonkey.co.uk, which require no programming skills. Surveys with more complex designs (e.g., complex randomization of blocks and items) may benefit from using Qualtrics, which also requires minimal programming skill. Researchers can also direct users to specialized platforms designed to program reaction-time experiments or group interactions.Researchers with complex designs or who wish to include a high degree of customization can program their own surveys. Early web-based experiments were often coded in Java or Flash, but these languages are now largely obsolete, being unavailable on some platforms and switched off by default and having warning messages on others. Most web experiments are now developed using HTML5, JavaScript (which is not Java), and CSS – the three core technologies behind most Web pages. Broadly, HTML5 provides the content, the CSS provides the styling, and JavaScript provides the interaction. Directly coding using these technologies provides considerable flexibility, allowing presentation of animations, video, and audio content [91Reimers S. Stewart N. Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments.Behav. Res. Methods. 2015; 47: 309-327https://doi.org/10.3758/s13428-014-0471-1Crossref PubMed Scopus (98) Google Scholar, 92Reimers S. Stewart N. Auditory presentation and synchronization in Adobe Flash and HTML5/JavaScript Web experiments.Behav. Res. Methods. 2016; 48: 897-908https://doi.org/10.3758/s13428-016-0758-5Crossref PubMed Scopus (17) Google Scholar]. There are also advanced libraries and platforms to assist, including www.jsPsych.org which requires minimal programming [93de Leeuw J. Jspsych: A javascript library for creating behavioral experiments in a web browser.Behav. Res. Methods. 2015; 47: 1-12https://doi.org/10.3758/s13428-014-0458-yCrossref PubMed Scopus (579) Google Scholar], the online platform www.psiturk.org [94Gureckis T.M. et al.Psiturk: An open-source framework for conducting replicable behavioral experiments online.Behav. Res. Methods. 2016; 48: 829-842https://doi.org/10.3758/ s13428-015-0642-8Crossref PubMed Google Scholar], www.psytoolkit.org which allows online or offline development [95Stoet G. PsyToolkit: A software package for programming psychological experiments using Linux.Behav. Res. Methods. 2010; 42: 1096-1104Crossref PubMed Scopus (352) Google Scholar, 96Stoet G. Psytoolkit: A novel web-based method for running online questionnaires and reaction-time experiments.Teach. Psychol. 2017; 44: 24-31https://doi.org/10.1177/0098628316677643Crossref Scopus (370) Google Scholar], and Flash-based scriptingRT [97Schubert T.W. et al.ScriptingRT: A software library for collecting response latencies in online studies of cognition.PLoS One. 2013; : 8https://doi.org/10.1371/journal.pone.0067769Crossref Scopus (44) Google Scholar].Web pages are not displayed in the same way across all common browsers, and so far not all browsers support all of the features of HTML5, JavaScript, and CSS. Further, the variety of browsers, browser versions, operating systems, hardware, and platforms (e.g., PC, tablet, phone) is considerable. It is certainly important to test a web-based experiment across many platforms, especially if the exact details of presentation and timing are important. Libraries such as jQuery can help to overcome some of the cross-platform difficulties.Box 4Online Reaction TimesMany cognitive science experiments require accurate measurement of reaction times. Originally these were recorded using specialist hardware, but the advent of the PC allowed recording of millisecond-accurate reaction times. It is now possible to measure reaction times sufficiently accurately in web experiments using HTML5 and Javascript. Alternatively, MTurk currently permits the downloading of software which allows products such as Inquisit Web to be used to record reaction times independently of the browser.Reimers and Stewart [91Reimers S. Stewart N. Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments.Behav. Res. Methods. 2015; 47: 309-327https://doi.org/10.3758/s13428-014-0471-1Crossref PubMed Scopus (98) Google Scholar] tested 20 different PCs with different processors and graphics cards, as well as a variety of MS Windows operating systems and browsers using the Black Box Toolkit. They compared tests of display duration and response timing using web experiments coded in Flash and HTML5. The variability in display and response times was mainly due to hardware and operating system differences, and not to Flash/HTML5 differences. All systems presented a stimulus intended to be 150 ms for too long, typically by 5–25 ms, but sometimes by 100 ms. Furthermore, all systems overestimated response times by between 30–100 ms and had trial-to-trial variability with a standard deviation of 6–17 ms (see also [35de Leeuw J.R. Motz B.A. Psychophysics in a web browser? Comparing response times collected with javascript and psychophysics toolbox in a visual search task.Behav. Res. Methods. 2016; 48: 1-12https://doi.org/10.3758/s13428-015-0567-2Crossref PubMed Scopus (101) Google Scholar]). If video and audio must be synchronized, this might be a problem. There are large stimulus onset asynchronies of ∼40 ms across different hardware and browsers, with audio lagging behind video [92Reimers S. Stewart N. Auditory presentation and synchronization in Adobe Flash and HTML5/JavaScript Web experiments.Behav. Res. Methods. 2016; 48: 897-908https://doi.org/10.3758/s13428-016-0758-5Crossref PubMed Scopus (17) Google Scholar]. Results for older Macintosh computers are similar [98Neath I. et al.Response time accuracy in Apple Macintosh computers.Behav. Res. Methods. 2011; 43: 353-362https://doi.org/10.3758/s13428-011-0069-9Crossref PubMed Scopus (38) Google Scholar].The measurement error added by running cognitive science experiments online is, perhaps surprisingly, not that important [99Ulrich R. Giray M. Time resolution of clocks: Effects on reaction time measurement-Good news for bad clocks.Brit. J. Math. Stat. Psy. 1989; 42: 1-12https://doi.org/10.1111/j.2044-8317.1989.tb01111.xCrossref Scopus (53) Google Scholar]. Reimers and Stewart simulated a between-participants experiment comparing two conditions with a known 50 ms effect size. Despite the considerable variability introduced by the (simulated) hardware differences across (simulated) participants, only 10% more participants are necessary to maintain the same power as in the counterfactual experiment with zero hardware bias and variability [91Reimers S. Stewart N. Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments.Behav. Res. Methods. 2015; 47: 309-327https://doi.org/10.3758/s13428-014-0471-1Crossref PubMed Scopus (98) Google Scholar]. In the more usual within-participants experiment, where the constant biasing of response times cancels the difference between conditions, there is no effective loss from hardware differences (see also [100Brand A. Bradley M.T. Assessing the effects of technical variance on the statistical outcomes of web experiments measuring response times.Soc. Sci. Comput. Rev. 2012; 30: 350-357https://doi.org/10.1177/0894439311415604Crossref Scopus (24) Google Scholar]). Accuracy is higher using the Web Audio API [92Reimers S. Stewart N. Auditory presentation and synchronization in Adobe Flash and HTML5/JavaScript Web experiments.Behav. Res. Methods. 2016; 48: 897-908https://doi.org/10.3758/s13428-016-0758-5Crossref PubMed Scopus (17) Google Scholar]. Reaction-time data have been collected and compared in the lab and online, with similar results for lexical decision and word-identification times, and Stroop, flanker, Simon, Posner cuing, visual search, and attentional blink experiments [36Crump M.J. et al.Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research.PLoS One. 2013; 8: e57410https://doi.org/10.1371/journal.pone.0057410Crossref PubMed Scopus (1023) Google Scholar, 37Hilbig B.E. Reaction time effects in lab- versus web-based research: Experimental evidence.Behav. Res. Methods. 2016; 48: 1718-1724https://doi.org/10.3758/s13428-015-0678-9Crossref PubMed Scopus (96) Google Scholar, 101Semmelmann K. Weigelt S. Online psychophysics: Reaction time effects in cognitive experiments.Behav. Res. Methods. 2016; (Published online August 5, 2016)https://doi.org/10.3758/s13428-016-0783-4Crossref Scopus (58) Google Scholar, 102Slote J. Strand J.F. Conducting spoken word recognition research online: Validation and a new timing method.Behav. Res. Methods. 2016; 48: 553-566https://doi.org/10.3758/s13428-015-0599-7Crossref PubMed Scopus (27) Google Scholar]. Infant Attention Parents have been recruited on MTurk to record the gaze patterns of their infants using a webcam while the infants views video clips [78Scott K. Schulz L. Lookit (part 1): A new online platform for developmental research.Open Mind. 2017; 1: 4-14https://doi.org/10.1162/opmi_ a_00002Crossref Google Scholar, 79Tran M. et al.Online recruitment and testing of infants with Mechanical Turk.J. Exp. Child Psychol. 2017; 156: 168-178https://doi.org/10.1016/j.jecp.2016.12.003Crossref PubMed Scopus (25) Google Scholar]. Some participants were excluded because of technical issues (e.g., the webcam recording became desynchronized from the video) or because the webcam recording was of insufficient quality to view the eyes of the infant. However, it was possible to identify the features of videos that attracted infant attention, including singing, camera zooms, and faces. Economic Games Many economic games have been run on MTurk, including social dilemma games and the prisoner's dilemma [69Rand D.G. et al.Social heuristics shape intuitive cooperation.Nat. Commun. 2014; 5: e3677https://doi.org/10.1038/ ncomms4677Crossref PubMed Google Scholar]. The innovation here was to have remotely located participants playing live against one another, where one person's decision affects the outcome that another receives, and vice versa. Software platforms to implement economic games include Lioness [80Arechar A. et al.Conducting interactive experiments online.Exp. Econ. 2017; (Published online May 9, 2017)https://doi.org/10.1007/s10683-017-9527-2Crossref PubMed Scopus (126) Google Scholar] and NodeGame [81Balietti S. nodeGame: Real-time, synchronous, online experiments in the browser.Behav. Res. Methods. 2016; (Published online November 18, 2016)https://doi.org/10.3758/s13428-016-0824-zCrossref PubMed Scopus (18) Google Scholar]. Crowd Creativity Several researchers have assembled groups of workers to engage in iterative creative tasks [82Yu L. Nickerson J.V. Cooks or cobblers? Crowd creativity through combination.Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 2011: 1393-1402Google Scholar]. More recently, workers have collaboratively written short stories, and changes in the task structure influence the end-results [83Kim, J. et al. (2016) Mechanical novel: Crowdsourcing complex work through reflection and revision. Comput. Res. Repository. http://dx.do
Referência(s)