Portal de Periódicos da CAPES

The Scored Society: Due Process for Automated Predictions

2014; University of Washington School of Law; Volume: 89; Issue: 1 Linguagem: Inglês

1942-9983

Danielle Keats Citron, Frank Pasquale,

Cybercrime and Law Enforcement Studies

[Jennifer is] ranked 1,396 out of 179,827 high school students in Iowa. . . . Jennifer's score is the result of comparing her test results, her class rank, her school's relative academic strength, and a number of other factors. . . .[C]an this be compared against all the other students in the country, and maybe even the world? . . .That's the idea . . . .That sounds very helpful. . . . And would eliminate a lot of doubt and stress out there.-Dave Eggers, The Circle1INTRODUCTION TO THE SCORED SOCIETYIn his novel The Circle, Dave Eggers imagines persistent surveillance technologies that score people in every imaginable way. Employees receive rankings for their participation in social media.2 Retinal apps allow police officers to see career criminals in distinct colors-yellow for low-level offenders, orange for slightly more dangerous, but still nonviolent offenders, and red for the truly violent.3 Intelligence agencies can create a web of all of a suspect's contacts so that criminals' associates are tagged in the same color scheme as the criminals themselves.4Eggers's imagination is not far from current practices. Although predictive algorithms may not yet be ranking high school students nationwide, or tagging criminals' associates with color-coded risk assessments, they are increasingly rating people in countless aspects of their lives.Consider these examples. Job candidates are ranked by what their online activities say about their creativity and leadership.5 Software engineers are assessed for their contributions to open source projects, with points awarded when others use their code.6 Individuals are assessed as to vote for a candidate based on their cable-usage patterns.7 Recently released prisoners are scored on their likelihood of recidivism.8How are these scores developed? Predictive algorithms mine personal information to make guesses about individuals' actions and risks.9 A person's on- and offline activities are turned into scores that rate them above or below others.10 Private and public entities rely on predictive algorithmic assessments to make important decisions about individuals.11Sometimes, individuals can score the scorers, so to speak. Landlords can report tenants to data brokers while tenants can check abusive landlords on sites like ApartmentRatings.com. On sites like Rate My Professors, students can score professors who can respond to critiques via video. In many online communities, commenters can in turn rank the interplay between the rated, the raters, and the raters of the rated, in an effort to make sense of it all (or at least award the most convincing or popular with points or karma). 12Although mutual-scoring opportunities among formally equal subjects exist in some communities, the realm of management and business more often features powerful entities who turn individuals into ranked and rated objects.13 While scorers often characterize their work as an oasis of opportunity for the hardworking, the following are examples of ranking systems that are used to individuals' detriment. A credit card company uses behavioral-scoring algorithms to rate consumers' credit risk because they used their cards to pay for marriage counseling, therapy, or tire-repair services.14 Automated systems rank candidates' talents by looking at how others rate their online contributions.15 Threat assessments result in arrests or the inability to fly even though they are based on erroneous information.16 Political activists are designated as likely to commit crimes.17And there is far more to come. Algorithmic predictions about health risks, based on information that individuals share with mobile apps about their caloric intake, may soon result in higher insurance premiums.18 Sites soliciting feedback on bad drivers may aggregate the information, and could possibly share it with insurance companies who score the risk potential of insured individuals. …