Portal de Periódicos da CAPES

Strange Bedfellows: Evidence of Accuracy in Professional Performance

2003; American Library Association; Volume: 43; Issue: 2 Linguagem: Inglês

2163-5242

Danny P. Wallace, Connie Van Fleet,

Library Science and Information Literacy

A brief article in a local newspaper revealed a disturbing set of statistics. In an unobtrusive test conducted by investigators from the Treasury Department, employees of the Internal Revenue Service provided correct answers to 57 percent of questions asked. The direct implication is that nearly half a million taxpayers may have received incorrect answers between July and December 2002. Furthermore, that 57 percent can be subdivided into 45 percent of questions answered correctly and completely and 12 percent answered basically correctly but incompletely. Perhaps most disturbingly, for 12 percent of questions posed by the surrogate taxpayers, IRS employees instructed the questioner to find his or her own answer, a direct violation of IRS policy. (1) The 55 Percent Rule That figure of 57 percent is remarkably close to the infamous 55 Percent Rule that has aroused so much light and heat in studies of the success rates of reference librarians. The term appears to have been formulated by Hernon and McClure, who summarized unobtrusive evaluation of reference services as revealing that staff generally answer 50-60 percent of the questions correctly. (2) Earlier studies revealed similar success rates. (3) A series of studies conducted at the University of Illinois Library Research Center in the 1980s emphasized that success rates can vary widely, even when the same methodology is applied consistently in repeated tests. (4) A recent study by Kaske and Arnold suggests that the 55 Percent Rule may apply to chat and e-mail reference. (5) Recent work by Richardson and Saxton has suggested that such studies are inherently flawed in that they used unacceptably small samples and unsophisticated models for assessing success. (6) Analogs from Other Arenas The newspaper article regarding success rates for IRS employees provides a very close analog to studies of reference success and invites a search for studies of success in other arenas. The results of such a search in areas other than reference are interesting, and in some cases alarming. Here are just a few: * Web usability expert Jakob Nielsen found that users of Web pages successfully complete fewer than 50 percent of desired tasks. (7) * A study of estimates made by Wall Street analysts between 1974 and 1991 indicated that they accurately predicted investment earnings about 45 percent of the time and that their accuracy was declining over time. (8) * Although oral health problems are among the most common chronic conditions experienced by older adults, family physicians and geriatricians were able to correctly diagnose oral health problems in older adults only about 55 percent of the time. (9) * Sixty-four percent of Secret Service agents, 58 percent of psychiatrists, 57 percent of judges, 56 percent of robbery investigators, and 56 percent of federal polygraphers are capable of determining whether an individual they are interviewing is lying. (10) How Bad Is 55 Percent? There are some environments in which a success rate of 55 percent would be viewed as exceptional or even unattainable. Examples from the world of sports are numerous: * Ty Cobb's career batting average of .366 has never been equaled. (11) * Dan Marino holds the record for passing yards completed, with a success rate of 59.4 percent. (12) * Kareem Abdul-Jabbar hit 53 percent of attempted field goals. (13) * Thirty-seven percent of horses that won both the Kentucky Derby and the Preakness went on to win the Triple Crown. (14) The Need for a Benchmark Studies of the accuracy of answers to reference questions have produced averages as low as 40 percent and as high as 90 percent. (15) A key problem in evaluating studies of reference accuracy is that there is no obvious target against which to compare results. A common thread in discussions of the 55 Percent Rule is a sense that it just can't be correct, that reference librarians surely must be performing at a higher level than that. …