Portal de Periódicos da CAPES

The Future of Double-Blind Review in Management

2007; Wiley; Volume: 44; Issue: 4 Linguagem: Inglês

10.1111/j.1467-6486.2007.00703.x

1467-6486

Amy J. Hillman, Sara L. Rynes,

Management and Organizational Studies

The double-blind review process – where authors do not know the identity of their reviewers and reviewers do not know the identity of authors – has been viewed as critical to the peer review process in top-tier management journals. This two-way anonymity helps to facilitate objectivity by both reviewers and authors and to reduce the effects of bias – either positive or negative – that may accompany known identities or affiliations. For example, several studies have documented bias in the review process in favour of authors from prestigious universities (Peters and Ceci, 1982), particularly when reviewers come from similarly prestigious universities (Gordon, 1980). In addition, Broder (1993) found evidence of bias by female reviewers against female authors independent of paper quality and reviewer affiliation and experience. Preserving the anonymity of authors and reviewers can also reduce tendencies for reviewers to attempt to ingratiate themselves with prestigious authors or to be more critical of the work of unknown scholars. Without anonymity, junior reviewers may become hesitant to offer critical evaluations for fear of career reprisals (McCook, 2006). Given these findings, it is hardly surprising that journals with double-blind review processes tend to have lower acceptance rates, even for articles from authors affiliated with high-prestige universities (Blank, 1991). Proponents of double-blind review suggest that '. . . any factors that increase the probability of particularistic decisions or increase their consequences are not likely to benefit the majority of scientists. A relatively small proportion of such decisions spread over time may serve to give some groups and individuals substantial cumulative advantage, because publication itself is convertible into the "scarce" evidence of competence that makes future selection for further advantage then based on competence, and therefore universalistic. Thus, a particularistic advantage can soon be transformed into a universalistic one' (Beyer, 1978, p. 75). The rapidly escalating cumulative advantages to 'those that have' have also been described by Merton (1968, p. 58), who calls it the Matthew effect: 'The Matthew effect consists in the accruing of greater increments of recognition for particular scientific contributions to scientists of considerable repute and the withholding of such recognition from scientists who have not yet made their mark.' In the present case, of course, 'recognition' consists of acceptance or rejection of one's work by scientific journals. Double-blind review is not without its critics, however. Some contend that when reviewers have to sign their reviews, they are more likely to provide conscientious, developmental comments. For example, McCook (2006, p. 3) quotes the editor of Journal of the American Medical Association as saying, 'I've always signed every review I've ever done, because I know if I sign something, I'm more accountable. Juries are not anonymous, neither are people who write letters to the editor, so why are peer reviewers?' If reviewers are identified, they may be more likely to provide a fair set of comments given that they may have to defend their views publicly (Armstrong, 1982). Studies comparing double- and single-blind reviews are scarce, but yield interesting findings. For example, Laband and Piette (1994) compare economics journals with single- and double-blind processes and find those with double-blind review outperform those with single-blind review. Specifically, they find that articles published in double-blind review journals receive more citations than would be expected controlling for author characteristics, length, and journal prestige. However, they also find that economics journals with single-blind review (i.e. anonymous reviewers but not anonymous authors) have authors with higher mean citations than double-blind review journals. Baland and Piette speculate that, for example, a Nobel prize winner's bad paper may have a greater total impact than good papers by less well-known economists, and suggest that for this reason, high-prestige authors may prefer to submit to single-blind journals because their reputation may bias the reviews positively. (An alternative view might be that not just 'bad' papers, but more speculative or avant-garde papers, might be accepted by these journals from 'proven' authors.) A similar phenomenon may operate in the case of 'special research forums' (SRFs), a growing phenomenon in management journals. For example, Conlon et al. (2006) recently found that lower-prestige management journals tend to use 'invited' SRFs more often than do higher-prestige journals, and also that the authors so invited tend to have more previous publications than authors of double-blind SRF manuscripts. As such, it appears that lower-prestige journals may use 'invited' papers, or non-anonymous review and decision processes, as strategies for attracting high-prestige authors who are given certain advantages in return for the prestige they lend to the journal. Although double-blind review has been regarded as the 'gold standard' in management journals, recent developments are putting its future in jeopardy. We briefly examine a few of these and speculate on their impact. Double-blind review processes are not always successful in preventing identification of authors. For example, Academy of Management meetings and other conference presentations often coincide with journal review. In addition, reviewers are commonly assigned on the basis of expertise in manuscript content, which further increases the probability that reviewers attend conference sessions in the same topical area and thus become aware of authors' identities in advance of journal submission. These discoveries of identity tend to be more happenstance, however, than more recent challenges to double-blind review presented by new technological developments and practices regarding working paper publication on the web. Reviewers who wish to circumvent the double-blind review process have always had means to do so by attempting to discover an author's identity. For example, every university or college maintains websites, with most including access to current faculty vitas and some including links to current working papers or publications. If reviewers guess who an author might be, perusing his or her vita on a university's website might confirm the suspicion. In an environment where many of us are at our computers for most of the working day, it only takes a few minutes to pursue such an effort. Moreover, beyond university websites, search engines such as Google Scholar enable searches of working paper titles and authors, often from fairly obscure sources (e.g. departmental working paper series or presentations inside the home university). Using search engines requires even less time than visiting a university's webpage, and often results in more comprehensive results (e.g. because reviewers can search by title and because search engines locate working papers and presentations that have not yet found their way onto an author's vita). Clearly, these new technologies reduce the time needed for reviewers to discover the identity of authors, if they are so inclined. And, as previous research clearly shows, the potential for biased reviews increases once anonymity is compromised. Another relatively new practice with implications for the future of double-blind review is the increased popularity of working paper series developed by many universities or disciplines as a way to disseminate research ideas and manuscripts prior to journal review or publication. Many disciplines encourage posting of working papers to on-line working paper series such as the Social Science Research Network in order to receive comments and have work read prior to publication. Some universities are even using 'hits' and/or citations to working papers published on the web as part of their tenure, promotion, and annual merit pay evaluations, rather than solely relying upon citations to published journal articles. Tenure and promotion packets that include information regarding the number of citations per article on Google Scholar or top author/manuscript status within the management research network portion of SSRN are not uncommon. Undoubtedly, such working paper series have several important advantages. For example, by posting working papers and commentaries early, arguments can be solidified or altered, methodological design improved, and so on. In addition, given that journals generally have lag times for publication, working paper series allow authors to stake claims to their ideas faster and to disseminate them earlier than would otherwise be the case. Third, working paper series make it easier for scholars to identify others working in their area and to benefit from the very latest ideas that might be relevant to their own work. Fourth, the development of working paper series in Europe and the UK in part relates to the need to develop productivity measures for external assessment and as outlets for doctoral training programmes. These series can give a doctoral student tangible evidence of an impact and opportunities for feedback.[1] These arguments suggest that posting manuscripts to web working paper series can potentially benefit science as a whole, as well as individual scholars. However, on the less positive side, such working paper series almost certainly reduce the probability of double-blind review. As such, they are also likely to increase the probability of the types of biases discussed earlier when these papers subsequently enter the journal review process. Opponents to posting working papers on the web argue that not only will double-blind review be compromised, but that establishing ownership of ideas will be more, rather than less, difficult given the interactive processes of web-facilitated dialogue. In addition, most casual readers are not likely to spend as much time with a manuscript as a reviewer will, so the arguments that early comments will improve a manuscript's chance of journal acceptance are only as good as the peer review comments received. Thus, while there are advantages and disadvantages to working paper series, the question remains regarding what to do about double-bind review. The Academy of Management has been a strong proponent of double-blind review processes. As a result, Academy of Management journals currently require authors to remove a manuscript from any public postings prior to review. However, if universities continue to encourage faculty to post their working papers and to reward working paper 'hits' and citations, this can present a dilemma for authors, particularly if journal review turnaround times are long or if personnel decisions within business schools are heavily influenced by norms towards electronic posting of working papers. If management scholars are increasingly pressured to post working papers to raise the profile of their university or college, and if their rewards are based on doing so, authors may be reticent to remove papers from such working series for journal review. As a result, the Academy of Management is currently pondering how to respond to the changed environment of electronic working paper series and highly efficient electronic search engines. At this point, it looks highly unlikely that either trend (working paper series or electronic searching) will reverse itself. Indeed, it appears likely that such trends will continue to escalate. So, what should management journals do? Should we continue to make authors remove their manuscripts from working paper series prior to journal review? Even if papers are not posted, most scholars have recent vitas on their university websites that list papers that are currently under review. Should we also make authors remove working papers titles from their vitas? This would seem difficult to enforce at best. What other options do we have? Perhaps instead of enforcement policies we should turn towards socialization (Abbott, 1988; Becker and Carper, 1956; Ouchi, 1980). Trying to remove subjectivity from our research and grading has always been a part of our profession. Strong norms conveyed to our doctoral students and good examples set by faculty regarding the preservation of the blind review process may go a long way to facilitating objectivity on the part of reviewers. In addition, journals can emphasize in their instructions to reviewers the importance of considering the merit of the work, and not trying to guess the identity of the authors. (As editors, we have sometimes had reviewers say, 'I suspect this work is by Professor X, and they have been wrong!') Using performance evaluation of past reviews to select board members may also serve as a strong incentive to preserve anonymity or, at the very least, reward individuals whose reviews are always developmental and exhibit a strong sense of objectivity. While technology has clearly made it faster and easier to trace an author's identity, if we increase our efforts to socialize new scholars to the view that ours is a system based on merit and that the integrity of our journals depends on this, fewer reviewers may deliberately pursue author identification. A more controversial option is to move towards more interactive publishing such as the new Interactive Exchange Journal from the Society for Industrial and Organizational Psychology (SIOP) wherein articles are posted along with peer commentaries and responses from the original authors. But, even for this journal, the focal articles will be peer reviewed prior to being placed on the website, and while others are invited to submit commentaries, the editors will decide which to post. Thus, there still appears to be a role in these interactive formats for double-blind review. An even more controversial option would be to reconsider double-blind review and its role in our journals. If reviewers have easier access to information that compromises the double-blind nature of our process, should we simply accept this as a result of the information age and make all authors equally identifiable via single-blind review? Currently, we are not willing to take this step, but recognize that there may come a point where such a move makes sense. In the meantime, we suggest that journal editors and scholars conduct additional research on the effects of non-blind, single-blind, and double-blind review processes. In particular, studies within the management discipline of the potential biases involved with each, and the broader costs and benefits (e.g. citations, impact, etc) would be useful. In addition, we urge those who study the economic, social, and psychological effects of information technology and search behaviours to conduct studies of the effects of working paper series and electronic searches over time. Do these practices produce efficient markets, in the sense that good ideas win out regardless of who proposes them? Or does the access to authors' names and institutions produce a sort of 'winner-take-all' effect (Frank and Cook, 1995), whereby if two papers are posted on the same subject, only the one by the renowned author gets read and cited? The flood of accessible information combined with increased pressure on people's time suggests that the latter may be the dominant effect. However, that is an empirical question worthy of serious study.