Reviewing Journal Rankings and Revisiting Peer Reviews: Editorial Perspectives*
2007; Wiley; Volume: 44; Issue: 4 Linguagem: Inglês
10.1111/j.1467-6486.2007.00701.x
ISSN1467-6486
Autores Tópico(s)Management and Marketing Education
ResumoIn this article we respond to the key points made by Macdonald and Kam (2007) in relation to journal quality and the peer review process. Whilst we appreciate that their tone is intentionally provocative, the picture they present is one of unremitting gloom and reluctant acquiescence to a system out of control. It is as if the publication process has a series of self-supporting logics that separate it from any notion of publishing in order to benefit the discipline through the advance of knowledge and understanding. From this perspective the publishing process and the consequent content of management journals are presented as the outcome of a series of 'games' that put more emphasis on where someone publishes than on what they publish and its subsequent impact. Such criticisms are not new in that they have been vigorously discussed for decades across a range of disciplines. Furthermore, many of these issues are raised whenever academics get together and discuss their experiences of journal publishing. Given the frustrations and vagaries of the review and publication process, such complaints are understandable. But they deserve further scrutiny. We write this article as two of the General Editors of Journal of Management Studies. This is considered by the broad management studies community to be a 'quality journal' and during our time as General Editors, so far, we have overseen the reviewing of 1463 articles. We are therefore insiders. Our broad purpose is to show how journal editors need to intervene in order build and maintain a journal's reputation since it is not as impervious to change as Macdonald and Kam (2007) imply, and to mitigate some of the problems associated with peer review (see also Bedeian, 2004; Campanario, 1998; Miller, 2006; Starbuck, 2003, 2005). In doing so, we demonstrate that journal editors have actively to engage with these issues if they are to ensure that their journal continues to create value for the community at large. Macdonald and Kam's critique therefore reminds us why we have instituted certain practices at JMS. We begin with some general comments about journal quality before turning to the peer review process. Macdonald and Kam (2007) are right in pointing out that the importance of a journal's ranking on lists of journal quality, of which there are now a great number within countries and between countries (see, for example, the list available at Harzing's website, http://www.harzing.com), has assumed greater pre-eminence. Furthermore, whereas the ranking of journals in which people publish has always had implications for personal reputation, peer recognition and career advancement, increasingly it is impacting much more directly on institutional reputations and rankings and in turn on potential recruitment of staff and students. Individual publication choices and success are therefore assuming greater institutional significance. In part this is a result of the increasing importance and proliferation of a number of rankings of different types of Masters programmes (e.g. the Financial Times list of 40 journals used as part of their assessment of MBAs). These often contain an assessment of an institution's research output on the basis of articles that faculty publish in a limited and specified list of journals. These issues are also particularly marked in those countries where departmental funding is partly linked to an aggregate research quality score based on peer assessment of the published work submitted for each member of staff. We agree that a range of forces are making discussions of the relative quality of journals more numerous and heated than in the past. Furthermore, it is clear that the isomorphic pressures alluded to earlier are encouraging institutions around the world to give stronger guidance to faculty in terms of which journals they should submit their manuscripts to; publication in such journals has taken up a more prominent position in tenure and promotion decisions. Although incentive practices vary and can be considerable across institutions, the general outcome of this process is that more material is being submitted to a narrow group of journals that the community identifies as being of high quality. Macdonald and Kam are correct in arguing that this trend is reinforcing the status of these journals. However, they perhaps overestimate the solidity of a journal's prestige and standing within the community since the relative status of a journal can change over time, sometimes quite dramatically. Indeed, on the basis of Kuhn's (1962) approach to understanding how new scientific fields emerge, we would expect new journals to be created to publish work in these areas which may in turn become more prestigious as the quality and contribution of the work is recognized. For example, in the finance area, the Journal of Financial Economics (JFE) was established as a result of frustrations by some academics (Michael Jensen and others) that the Journal of Finance was not interested in publishing research in their areas (Jensen, 2006). Some time later, the Review of Financial Studies was established by those academics frustrated at the editorial policy of JFE under Jensen's tenure. Comparing the top ten journals in the ISI citation ranking in Management in 1995 and 2005, we can see that there have been some significant changes. Some new journals have appeared in the list and assumed a top position very rapidly, and there has been upwards and downwards movement of a number of established journals. Only the Academy of Management Review, Academy of Management Journal and Administrative Science Quarterly have remained in the top 10 during this ten year period. The point is that what are considered quality journals changes over time. It is also interesting to note that journals with a broader remit tend to be more stable than those with a narrower focus. Furthermore, over this period the number of broad-based management journals in the upper reaches of journal quality lists has increased. The polarization that Kam and Macdonald bemoan would be more worrying if the 'quality' journals were increasingly associated with particular domains, theoretical positions or methodological approaches. However, the reverse appears to have happened, with the consequence that there are a greater number of opportunities for a broad body of work. We as editors are very aware that we cannot take for granted the regard within which a journal is held. This is something that we have to constantly manage. For example, it does not take much of a slip in review decision times or time to publication for word to spread within the community and submissions to suffer. Whilst broad measures of journal quality, such as citation counts, may not change for some time, academic behaviour towards journals does in that word-of-mouth reports, particularly on certain aspects of the reviewing process, can greatly influence submission patterns. With this in mind we turn to discuss peer review. A quick immersion into the extensive literature on the peer review process indicates that the situation is possibly more serious than that identified by Macdonald and Kam. There are a number of failings that either individually or in combination may prevent or forestall innovative material from being published in highly regarded journals. In a much cited study, Peters and Ceci (1982) resubmitted 12 articles to the psychology journals in which they had already been published in the last 18–32 months. Prior to resubmitting the articles, the names and institutional affiliations of the authors were changed to fictitious ones (e.g. Tri-Valley Centre for Human Potential). To disguise the papers further a number of 'cosmetic' changes were made to the titles, abstracts and opening paragraphs in the introduction. Nine of the manuscripts were sent out for review (three were detected) and only one article was accepted, so that one journal made the same evaluation over the two periods. All that had changed was the authors' names and institutional affiliations. This article generated a voluminous response (over 70 commentaries in the same issue) and heated discussion over the nature of the peer review process. A review of this and other literature on the review process (e.g. Bedeian, 2004; Campanario, 1995, 1998; Miller, 2006; Starbuck, 2003) highlights three key flaws. First, reviewers are often accused of being overly harsh in their judgements. They approach manuscripts vigorously, looking to expose deficiencies, and fail to balance these with an equivalent appreciation of the merits. Reviewing thus becomes a fault finding procedure rather than one that seeks to engage constructively with the ideas, arguments and evidence in the paper. As Starbuck (2003) suggests, this approach may in part arise from reviewers seeing themselves as superior to the authors (in their role as reviewers). As he comments, 'Occasionally reviewers seem arrogant, disrespectful, even nasty' (p. 344). More generally, any system where people are evaluating each other, particularly anonymously, results in a hierarchical ordering that has to be carefully managed. Where editors privilege the reviewers' pronouncements, these change from being treated as suggestions to judgements that have to be attended to, no matter how carefully phrased. Failure to do otherwise may result in the rejection of the manuscript. This can lead to a situation where authors feel their ideas are unappreciated, undervalued and misunderstood. Taken to extremes, authors may feel that they are being forced to write the paper the reviewers want them to write, or respond to points that are superficial, misguided or just wrong. In these circumstances reviewers may become uninvited ghost-writers. Second, as Peters and Ceci (1982) suggest, a number of factors have been found to bias reviewer judgements (see Campanario (1998) for a full review). Where authors' identities are known, their institutional affiliations and social networks may influence judgements. However, these factors are reduced when a blind review process is used, although not eradicated because with the internet and electronic dissemination of manuscripts, an author's identity is very difficult to hide (see Hillman and Rynes, 2007). Here issues such as the apparent complexity of language, use of citations, appropriate jargon, sophisticated statistical procedures, and presentation of positive and significant results that do not merely replicate previous findings have all been found to influence or bias decisions. Third, and perhaps the area which has received greatest empirical attention, studies in a range of disciplines have consistently reported low overall inter-referee agreement (see Cicchetti, 1980, 1991; Gottfredson, 1978; Hendrick, 1977; Scar and Weber, 1978; Scott, 1974). Evidence within management studies is more scant. Starbuck (2003) reports a study he undertook whilst editor of Administrative Science Quarterly. An examination of 500 pairs of reviews revealed a correlation of 0.12. He concluded that 'It was so low that knowing what one reviewer had said about a manuscript would tell me almost nothing about what a second reviewer had said or would say' (p. 346). This review indicates that the points raised by Macdonald and Kam in relation to peer review have been identified in a number of disciplines. Peer review is not just a management problem; it is a problem for all academic disciplines. If we reject the arguments and findings of this body of literature then we close down any discussion as to how it might be improved. These criticisms of the review process therefore help remind us that as journal editors we have a responsibility to ensure that certain practices are instituted, that these are clearly communicated to all relevant parties and that we revisit them on a regular basis to ensure that they are working as intended. In the reflective spirit encouraged by this section of the Journal, we identify a number of areas where we have sought to mitigate the impact of some of these factors. Admittedly, our approach is designed to enhance the workings of the existing system, by addressing some key areas where actions can be taken, rather than seeking to replace it. This is only one possible approach. Easton (2007) examines more radical ones. Some commentators have argued that authors ought to be able to select, or at least nominate, potential referees (Bedeian, 2004; Campanario, 1998). This argument builds on the notion that, in writing their article, authors have become immersed in the field. They are therefore in the best position to determine who amongst their peers is able to judge the level and nature of their contribution. Indeed, it may be that editors are not experts in every area that is encompassed by the scope of their journal (this is a particular issue for journals that have a broad remit such as Journal of Management Studies). However, apart from the possible bias that this introduces in that authors may choose people who are well pre-disposed to them and their work, it is not as if authors do not already have some influence on the selection of reviewers. At JMS we ask all authors to write a contextual note in which they identify not only their contribution in relation to prior work but why the article is particularly relevant to the Journal. In doing this, authors clearly identify not only the broad community that is best able to judge the article but also that segment of it which is relevant to the interests of the Journal. Some of these individuals may or may not have reviewed for the Journal before. In addition, a close reading of any article, if it is well constructed, cogently written and positioned in relation to prior work, should enable an editor to identify a number of individuals who possess the relevant subject specific knowledge and/or technical expertise. We therefore go to considerable lengths to invite reviewers who we believe are familiar with the area that a paper is addressing and the method that is being used. We believe that this provides for a fairer review process than selecting one or more reviewers as 'general readers' or selecting reviewers only from the editorial board. In this way it is possible to identify peers who possess knowledge that is equivalent to that which was needed to author the article. It is often the case that they possess greater knowledge! It should be remembered that as editors we do not mandate people to review articles. Indeed, reviewing is a voluntary activity that we all undertake for a variety of professional reasons. When we send an invitation to a potential reviewer we make it clear that if they feel the article is outside their areas of expertise they should decline to review. We would all recognize that not every article submitted to a journal should or could be accepted for publication. Indeed, the acceptance rates for leading journals are well below 10 per cent, in some cases below 5 per cent, and falling as submissions increase. This partly relates to a practical issue of page budgets but also to that fact that a key role for journals is to 'certify knowledge' (Merton, 1973). The knowledge presented in journals represents certain standards and values that differentiate it from other knowledge forms. Editors of academic journals are not trying to edit a newspaper, for example. Publication in a journal therefore means that an article is expected to minimally reflect certain norms about the practice of theory-building and empirical research. In addition, an article is expected to advance knowledge in some way. In general, so-called quality journals are differentiated on the basis of the significance of this contribution to knowledge. Clearly, many articles may fail to meet these basic criteria. Indeed, as we noted in an earlier commentary on publishing in Journal of Management Studies (Clark et al., 2006), in 92 per cent of cases the reason why a paper was rejected was because it failed to make a contribution. Editorial decisions are therefore focused on evaluating the relative quality of a small number of manuscripts since few submissions have the potential to make a significant contribution. In these circumstances providing a convincing signal of the contribution of a paper is absolutely critical both in areas that are roads well-travelled and in more novel areas where the author may be attempting to blaze a trail. In the former, reviewers and editors want to see that a paper is doing more than 'filling-in the pot-holes', no matter how neatly packaged the paper may be. The latter may be of two broad types. First, a paper may seek to bring a new theoretical lens to an established area; here it is important to demonstrate the shortcomings of existing approaches and how the new lens causes us to see a topic quite differently. Second, a paper may be opening up an entirely new topic that has not been addressed before. Following Kuhn (1962), we might expect such work to be published in more prestigious journals only once its validity has been established. For example, see the review of the development of university entrepreneurship by Rothaermel et al. (2007). There is some debate about what constitutes a contribution, especially a theoretical one (Sutton and Staw, 1995). Some papers may represent 'interim struggles in which people intentionally inch toward stronger theories' (Weick, 1995, p. 385) and ruling these out may slow inquiry in the early stages of theory development. Yet, as Barley (2006, p. 19) argues, 'there are limits on how far transgressive papers can go'. Truly innovative papers cannot 'break too many substantive, methodological or theoretical rules' without being considered wacky or off-beam. Thus, even innovative work has to conform to certain 'genre constraints'. However, deciding on the extent of a paper's innovativeness is a matter of judgement. Ultimately that judgement is the role of the editor(s) since a decision will often need to be made about trading off the rough edges of a potentially path breaking paper with the need for a paper to meet a threshold of quality and rigour with which the research has been conducted. In order to minimize the chances of rejecting a paper that makes a significant contribution, JMS has an appeals process and also encourages dialogue throughout the review process, as we explain below. The degree of reviewer dissensus alluded to by Macdonald and Kam and in other studies reviewed above may be over-inflated. Studies examining the extent of reviewer agreement, with some exceptions, are based on the global or overall recommendation of reviewers (i.e. whether they jointly recommend: 'acceptance', 'revise with minor amendments', 'revise with major amendments', 'reject and recommend resubmission', or 'reject'). These studies fail to examine two critical issues: (1) whether disagreement at a summative level is reflected in the qualitative comments contained with the reviews themselves; and, (2) whether journal publishing is simply a matter of a reviewer vote. It is our experience that the points made in reviews are more often than not similar. Reviewers identify similar strengths and weaknesses in the paper. Where they differ is in terms of the emphasis that they may place on specific issues and then how they aggregate these into an overall evaluation. The point is that reviewers may judge claims through the filter of 'personal reading lenses' (Bedeian, 2004, p. 201) which result in slightly different, rather than completely unique, interpretations of papers. These then underpin their judgements as to the extent of the 'fatal flaws' within a paper and the likelihood of publication. Furthermore, whereas these studies are based on pairs of reviewers' decisions, Journal of Management Studies, like most leading management journals, uses at least three reviewers. In these circumstances, it is frequently the case that at least two reviewers agree on the key issues and make identical overall recommendations. A second deficiency with the research on peer review is that it assumes that the recommendations reviewers make are binding and that they therefore make the publication decisions. Yet it is the editors' responsibility to determine whether the reviewer reports represent an accurate and fair evaluation of the merits and weaknesses of a paper. It is also the editors' role to act as judges between reviewers' reports and authors' responses. In this respect we seek to be active editors rather than mere inexpressive conduits between authors and reviewers. If our judgements are more reliable than those of reviewers, then low levels of agreement in the summary judgements by reviewers may not be as important an issue as commonly assumed. However, we do not know how reliable our judgements are and there is plenty of evidence to indicate that the person in control feels most certain of reliability. But we do employ two systems that may mitigate our own biases. First, as an article nears the point of publication another editor independently reads the paper and provides a set of comments as to what they perceive to be the strengths and weaknesses. On the rare occasions involving a difference of opinion between editors, we first seek to resolve the difference. If further input is required, we then either seek the opinion of another General Editor or Associate Editor. Very occasionally, if the issue is very domain specific, we ask a member of the Editorial Board in that area to provide comments. Second, where an author feels particularly aggrieved by a decision we have a formal appeal procedure. Here, one of the General Editors or Associate Editors not involved in handling the paper and an editorial board member, completely independent of the process, will read the paper, reviews and editorial correspondence and make a judgement as to what should happen next. Given that we try and explain our decisions to authors in detail this is used very infrequently. In the handful of times this has been deployed in recent years, the initial decision has been confirmed in all cases except one where the authors were given a further opportunity to revise and resubmit the paper. As a further mechanism partly designed to reduce the potential problems associated with reviewing contentious and emerging topics, the Point–Counterpoint section in JMS (and similar sections in some other journals) provides a different forum from regular papers in which to air such ideas. We as editors may invite contributions on such topics or may respond to proposals put forward by prospective authors. This is an area where, with the support of the Society for the Advancement of Management Studies (SAMS) which provides a governance structure for the Journal, we have sought to introduce a number of initiatives that provide checks and balances for our work as editors. The Editors of the Journal are accountable for the way in which they manage the Journal to the Council of the Society. Indeed, SAMS advertises all editorial positions in the Journal on its website and in other leading general management journals. The Editors are selected following an interview with members of SAMS Council. As Editors, our performance is assessed in a number of ways. The Editorial Office collects all feedback from authors, reviewers and Editorial Board members on the nature of our general procedures, as well as how they have been applied and experienced in particular circumstances. Are they swift or slow, fair, biased, clear, opaque and so forth? Separately authors and reviewers submit to the Office comments based on their individual experiences of our procedures. These are all collected independently of the Editors, whether positive or negative, and submitted to the quarterly SAMS Council meetings as a standing agenda item. Council members are free to quiz the Editors on any of these comments. In addition, we meet with Editorial Board members at a number of international conferences and seek their feedback on what we are doing. Each Editor also conducts sessions on journal publishing at a range of forums in different parts of the world. Again our processes are opened to further scrutiny and these sessions provide opportunities for all attendees to discuss any issues that may be of concern. We also provide reviewers of a paper with a copy of the other reviewers' comments and of the editor's letter to the author; such transparency enables reviewers to see how their comments sit within the overall review process and provides them with an opportunity to comment on the decision made. Wherever possible we encourage dialogue between reviewers and authors as well as the editorial team. Quite separately, anyone in the community is free to contact us individually with regard to the decisions we make and the processes we employ. We are all very aware that the standing of a journal is related to the goodwill of the community at large and in particular to the willingness of individuals to either submit their papers for possible publication or to act as referees. In Bedeian's (2004, p. 212) list of recommendations aimed at increasing the integrity of the peer review process, he observed that 'Editors should not be trained "on the job" '. Drawing on Goodstein (1982), Bedeian's argument is that editors are not given the 'training necessary to fulfil their roles'. Editors are selected for a variety of reasons, some of which include their research skills, standing in the community, access to networks, professional visibility and so forth. His point is that these factors do not necessarily make good editors. So where might editors obtain their training and what kind of training is necessary? Given the focus of his article, Bedeian concentrates on the skills necessary for making judgements on manuscripts and identifying reviewers. A perusal of editors of any leading journal within the management field shows that prior to their appointment they are people who have often been involved in running journals, whether as members of an editorial team or editorial board, and have considerable experience as a reviewer for a range of journals. In other words, they have been making judgements on manuscripts in a number of capacities prior to being appointed an editor.[1] One should also not forget that editors are increasingly part of a team. The days of the sole editor are fast diminishing. Due to a huge increase in submissions over recent years (222 per cent from 2002 to 2005), we now have six editors at the Journal. Whilst we may have responsibility for overseeing the reviewing of particular manuscripts, we are constantly discussing papers and reviews with each other at different stages in the review process. Prior to final acceptance, a paper is read by another editor in order to ensure consistency of decision making between the editors. Finally, at editorial meetings we have a standing item where we discuss a number of case studies so that we may learn from each other. Whether or not members of the community are interested in training with respect to editorial duties is another issue. At an international conference attended by over 1000 delegates, one of the Editors participated in a 'meet the editors' session and organized a workshop on 'becoming an editor'. The former was attended by over 100 delegates, the latter by three. Gaining insights into writing papers for publication is clearly, and understandably, more important for many people in the field than becoming an editor. Pressures that are combining to make discussions of journal and article quality more heated than before have drawn attention to weaknesses in the peer review process. However, these are long-standing issues that have exercised researchers in many disciplines, not just management. We have tried to show in this article that as editors we are highly cognisant of the potential shortcomings of the peer review process. A number of approaches can be adopted to mitigate these problems which we identify as: selection of reviewers, journal acceptance rates, dealing with reviewer dissensus and bias, greater transparency in the operation of the journal that places checks and balances on our work as editors, and editorial training and learning from other editorial team colleagues. Overall, as editors we seek to ensure that the peer review process provides for the fairest outcome for authors while at the same time fulfilling our role as stewards of the Journal's long-term standing and reputation in the community at large. To the extent that JMS is representative of editing and review practices more generally, we can interpret Macdonald and Kam's concerns as helpful reminders rather than causes for alarm.
Referência(s)