Revised guidelines to enhance the rigor and reproducibility of research published in American Physiological Society journals
2018; American Physiological Society; Volume: 315; Issue: 6 Linguagem: Inglês
10.1152/ajpregu.00274.2018
ISSN1522-1490
AutoresGina L. C. Yosten, Josephine C. Adams, Christina N. Bennett, Nigel W. Bunnett, Rita Scheman, Curt D. Sigmund, Bill J. Yates, Irving H. Zucker, Willis K. Samson,
Tópico(s)Academic Writing and Publishing
ResumoEditorialRevised guidelines to enhance the rigor and reproducibility of research published in American Physiological Society journalsGina L. C. Yosten, Josephine C. Adams, Christina N. Bennett, Nigel W. Bunnett, Rita Scheman, Curt D. Sigmund, Bill J. Yates, Irving H. Zucker, and Willis K. SamsonGina L. C. YostenDepartment of Pharmacology and Physiology, Saint Louis University, St. Louis, Missouri, Josephine C. AdamsDepartment of Biochemistry, University of Bristol, Bristol, United Kingdom, Christina N. BennettPublications Department, The American Physiological Society, Rockville, Maryland, Nigel W. BunnettDepartments of Surgery and Pharmacology, Columbia University, New York, New York, Rita SchemanPublications Department, The American Physiological Society, Rockville, Maryland, Curt D. SigmundDepartment of Pharmacology, University of Iowa, Iowa City, Iowa, Bill J. YatesDepartment of Otolaryngology, University of Pittsburgh, Pittsburgh, Pennsylvania, Irving H. ZuckerDepartment of Cellular and Integrative Physiology, University of Nebraska Medical Center, Omaha, Nebraska, and Willis K. SamsonDepartment of Pharmacology and Physiology, Saint Louis University, St. Louis, MissouriPublished Online:11 Dec 2018https://doi.org/10.1152/ajpregu.00274.2018This is the final version - click for previous versionMoreSectionsPDF (75 KB)Download PDF ToolsExport citationAdd to favoritesGet permissionsTrack citations ShareShare onFacebookTwitterLinkedInWeChat A challenge in modern research is the common inability to repeat novel findings published in even the most "impact-heavy" journals. In the great majority of instances, this may simply be due to a failure of the published manuscripts to include—and the publisher to require— comprehensive information on experimental design, methods, reagents, or the in vitro and in vivo systems under study. Failure to accurately reproduce all environmental influences on an experiment, particularly those using animals, also contributes to inability to repeat novel findings. The most common reason for failures of reproducibility may well be in the rigor and transparency with which methodology is described by authors. Another reason may be the reluctance by more established investigators to break with traditional methods of data presentation. However, one size does not fit all when it comes to data presentation, particularly because of the wide variety of data formats presented in individual disciplines represented by journals. Thus, some flexibility needs to be allowed. The American Physiological Society (APS) has made available guidelines for transparent reporting that it recommends all authors follow (https://www.physiology.org/author-info.promoting-transparent-reporting) (https://www.physiology.org/author-info.experimental-details-to-report). These are just some of the efforts being made to facilitate the communication of discovery in a transparent manner, which complement what has been a strength of the discipline for many years—the ability of the scientists and scientific literature to self-correct (1a).Reproducibility is a problem, not only in the publication of results, but also in grant applications. Quite often, important details are omitted, making it difficult for reviewers to judge the merit of individual proposals in an unbiased manner. In 2012, the National Institute of Neurological Disorders and Stroke (NINDS) organized a workshop to address these issues. In their report, Landis and coauthors (11) identified shortcomings in rigor in terms of study design and reporting. They identified several key issues that needed more attention, not only in publications so that reproducibility would be enhanced, but also in funding requests. These included issues of study design, such as randomization, blinding, and sample size determination. Also identified were several important elements of data handling, including unplanned interim analyses (which may lead to false assumptions), handling of data outliers, secondary end-point selection once the data are assembled, and the problem of how to define appropriately replicated data points (pseudo-replicates).RECENT FUNDER GUIDELINES FOR RIGOR AND REPRODUCIBILITYThe report by this NINDS working group and those of other groups commenting on shortcomings in clinical trials (15) and preclinical studies (2, 10) contributed to a refocusing of guidelines for enhancing scientific rigor and reproducibility published by the National Institutes of Health (NIH) (https://grants.nih.gov/reproducibility/index.htm) and the National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs, https://www.nc3rs.org.uk/arrive-guidelines). The NIH statement focused specifically on rigor and reproducibility in grant applications and addressed several key (and now required) elements of requests for funding. These included a clear definition of the scientific premise of the proposed work and a description of how the proposal would advance the field. Important areas of focus included strict attention (rigor) to scientific method, unbiased design, and data analysis and interpretation. The guidelines also addressed issues that contribute to reproducibility, requiring, in particular, a comprehensive description of biological variables (including but not limited to sex, age, weight, health conditions, and criteria for exclusion) and careful description of key biological resources. This latter issue is extremely important to the concept of reproducibility and places the burden of validation of reagents on the grant applicant and, as we will discuss below, potential authors. These issues also apply to non-USA authors because many funding bodies around the world have put in place new statements of policy (e.g., http://www.dfg.de/en/service/press/press_releases/2017/press_release_no_13/) or requirements for grant applications (e.g., https://mrc.ukri.org/documents/pdf/reproducibility-update-from-sponsors/).The Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines (14) outline a comprehensive compendium of recommendations for improving the reporting of animal research. ARRIVE extended beyond the NIH recommendations (above) by including issues such as appropriate description of article content in the title and abstract, clear delineation of the scientific objective as it relates to the existing knowledge base, and the inclusion of a concise ethical statement. That was just the start. The guidelines focus on detailed study design and the complete description of experimental procedures (rationale for species choice, experimental and control groups, group allocation and size, housing conditions, sex, health, time of day for experimentation, choice of appropriate anesthetics and analgesia, determination of dosing of test substances, and origin of reagents). How results are to be discussed and conclusions drawn also came under scrutiny, as did requirements for clear reporting of exclusion criteria and outcomes, observed adverse effects, and justification of employed statistical analyses. Finally, the discussion of how the results related to the study objectives and hypotheses needed attention. This resulted in the recommendation that authors place their findings in the context of existing literature; that is, describe/explain what their results contribute that is novel or perhaps paradigm-shifting. A description of the translational nature of the work was recommended, as was a clear statement of the source of funding for the project.Although these NIH and NC3Rs guidelines focus primarily on in vivo, animal-based studies, these same issues apply when cell culture-based work is reported. This has been recognized as a particular issue for preclinical studies in cancer biology, leading to large-scale efforts to repeat selected experiments (9). Often, it is the omission of sufficient details in these studies, including a description of the source and handling of cell lines, reagent validation and procurement, experimental protocol, and replicate designation, that lead to a problem with reproducibility. Therefore, there is a need for all scientific journals to improve their standards for reporting, rigor, and reproducibility.NEW APS GUIDELINESThe American Physiological Society has recognized the need for increased transparency (https://www.physiology.org/author-info.promoting-transparent-reporting) and introduced new guidelines for authors in 2016, along with new questions for manuscript reviewers in early 2017. Members of the APS Publications Committee, APS editorial office staff, journal editors, and society leaders understood that authors would need time to put those new recommendations into practice and that reviewers would need time to become familiar with the changing standards. Now that authors have had the opportunity to adjust to what were initially recommendations and to consider their own best practices, the culture has become embedded widely and the time has come to require more compliance.In 2018, the APS introduced additional guidelines with regard to its policies on rigor and reproducibility. The new policies contain updated requirements for rigor of reporting and encourage authors to make variability within data sets more transparent.Those updated policies include the following requirements: 1. Figure legends are now required to include: a. Detail of the statistical test used for each experiment.b. Inclusion of "n" values for each condition in every panel.c. Reporting of sex for studies in vertebrates if not stated as only one sex in the methods (e.g., sex of originating cell lines should be included when known in the methods).d. Declaration of nonuniform image modifications.2. The methods section must describe: a. Antibody validation details if the reagent has not been previously validated.b. Information on whether control and treatment groups for microscopy experiments were collected at the same time and under the same conditions.3. Data presentation suggestion/requirements are noted: a. Presentation of bar graphs as "dot-whisker plots" is strongly encouraged; it is a requirement in some APS journals.b. It is recognized that data can be displayed in different ways, often requiring unique presentation style, and that not all data sets are best displayed in "dot-whisker plots." Thus, some flexibility must be allowed. In the following areas, authors are strongly encouraged to consider further aspects of rigor and data transparency: 4. Authors are strongly encouraged to include Research Resource Identifiers (RRIDs) to reference research resources (i.e., reagents, tools, and materials that are used in the experiments): a. methods sections should include source details for cells, animals, reagents, and other research tools.b. Details are available at https://scicrunch.org/resources.5. Authors are strongly encouraged to make data available: a. Primary data may be requested upon review and should be provided.b. Authors should consider storing primary data in a public repository linked to the manuscript.6. In support of data availability, the data supplement policy has been extended from long data sets, audio/video files, and computer simulations to allow the inclusion of supplementary figures/tables (e.g., for reagent validation) up to 10 multipaneled figures or tables and explanatory text: a. To better facilitate reporting of information and data that are auxiliary to the main content of the article, data supplements must be deposited into a community-recognized public repository (see: https://www.physiology.org/author-info.data-repositories) or to a data-type-specific public access repository (such as Dryad, Figshare, Zenodo, GitHub) that associates a unique identifier with the data.b. Reviewers and Editors will review and approve supplemental data files for publication. As mentioned above, some flexibility in terms of data presentation must be maintained because of discipline-specific types of data that may not be common in all journals' offerings. In other words, whereas the requirements may not vary among journals (items 1–3), some aspects of the move toward greater transparency (items 4–6) require oversight that is unique to each subdiscipline. Although the individual standards for rigor and reproducibility may vary from one journal to another, all APS journals share similar goals. Thus, enforcement of those unique standards is the responsibility of each journal. It is likely that over time, recommendations will become requirements, other recommendations will evolve, and specific journals will develop other recommendations and guidelines that are relevant for their disciplines.Although the list of requirements and suggestions for publication in APS journals addresses the recommendations of NIH for comprehensive reporting of scientific design and experimental details, one can see that it accommodates many of the more stringent recommendations of the ARRIVE guidelines. APS recognizes the importance of these guidelines and, by adopting the revisions to policy detailed above for publication in the APS journals, it is moving toward shared goals of improved reporting, rigor, and reproducibility. We must at the same time be mindful that these new requirements may place an additional burden on editors and reviewers. Just as the requirements and recommendations have evolved over time, we recognize the need to build an infrastructure to facilitate the efforts of the reviewers and not overburden them. This brings us back to self-correction in science. With time, as these become the norms of data collection and display that students are taught in laboratories, the burden should decrease. More senior scientists will learn, as they have for many years, to adapt if these new requirements and recommendations can be demonstrated to improve rigor and reproducibility.Importantly, several excellent editorials and recommendations have been published in APS journals that provide guidance for authors when adapting to the new Rigor and Reproducibility Instructions. For instance, guidelines for appropriate statistical evaluation of cardiovascular research have been recently published in AJP-Heart and Circulatory Physiology (12). In the past, Curran-Everett and Benos published an important editorial that outlines the need for appropriate statistical evaluation and the use of standard deviation and standard error (5). Bron and Bunnett (3), Brooks and Lindsey (4), and Delpire (6) recently provided guidance on the various ways of validating antibodies. These articles provide links to important databases and recommendations for acceptable validation techniques. Methods of particular relevance to cell physiology include important considerations for quantitative image analysis to assess protein colocalization (7), appropriate validation of antibodies and the avoidance of pitfalls for quantitation from immunoblots (6, 13), and the value of reporting on the sex of primary cells and cell lines (16). AJP-Cell Physiology has also published guidance to authors and reviewers for approaching the new reporting expectations (1).These are examples of resources available in The American Journal of Physiology that will help authors provide and apply appropriate rigor and reproducibility principles to their research publications.DISCLOSURESConflict of interest statement: No conflicts of interest, financial or otherwise, are declared by the authors.AUTHOR CONTRIBUTIONSG.L.Y. and W.K.S. drafted manuscript; G.L.Y., J.C.A., C.N.B., N.W.B., R.S., C.D.S., B.J.Y., and W.K.S. edited and revised manuscript; G.L.Y., J.C.A., C.N.B., N.W.B., R.S., C.D.S., B.J.Y., I.H.Z., and W.K.S. approved final version of manuscript.ACKNOWLEDGMENTSG. L. C. Yosten is an Associate Editor, and W. K. Samson is the Editor-in-Chief, American Journal of Physiology-Regulatory, Integrative and Comparative Physiology. J. C. Adams is the Editor-in-Chief, American Journal of Physiology-Cell Physiology. C. N. Bennett is the Associate Publisher for Ethics and Policy, the American Physiological Society. N. W. Bunnett is the Editor-in-Chief, American Journal of Physiology-Gastrointestinal and Liver Physiology. R. Scheman is the Director of Publications and Executive Editor, the American Physiological Society. C. D. Sigmund is the Chair of the Publications Committee of the American Physiological Society and the former Editor-in-Chief, American Journal of Physiology-Regulatory, Integrative and Comparative Physiology. B. J. Yates is the Editor-in-Chief, Journal of Neurophysiology. I. H. Zucker is the Editor-in-Chief, American Journal of Physiology-Heart and Circulatory Physiology.REFERENCES1. Adams JC. Onward and upward with transparent research reporting. Am J Physiol Cell Physiol 312: C355–C356, 2017. doi:10.1152/ajpcell.00045.2017. Link | ISI | Google Scholar1a. Anonymous. Editorial: the challenges of replication. eLife 6: e23693, 2017. doi:10.7554/eLife.23693.Crossref | PubMed | ISI | Google Scholar2. Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature 483: 531–533, 2012. [Editorial note: Nature 485, May 3, 2012]: doi:10.1038/483531a. Crossref | PubMed | ISI | Google Scholar3. Bron R, Bunnett NW. Antibodies: friend or foe? Am J Physiol Gastrointest Liver Physiol 309: G717–G718, 2015. doi:10.1152/ajpgi.00312.2015. Link | ISI | Google Scholar4. Brooks HL, Lindsey ML. Guidelines for authors and reviewers on antibody use in physiology studies. Am J Physiol Heart Circ Physiol 314: H724–H732, 2018. doi:10.1152/ajpheart.00512.2017. Link | ISI | Google Scholar5. Curran-Everett D, Benos DJ. Statistics, authors, and reviewers: the heart of the matter. Adv Physiol Educ 33: 80, 2009. doi:10.1152/advan.90216.2008. Link | ISI | Google Scholar6. Delpire E. Research antibodies: do not use them to stain your reputation. Am J Physiol Cell Physiol 309: C707–C708, 2015. doi:10.1152/ajpcell.zh0-7843-editorial.2015. Link | ISI | Google Scholar7. Dunn KW, Kamocka MM, McDonald JH. A practical guide to evaluating colocalization in biological microscopy. Am J Physiol Cell Physiol 300: C723–C742, 2011. doi:10.1152/ajpcell.00462.2010. Link | ISI | Google Scholar9. Jamieson KH. Crisis or self-correction: Rethinking media narratives about the well-being of science. Proc Natl Acad Sci USA 115: 2620–2627, 2018. doi:10.1073/pnas.1708276114. Crossref | PubMed | ISI | Google Scholar10. Kilkenny C, Parsons N, Kadyszewski E, Festing MF, Cuthill IC, Fry D, Hutton J, Altman DG. Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One 4: e7824, 2009. doi:10.1371/journal.pone.0007824. Crossref | PubMed | ISI | Google Scholar11. Landis SC, Amara SG, Asadullah K, Austin CP, Blumenstein R, Bradley EW, Crystal RG, Darnell RB, Ferrante RJ, Fillit H, Finkelstein R, Fisher M, Gendelman HE, Golub RM, Goudreau JL, Gross RA, Gubitz AK, Hesterlee SE, Howells DW, Huguenard J, Kelner K, Koroshetz W, Krainc D, Lazic SE, Levine MS, Macleod MR, McCall JM, Moxley RT III, Narasimhan K, Noble LJ, Perrin S, Porter JD, Steward O, Unger E, Utz U, Silberberg SD. A call for transparent reporting to optimize the predictive value of preclinical research. Nature 490: 187–191, 2012. doi:10.1038/nature11556. Crossref | PubMed | ISI | Google Scholar12. Lindsey ML, Gray GA, Wood SK, Curran-Everett D. Statistical considerations in reporting cardiovascular research. Am J Physiol Heart Circ Physiol 315: H303–H313, 2018. doi:10.1152/ajpheart.00309.2018. Link | ISI | Google Scholar13. McDonough AA, Veiras LC, Minas JN, Ralph DL. Considerations when quantitating protein abundance by immunoblot. Am J Physiol Cell Physiol 308: C426–C433, 2015. doi:10.1152/ajpcell.00400.2014. Link | ISI | Google Scholar14. McGrath JC, Drummond GB, McLachlan EM, Kilkenny C, Wainwright CL. Guidelines for reporting experiments involving animals: the ARRIVE guidelines. Br J Pharmacol 160: 1573–1576, 2010. doi:10.1111/j.1476-5381.2010.00873.x. Crossref | PubMed | ISI | Google Scholar15. Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. Int J Surg 9: 672–677, 2011. doi:10.1016/j.ijsu.2011.09.004. Crossref | PubMed | ISI | Google Scholar16. Shah K, McCormack CE, Bradbury NA. Do you know the sex of your cells? Am J Physiol Cell Physiol 306: C3–C18, 2014. doi:10.1152/ajpcell.00281.2013. Link | ISI | Google ScholarAUTHOR NOTESAddress for reprint requests and other correspondence: W. K. Samson, Dept. of Pharmacology and Physiology, Saint Louis University, 1402 South Grand Blvd., St. Louis, MO (e-mail: [email protected]edu). Download PDF Previous Back to Top Next FiguresReferencesRelatedInformation Collections Cited ByTime of day as a critical variable in biology15 June 2022 | BMC Biology, Vol. 20, No. 1Insufficient reporting of experimental variables as a cause for nonreproducibility in animal physiology? A case studyTorben Göpel and Warren W. Burggren2 September 2022 | American Journal of Physiology-Regulatory, Integrative and Comparative Physiology, Vol. 323, No. 3Quality Output Checklist and Content Assessment (QuOCCA): a new tool for assessing research quality and reproducibility26 September 2022 | BMJ Open, Vol. 12, No. 9No guidelines for vascular nerves?Jo G. R. De Mey, Ulf Simonsen, and Christian Aalkjær24 March 2022 | American Journal of Physiology-Heart and Circulatory Physiology, Vol. 322, No. 4A snapshot of statistical methods used in experimental immunoblotting: a scoping review13 May 2022 | 4open, Vol. 5A System to Easily Manage Metadata in Biomedical Research Labs Based on Open-source SoftwareBIO-PROTOCOL, Vol. 12, No. 9Guidelines for in vivo mouse models of myocardial infarctionMerry L. Lindsey, Keith R. Brunt, Jonathan A. Kirk, Petra Kleinbongard, John W. Calvert, Lisandra E. de Castro Brás, Kristine Y. DeLeon-Pennell, Dominic P. Del Re, Nikolaos G. Frangogiannis, Stefan Frantz, Richard J. Gumina, Ganesh V. Halade, Steven P. Jones, Rebecca H. Ritchie, Francis G. Spinale, Edward B. Thorp, Crystal M. Ripplinger, and Zamaneh Kassiri17 November 2021 | American Journal of Physiology-Heart and Circulatory Physiology, Vol. 321, No. 6Insufficient transparency of statistical reporting in preclinical research: a scoping review8 February 2021 | Scientific Reports, Vol. 11, No. 1Promoting our early career members at AJP-Lung: the Editorial Board Fellowship Program and the Next Generation Physiologist Highlights section at our JournalLarissa A. Shimoda, Chunxue Bai, Nathan W. Bartlett, Julie A. Bastarache, Carol Feghali-Bostwick, Susan J. Gunst, Wolfgang M. Kuebler, Eric P. Schmidt, and Rory E. Morty20 October 2021 | American Journal of Physiology-Lung Cellular and Molecular Physiology, Vol. 321, No. 5Announcing the Editorial Board Fellowship Program of the American Journal of Physiology-Lung Cellular and Molecular PhysiologyLarissa A. Shimoda, Chunxue Bai, Nathan W. Bartlett, Julie A. Bastarache, Carol Feghali-Bostwick, Wolfgang M. Kuebler, Y. S. Prakash, Eric P. Schmidt, and Rory E. Morty7 July 2021 | American Journal of Physiology-Lung Cellular and Molecular Physiology, Vol. 321, No. 1Scholarly Publishing and Scientific Reproducibility1 September 2021Reproducibility of the neurocardiovascular responses to common laboratory-based sympathoexcitatory stimuli in young adultsGabrielle A. Dillon, Zachary S. Lichter, Lacy M. Alexander, Lauro C. Vianna, Jing Wang, Paul J. Fadel, and Jody L. Greaney18 November 2020 | Journal of Applied Physiology, Vol. 129, No. 5AJP-Regulatory, Integrative and Comparative Physiology: Looking Toward the FutureGina L. C. Yosten7 July 2020 | American Journal of Physiology-Regulatory, Integrative and Comparative Physiology, Vol. 319, No. 1A reflection on my editorship of Journal of NeurophysiologyBill J. Yates27 May 2020 | Journal of Neurophysiology, Vol. 123, No. 6CORP: Assessing author compliance with data presentation guidelines for manuscript figuresKara Hansell Keehan, Michelle C. Gaffney, and Irving H. Zucker8 April 2020 | American Journal of Physiology-Heart and Circulatory Physiology, Vol. 318, No. 5Will the real Dahl S rat please stand up?John P. Rapp and Michael R. Garrett7 November 2019 | American Journal of Physiology-Renal Physiology, Vol. 317, No. 5A new initiative for AJP-Cell Physiology: "Making Cell Culture More Physiological"Josephine C. Adams29 May 2019 | American Journal of Physiology-Cell Physiology, Vol. 316, No. 6 More from this issue > Volume 315Issue 6December 2018Pages R1251-R1253 Copyright & PermissionsCopyright © 2018 the American Physiological Societyhttps://doi.org/10.1152/ajpregu.00274.2018PubMed30332303History Received 5 September 2018 Accepted 13 October 2018 Published online 11 December 2018 Published in print 1 December 2018 Metrics
Referência(s)