Defining success in graduate school
2014; American Society for Cell Biology; Volume: 25; Issue: 13 Linguagem: Inglês
10.1091/mbc.e14-03-0793
ISSN1939-4586
AutoresSean Bell, Jessica Blumstein, Katja Brose, Adam Carroll, Jean Chang, Julia F. Charles, Elizabeth S. Haswell, Melissa Michelitsch, Julia C. Owens, Christopher K. Patil, Rebecca Smith, Jon Tupy, Emily Walsh, Tracy L. Ware,
Tópico(s)Health and Medical Research Impacts
ResumoMolecular Biology of the CellVol. 25, No. 13 Letter to the EditorFree AccessDefining success in graduate schoolis companion ofResponse to Bell et al.Sean M. Bell, Jessica Blumstein, Katja Brose, Adam Carroll, Jean Chang, Julia Charles, Elizabeth S. Haswell, Melissa Michelitsch, Julia C. Owens, Christopher K. Patil, Rebecca Smith, Jon Tupy, Emily Walsh, and Tracy WareSean M. BellBiomarin Pharmaceutical, Inc., Novato, CA 94949, Jessica BlumsteinVertex Pharmaceuticals, Boston, MA 02110, Katja BroseNeuron, Cell Press, Cambridge, MA 02139, Adam CarrollAmplion Research, Bend, OR 97702, Jean ChangBroad Institute of Massachusetts Institute of Technology and Harvard University, Cambridge, MA 02142, Julia CharlesHarvard Medical School and Brigham and Women's Hospital, Boston, MA 02115, Elizabeth S. HaswellDepartment of Biology, Washington University in Saint Louis, St. Louis, MO 63130, Melissa MichelitschDepartment of Biology and Chemistry, Skyline College, San Bruno, CA 94066, Julia C. OwensAtterocor, Inc. Ann Arbor, MI 48104, Christopher K. PatilSomerville, MA 02144, Rebecca SmithScience & Health Education Partnership, University of California–San Francisco, San Francisco, CA 94143, Jon TupyGenedata, San Francisco, CA 94109, Emily WalshTremont Therapeutics LLC, Somerville, MA 02143, and Tracy WareDepartment of Biology, Salem State University, Salem, MA 01970Published Online:13 Oct 2017https://doi.org/10.1091/mbc.e14-03-0793AboutSectionsView PDF ToolsAdd to favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinked InRedditEmail To the Editor:As graduates of the University of California–San Francisco (UCSF) Tetrad program from the matriculating classes of 1992–1996, we read with interest a recently published Perspective entitled "How Should We Be Selecting Our Graduate Students?," written by fellow alumnus and classmate Orion Weiner (Weiner, 2014). The author, who is now a member of the UCSF Tetrad faculty and cochair of graduate admissions, reported that success in this graduate program over the past 20 years correlated with years of undergraduate research experience and Graduate Record Examination (GRE) subject scores but did not correlate with other commonly used admissions metrics, such as undergraduate grades, general GRE scores, or ranking of the undergraduate institution. We applaud the author for taking on the important challenge of examining predictors of success in graduate school in the life sciences. As alumni of the program, however, we wish to respond to a number of issues, including the design and execution of the study and the implications of its conclusions. Most importantly, we propose a new, broader definition of success in graduate school. Our aim is to promote a discussion about meaningful, reliable, and scientific ways to define, analyze, and evaluate success in bioscience graduate programs.PROBLEMS WITH USING "SUCCESS" AS AN OUTCOME MEASUREGraduate programs in the biosciences have used essentially the same metrics to evaluate applicants to graduate school for decades, and we fully support the author's intention to evaluate whether or not these metrics continue to serve as predictors of student success. However, establishing unbiased criteria for graduate student success is a challenging endeavor, as previously recognized (Hartnett and Willingham, 1980; Meade and Fetzer, 2009). A number of attempts have been made to establish and evaluate objective (grades, passing qualifying exams, time to doctorate, number of publications) and subjective (professors' ratings of dissertations or predictions of future success) criteria (Stricker and Rock, 1993; Sternberg and Williams, 1997; Burmeister et al., 2014). In this study of the UCSF Tetrad program, past students were classified as "successful" or "underperforming" by a group of 30 current faculty members. Neither the criteria by which students were assigned to these two categories nor the career stage at which the assessment was made were reported. Thus, "successful" students could be those who published multiple first-author papers, were remembered as having worked long hours in the lab, got along well with others in the lab, or went on to academic careers at high-profile institutions. A student identified as "underperforming" might be one who took a long time to graduate, switched labs, was perceived to have a combative personality, or left academia after graduation.Thus, in the UCSF Tetrad study, the relationship between the subjective and undefined dependent variable (student success) and the quantitative independent variables (admissions metrics) cannot be reliably interpreted. For example, it could be—as suggested by the author—that students who did more research before graduate school were more likely to know what getting a graduate degree would entail and were therefore more likely to be considered successful. However, "successful" students might simply be those who met cultural expectations for graduate students—for example, they needed less instruction in bench skills, were more likely to listen to authority, or came from academic families or privileged economic backgrounds. These examples illustrate both how the binary outcome of "successful or underperforming" simplifies the assessment of a complex process and, furthermore, leaves unaddressed the possible transformative or educational value of the graduate program.CONFOUNDING FACTORSThe UCSF Tetrad study is a post hoc analysis that relies on the recollections of 30 current faculty members, an approach that excludes the many faculty members who left over the past 20 years. Relying on individual memories is not only inaccurate (as in the well-documented phenomenon of recall bias [ Pannucci and Wilkins, 2010]), but also limits the scope of the analysis to students who had personalities strong enough to make them memorable to multiple faculty members or who remained professionally connected to faculty after graduation. In fact, faculty-based evaluations can be subject to a host of unconscious biases, as demonstrated recently in a high-profile study of gender bias among science faculty (Moss-Racusin et al., 2012). Any of these biases may have been inadvertently in play here, and eliminating their effect on the results would require a different study design.Students for whom there was no consensus among faculty regarding their assignment as either "successful" or "underperforming" were excluded from the analysis. While several hundred students must have gone through the UCSF Tetrad program during the past 20 years, only 52 were included in the analysis. It seems likely that analyzing only those students on whom there was complete agreement—while failing to provide an external definition of success—would have a flattening effect and allow only strongly conservative or traditional definitions of success (and underperformance) to operate within the study.The author's personal and professional relationships with the students being evaluated, past or present, introduces the possibility of interviewer bias (Pannucci and Wilkins, 2010). Furthermore, Dr. Weiner himself, as well as a number of other current faculty members, could have been included in the student data set under analysis, creating the appearance of a conflict of interest.IMPLICATIONS OF RECOMMENDING TWO YEARS PRIOR RESEARCH EXPERIENCEThe author concludes that of all admissions metrics used, only the number of years of prior research experience and subject GREs were strong discriminators between successful or underperforming students. Based on this finding, he argues that admitting only students with two years of prior research experience would enrich for successful students. However, many young people who are excited about research and who have the ability to contribute have not had the opportunity to do two full years of research before matriculation, because they did not attend a university with a research program, needed to work in a nonresearch position during school and summers, lacked an early awareness of undergraduate research opportunities, or changed course from other pre-professional studies. As a result, taking this report as a prescription for improved student performance might serve to further disenfranchise applicants already disadvantaged due to gender or economic, racial, or family educational background. Thus, improving the availability of opportunities for inquiry-based lab experiences during and after undergraduate education should go hand in hand with any admissions requirement for multiple years of research experience, lest we lose these students to postgraduate education and to the research enterprise as a whole.As the author correctly notes, the results of this particular study may not apply to all institutions, and we hope that each graduate program will clearly define student outcome metrics before moving forward with a similar study. For graduate programs interested in producing leaders—not only in the advancement of academic science but also in the entrepreneurial world, biotechnology, the pharmaceutical industry, science policy, intellectual property law, liberal arts education, scientific communication, or in recruiting the next generation of scientists—students who are willing to take on a risky project, work on a team rather than an independent project, or pursue teaching and outreach activities in addition to their thesis projects may in fact be the best candidates. Whether or not these types of students would be identified as successful in the UCSF Tetrad study is not clear, but this example illustrates that admissions metrics should not be evaluated for their predictive value until the outcome measure of success is clearly defined.TOWARD A NEW AND BROADER DEFINITION OF GRADUATE STUDENT SUCCESSAs suggested above, graduate programs in the biosciences may need to reassess their definition of graduate student success, as training excellent future academic faculty can no longer be the sole guiding principle. The stated mission of UCSF includes "improving health worldwide through innovation," as well as "attracting and supporting the most talented and diverse trainees" (UCSF, 2014 [emphasis added]); the UCSF Tetrad program surely supports this goal. Currently, graduate programs in general are doing a questionable job of preparing students for many of the careers that will be available to them when they graduate. Longtime UCSF Tetrad faculty member and current vice chancellor for research Keith Yamamoto has argued that trainees spend too long trying to get the faculty jobs that account for only 15% of the careers of life sciences PhDs (Yamamoto, 2014). In his view, graduate programs should value a range of careers and train students to pursue them. This impetus to broaden the training focus of graduate programs is shared by funding agencies, judging by the recent announcement of the National Institute of Health's Broadening Experiences in Scientific Training initiative (NIH, 2013) and the National Science Foundation's research traineeships program (NSF, 2013).All of these issues may be considered if we are to develop valid and meaningful measures of graduate school success. In our view, a successful graduate student is one who goes on to contribute relevant scientific expertise in his or her chosen context, whether that be academia, law, industry, publishing, business, teaching, politics, or another career. Contributions could be measured in a longitudinal study in a number of ways, including the self-reporting of graduates, evaluation by outside experts, or, in economic terms, as a return on investment as previously done with teacher development (Silverstein et al., 2009). While "contribution" to a chosen field after graduation is still a subjective metric, and these methods of measuring it may be complex, it has the potential to be less biased, more meaningful, and more inclusive than one solely based on the recollections of faculty members.PRIVACY CONCERNSWe would like to draw attention to one final issue regarding publication of the UCSF Tetrad study. We believe that the article would have benefited from a report of the precautions taken to ensure 1) the future confidentiality of the success rankings generated in the study, 2) the exemption by the UCSF Committee on Human Research office obtained by Dr. Weiner (Weiner and UCSF Committee on Human Research, personal communication), and 3) adherence to the standards of the Family Educational Rights and Privacy Act during the access of student records. While Molecular Biology of the Cell does not specifically instruct authors to describe institutional review board exemption or approval (MBoC, 2012), a growing number of journals do (Atlas, 2003), and the complex nature of the UCSF Tetrad graduate program study highlights the need for scientific journals to promote the rights of human subjects of experiments reported in their pages.SUMMARYWe agree with the author that a quantitative analysis of the predictive nature of the metrics used in graduate student admissions is a worthy pursuit and value the sincere intentions behind the UCSF Tetrad study. However, these types of analyses would benefit from the same rigorous approaches that we employ in our other research endeavors. As UCSF Tetrad graduates with diverse careers in academia, medicine, industry, and publishing, we hope that the definition of success in graduate school can be as thoughtfully and scientifically examined as the measurements used to select the next young people to follow in our footsteps.FOOTNOTESmbc.E14-03-0793The authors are listed in alphabetical order.ACKNOWLEDGMENTSWe thank Ivan Baxter, Brian Dilkes, Miriam B. Goodman, Eric S. Hamilton, Richard H. Haswell, Mary O'Riordan, Kira M. Veley, Keith Yamamoto, and an anonymous reader for enlightening conversations and comments on this Letter.Editor's Note: Molecular Biology of the Cell's written policy requires authors reporting research involving humans to have carried out all of the experiments in accordance with the recommendations from the appropriate National Institutes of Health guidelines and to have had the research protocols approved where necessary by the appropriate institutional committees. Orion Weiner provided proof with his manuscript that the research described was exempt from review by his Institutional Review Board. In retrospect, the journal agrees that it would have been best to state that fact explicitly in the article.REFERENCES Atlas MC (2003). Emerging ethical issues in instructions to authors of high-impact biomedical journals. J Med Libr Assoc 91, 442-449. Medline, Google Scholar Burmeister J, McSpadden E, Rakowski J, Nalichowski A, Yudelev M, Snyder M (2014). Correlation of admissions statistics to graduate student success in medical physics. J Appl Clin Med Phys 15, 4451. Crossref, Medline, Google Scholar Hartnett RT, Willingham WW (1980). The criterion problem: what measure of success in graduate education. Appl Psychol Meas 4, 281-291. Crossref, Google Scholar Meade AW, Fetzer M (2009). Test bias, differential prediction, and a revised approach for determining the suitability of a predictor in a selection context. Organ Res Methods 12, 738-761. Crossref, Google Scholar Molecular Biology of the Cell (2012). Instructions for authors, www.molbiolcell.org/site/misc/ifora.xhtml (accessed 10 March 2014). Google Scholar Moss-Racusin CA, Dovidio JF, Brescoll VL, Graham MJ, Handelsman J (2012). Science faculty's subtle gender biases favor male students. Proc Natl Acad Sci USA 109, 16474-16479. Crossref, Medline, Google Scholar National Institutes of Health (2013). NIH Director's Biomedical Research Workforce Innovation Award: Broadening Experiences in Scientific Training (BEST) (DP7), http://grants.nih.gov/grants/guide/rfa-files/RFA-RM-12-022.html (accessed 25 February 2014). Google Scholar National Science Foundation (2013). FY 2014 Budget Request to Congress, www.nsf.gov/about/budget/fy2014/pdf/01_fy2014.pdf (accessed 10 March 2014). Google Scholar Pannucci CJ, Wilkins EG (2010). Identifying and avoiding bias in research. Plast Reconstr Surg 126, 619-625. Crossref, Medline, Google Scholar Silverstein SC, Dubner J, Miller J, Glied S, Loike JD (2009). Teachers' participation in research programs improves their students' achievement in science. Science 326, 440-442. Crossref, Medline, Google Scholar Sternberg RJ, Williams WM (1997). Does the Graduate Record Examination predict meaningful success in the graduate training of psychologists? A case study. Am Psychol 52, 630-641. Crossref, Medline, Google Scholar Stricker LJ, Rock DA (1993). Examinee Background Characteristics and GRE General Test Performance, Princeton, NJ: Educational Testing Service, (accessed 10 March 2014). Google Scholar University of California–San Francisco (2014). Mission and Vision, www.ucsf.edu/about/mission-vision. Google Scholar Weiner OD (2014). How should we be selecting our graduate students. Mol Biol Cell 25, 429-430. Link, Google Scholar Yamamoto K (2014). Time to rethink graduate and postdoc education. iBiomagazine issue 11, www.ibiology.org/ibiomagazine/issue-11/keith-yamamoto-time-to-rethink-graduate-and-postdoc-education.html (accessed 10 March 2014). Google ScholarFiguresReferencesRelatedDetailsCited byThe GRE over the entire range of scores lacks predictive ability for PhD outcomes in the biomedical sciences21 March 2019 | PLOS ONE, Vol. 14, No. 3The art and science of selecting graduate students in the biomedical sciences: Performance in doctoral study of the foundational sciences3 April 2018 | PLOS ONE, Vol. 13, No. 4The Limitations of the GRE in Predicting Success in Biomedical Graduate School11 January 2017 | PLOS ONE, Vol. 12, No. 1Predictors of Student Productivity in Biomedical Graduate School Applications11 January 2017 | PLOS ONE, Vol. 12, No. 1Response to Bell et al.Orion Weiner13 October 2017 | Molecular Biology of the Cell, Vol. 25, No. 13Related articlesResponse to Bell et al.13 Oct 2017Molecular Biology of the Cell Vol. 25, No. 13 July 01, 20141937-2127 Metrics Downloads & Citations Downloads: 469Citations: 5 History Information© 2014 Bell et al. This article is distributed by The American Society for Cell Biology under license from the author(s). Two months after publication it is available to the public under an Attribution–Noncommercial–Share Alike 3.0 Unported Creative Commons License (http://creativecommons.org/licenses/by-nc-sa/3.0).We thank Ivan Baxter, Brian Dilkes, Miriam B. Goodman, Eric S. Hamilton, Richard H. Haswell, Mary O'Riordan, Kira M. Veley, Keith Yamamoto, and an anonymous reader for enlightening conversations and comments on this Letter.Editor's Note: Molecular Biology of the Cell's written policy requires authors reporting research involving humans to have carried out all of the experiments in accordance with the recommendations from the appropriate National Institutes of Health guidelines and to have had the research protocols approved where necessary by the appropriate institutional committees. Orion Weiner provided proof with his manuscript that the research described was exempt from review by his Institutional Review Board. In retrospect, the journal agrees that it would have been best to state that fact explicitly in the article.PDF download
Referência(s)