Portal de Periódicos da CAPES

RIME Foreword: What Constitutes Science in Educational Research? Applying Rigor in Our Research Approaches

2020; Lippincott Williams & Wilkins; Volume: 95; Issue: 11S Linguagem: Inglês

10.1097/acm.0000000000003636

1938-808X

Yoon Soo Park, Zareen Zaidi, Bridget C. O’Brien,

Clinical Reasoning and Diagnostic Skills

What constitutes science in educational research? This question is not as straightforward to answer given the diversity of disciplines represented in health professions education.1 As researchers, we engage in scientific study to advance our field; we are also an applied field, with goals to produce knowledge that can translate into practice and change. Our field has roots in multiple disciplines, including education, psychology, sociology, biomedical and clinical sciences, as well as in emerging areas, such as data science and linguistic dynamics. As such, the specific elements that define science and principles of scientific inquiry may vary.2 And yet, we can agree that the quality of science generated from a study stems, in part, from the type and relevance of questions asked and how the answers are derived.3,4 The 2019 Research in Medical Education (RIME) Foreword by O’Brien and colleagues5 discussed how to identify questions worth pursuing and factors that make research questions promising for investigation. Promising questions can be investigated through several different research approaches, though some approaches may be more fruitful to pursue based on the type of question. Each approach may yield “scientific” studies according to the standards and accepted practices in the educational community. To help researchers choose the approach that best fits their question, Ringsted and colleagues6 created the research compass to navigate the diversity of pathways toward reaching relevant answers. In the RIME commentary, Sewell and colleagues7 use the research compass to map the studies selected for this year’s RIME supplement. The research approach is what sets science apart from other types of investigation and inquiry such as journalism or product testing or personal reflection. Scientific research approaches involve continuous efforts by the research community to find better explanations and more nuanced understanding of phenomena by asking “what” works as well as “how” or “why” it works by providing better explanations of the phenomenon. One approach rarely suffices to answer all these questions. And yet, combining knowledge obtained through different approaches can make it challenging to evaluate the quality of knowledge generated and to determine whether evidence is both credible and improves our current explanations and understandings of phenomena. How, then, do we know if research is advancing the field and bringing our science to higher state? In this Foreword to the RIME supplement, we—the past, present, and future RIME chairs—take up this question and offer our perspectives on essential components to achieving science in health professions education research. As researchers, we believe that part of the catalyst to achieving best science in health professions education includes scientific rigor in the research approach. We believe this involves attention to standards of rigor appropriate to familiar approaches in our field as well as adoption of cutting-edge and boundary-stretching approaches that may require new standards of rigor. Research Approaches: Aligning Questions With Research Design, Research Methodology, and Philosophy of Science How we engage in research is described as the research approach, and a research question is the starting point for the research approach. Research approaches articulate the plans and decisions that outline the process from study formation to methodologies involved in data collection, analysis, and interpretation.8–10 Similar to research questions in health professions education that can be characterized as description, justification, and clarification studies,11 research approaches can have multiple layers that inform and link the subsequent research process following the identification of a research question. Researchers often characterize an approach as being quantitative, qualitative, or mixed-methods in nature. As the state of the science advances in HPE, these classifications, based on the type of data, are insufficient given the array of disciplinary traditions and practices available in our field.12 Research approaches require decisions based on research questions, assumptions, and conceptual frameworks guiding the study.9 More specifically in scientific studies, there are 3 principles of research approaches that influence the decision-making process: (1) philosophical assumptions (research paradigm or philosophies of science), (2) research designs (processes of inquiry), and (3) research methods (data collection, analysis, and interpretation). Figure 1 illustrates this relationship of the 3 research approaches, operationalizing the framework within research questions and answers. Research approaches inform the direction and scope examined by the researcher.Figure 1: A framework for scientific research: Aligning questions and answers with research approaches. A researcher conducting a study begins by identifying a research question, selecting research approaches that align with the question, and using the results to form answers. Within this process, the researcher identifies three research approaches (philosophy, design, and methods), applying standards of rigor. The application of scientific rigor in the three research approaches support making a scientific study. The bidirectional arrows for question and answer reflect how they build on each other.Philosophies of science2,13 make assumptions about how the world operates, guiding the way in which science is conducted based on ontology (how reality is viewed), epistemology (how the nature of knowledge is conceived), axiology (role and place of values in the research process specifically the influence of values on the relationship between epistemology, methodology, and methods), methodology (how the paradigm defines and processes science), and rigor (criteria to justify the quality of research). The philosophies listed in the figure are a subset of the wide range of paradigms available for health professions education researchers, including positivism, postpositivism, critical theory, constructionism, pragmatism, realism, and postmodernism.14 Research designs (strategies of inquiry) provide guidance on the specific direction and process for the study.15 Health professions education researchers have a wide selection of research designs available, ranging from quantitatively oriented designs such as experimental designs (including randomized controlled studies or quasi-experimental studies) and surveys, to qualitatively oriented designs such as narrative research, phenomenology, grounded theory, ethnography, case studies, and mixed-methods designs that include convergent, explanatory/exploratory sequential, and transformative studies. The research methodology provides the framework for how data are collected, analyses performed, and interpretation conducted to synthesize findings and arrive at answering the research question.9,15 Research methodologies may also have a quantitative, qualitative, or mixed-methods orientation, depending on the specific method for identifying the tools to collect data, compile the different sources of data, perform analysis, and conduct interpretation. For example, this may involve using psychometric scales to gather information that is scored and analyzed using statistical methods such as correlations or regression analysis for interpretation; open-ended questions may be used to gather data through interviews, with analysis of transcripts to identify themes and patterns as part of the interpretive process. Alternatively, a combination of these methodologies can be used, drawing inference from numeric and text-based data. It becomes somewhat clear from the distinction of these 3 core components of research approaches that there is no linear pathway for conducting research in health professions education. A researcher with a positivist or postpositivist orientation may begin with an a priori testable hypothesis (philosophy of science) and choose to conduct an experimental study (research design) using data collected from a survey and knowledge test (research methodology). A researcher working from a constructivist orientation (philosophy of science) may choose to conduct an ethnographic study (research design) that involves collecting and analyzing interview and observational data captured in field notes (research methodology).15,16 Moreover, studies in medical education may involve multiple research approaches, prompting scientific research with “methodological pluralism.”17 Advancing Research Approach Using Scientific Rigor in Health Professions Education Our field draws on many approaches—some of which come from well-established disciplines and research traditions, others that span multiple disciplines and blend traditions to create new approaches. So what makes a research study scientific? If we consider the 3 components of research approach, then there must be alignment among the 3 components, careful consideration of standards of rigor that apply to the approach, and evidence that such standards have been met throughout the research process.10 Standards for rigor vary depending on the components of the research approach. For example, in an experimental design, the evidence of rigor is in the degree to which the researcher demonstrates objectivity and minimizes subjective bias that can influence the study, providing evidence on how threats to hypothesis testing is minimized and thereby achieving the association or causal relationship between the factors studied. There is also an aim toward study replication and generalizing findings through large sample sizes and inferential statistics. In contrast, in a study employing constructivist grounded theory, the criteria for rigor may include concepts of reflexivity as evidence of how the research team considered their own assumptions and perspectives throughout the research process and constant comparative techniques to explain how the researchers purposefully selected study participants and used an iterative process to identify themes and relationships among themes.18,19 Conceptual or theoretical generalizability is a goal rather than statistical generalizability and replication. For research design, standards for rigor should be applied and demonstrated. If a survey was administered, principles of rigor such as pilot testing the items or applying best-practice guidelines to survey construction should be reported. In an ethnographic study, for example, evidence of prolonged engagement triangulation of data sources and consideration of participant reactivity can demonstrate rigor. Likewise, for research methodology, there are consensus guidelines on standards of rigor in data collection such as efforts to minimize participant attrition, use of measurement tools with supporting validity evidence, or evaluating the statistical model fit and standard error of estimates in a regression model. Standards for each of these research approaches are reported in the literature and provide guidance to researchers.20,21 However, there are also research approaches that lack principles of rigor and guidance for newer and emerging approaches, particularly in mixed-methods domains that require further attention from the field on what constitutes rigor and evidence of science. Depending on context, there may be a need to balance the pendulum between rigor and relevance.22 For research to be relevant, it must focus on the fundamental concerns of the end users. Yet the research approach supporting relevance may require additional discussion, further consensus, and sensitivity to cultural contexts and norms.23,24 Research Approaches in RIME 2020: Emerging and Frontier Research Approaches for Health Professions Education The focus of this year’s RIME plenary is on newer research approaches in HPE, and the commentaries from each of the speakers appear in this supplement.25–27 Such approaches offer exciting possibilities to advance science in our field and also require that we clarify the components of these approaches and identify standards of rigor that correspond to such approaches.28 Recent advances in technology and ability to gather large quantities of data (quantitative, qualitative, and visual images) have prompted numerous opportunities to expand our horizon on the available methodologies.29 What research approaches can be leveraged with the emergence of data science and improvements in computing power? This was the focus of the 2020 RIME plenary, with panelists discussing emerging and frontier research approaches covering the domains of artificial intelligence,25 learner analytics,26 and practice-based research networks.27 Significant advances to synthesizing large-scale data have shown potential for their use to train health professionals using deep learning and artificial intelligence methodologies30; real-time analytic data may be provided to learners and faculty, allowing improved systems of training for longitudinal and developmental progress31; and the ability to gather large quantities of data that cover multiple levels of data across assessments, learners, programs, and institutions requires collaborative infrastructure and data sharing, as guided by the science of practice-based research networks.32 The commentaries present guidance on research approaches and associated rigor. At the same time, standards for rigor in these approaches may need further development and consensus in the health professions education. We hope that these commentaries prompt discussions to advance the scientific rigor in our studies, including frontier and emerging methodologies. Contributions in Research Approach From Health Professions Education We often regard many of our core research approaches and conceptual frameworks as being transferred or adopted from traditional disciplines such as psychology, sociology, and statistics. While many principles of social science research have contributed to our profile of core research approaches, we have modified and also developed new approaches.33 For example, many research approaches used to study assessments have originated from K-12 educational settings including psychometric approaches. Over the past decades, the field has modified these ideas in how we implement and operationalize research for performance-based assessments—these methods are now influencing disciplines outside health professions education, including how we analyze objective structured clinical examinations and workplace-based assessments, to name a few.34 In addition, the standardized patient methodology is a unique research approach adopted in health professions education that is now revolutionizing the training of professionals in other fields. Standards for rigor in medical education are also impacting research standards in dental education and in other health education fields, including domains of engineering education and legal education. We have gradually shifted into a field that now exports innovative research methodology and approaches to other disciplines. We are seeing newer approaches that adopt a mixture of methodologies, as exemplified in generic qualitative approaches that refuse to follow a single established methodology.35,36 The impact of such newer approaches requires thoughtful consideration for rigor among health professions education researchers. Furthermore, there are research approaches and standards of rigor that still await development for the field, including methodologies that advance issues in equity, diversity, and inclusion.37 Evaluation studies examining pipeline programs that examine workforce composition and diversity still lack guidance in research approach and rigor, particularly among research questions that align with these important topics. Additionally, praxeology of education needs to be given specific consideration.38 Praxeology stems from “praxis”—action, which has to do with the question of how education is to be practiced. It specifically answers the question of what research means for making education work. Praxeology is not value-neutral and depends on the priorities of institutions and national bodies. In the midst of the BLM (Black Lives Matter) movement, we challenge our readers to reflect on how medical education research has impacted minorities. Has the research resulted in praxis, change, and action? It is important to note that research in the field of diversity, equity, and inclusion need to be met with institutional action to implement changes in education. Contributions in these areas can have significant impact for our field as well as other related disciplines in how we advance science.39 Concluding Remarks In a field with scholars from many disciplines, it is not easy to articulate a single standard of rigor that encompasses all principles and values of best science—there still are many contending standards of rigor. We have made significant progress in the types of questions we ask and also in the reporting standards of our work. And perhaps the scientific rigor associated with our research approach still remains to be a domain worth continuously refining. Yet, the take-home message is for researchers to demonstrate and to strive toward adherence in standards of rigor across their selection of research design, research methodology, and assumption and choices in the philosophy of science. Clear articulation of how a research study advances knowledge relevant to a problem or phenomenon in our field, the relevance and applicability of the study in different cultural contexts, what needs to change in education as a result of the finings (praxeology), and how the research speaks to core principles and values of our field (axiology) all contribute to the process of scientific rigor. The year 2020 marks an unusual year, full of uncertainties and challenges in how we function as educational researchers and practitioners and more importantly as professionals; we are at a crucial time to pause and reflect what character and value we bring to the profession and communities we serve. This year also marks our 59th RIME program, where we are at a cusp entering the next milestone for RIME, as we look toward a new decade of excellence in research and discoveries that drive the science of health professions education. In this special time in our 59-year history, we should take a moment to look back to celebrate our achievements, but perhaps more importantly, continuously build and articulate how we as a field can continuously improve the training of health professionals representing our core values through rigorous scientific research and evidence.