Portal de Periódicos da CAPES

A Qualitative Investigation of Panelists' Experiences of Standard Setting Using Two Variations of the Bookmark Method

2009; Taylor & Francis; Volume: 22; Issue: 3 Linguagem: Inglês

10.1080/08957340902983978

1532-4818

Serge F. Hein, Gary Skaggs,

Meta-analysis and systematic reviews

Abstract Only a small number of qualitative studies have investigated panelists' experiences during standard-setting activities or the thought processes associated with panelists' actions. This qualitative study involved an examination of the experiences of 11 panelists who participated in a prior, one-day standard-setting meeting in which either the traditional bookmark method or the single-passage bookmark method was used. Data were collected using in-depth focus group interviews. A whole text analysis resulted in three major categories. Many participants found the fundamental task of choosing a specific item as a bookmark to be inherently difficult. Some also deviated from the prescribed bookmark procedure and relied on alternative strategies that were highly resistant to change. Participants' views of item difficulty and correct item order also directly impacted the bookmarking process. Implications of the findings for standard-setting procedures and for further research are also addressed. ACKNOWLEDGMENTS The authors gratefully acknowledge the support received from an ASPIRES grant from Virginia Polytechnic Institute and State University. We also thank Dr. David Mott, of Tests for Higher Standards, and the school district coordinator for their assistance in this research. Notes 1It is important to add that after the break, several participants in the traditional bookmark group asked the facilitator questions about the bookmark procedure. Answering these questions prompted him to repeat the initial instructions about bookmarking. Most of the questions involved concerns about the two-thirds response probability (RP) criterion and, in responding to them, a portion of the instructions was placed on a flip chart (“a classroom of barely proficient students” and “2/3 of these barely proficient students”). After this information was presented, the participants were given an opportunity to change their bookmark if they wished but no group discussion was allowed. The instructions given to the single-passage bookmark group were the same as those presented to the traditional bookmark group, except that the information that had previously been presented on the flip chart was now incorporated (visually) into the original instructions. 2It should be noted here that the topic of the two-thirds RP criterion was addressed in both of the focus group interviews and only one of the participants indicated that she continued to be confused about this concept after the panel meeting facilitator had repeated the initial instructions about bookmarking. The remaining participants' responses indicated that using the two-thirds RP criterion did not pose any difficulties and these responses also revealed an awareness of the criterion's role in producing their judgments.