Portal de Periódicos da CAPES

Testing children's information retrieval systems: Challenges in a new era

2010; Wiley; Volume: 47; Issue: 1 Linguagem: Inglês

10.1002/meet.14504701222

0044-7870

Jamshid Beheshti, Dania Bilal, Allison Druin, Andrew Large,

Child Development and Digital Technology

Proceedings of the American Society for Information Science and TechnologyVolume 47, Issue 1 p. 1-4 ArticleFree Access Testing children's information retrieval systems: Challenges in a new era Jamshid Beheshti, Jamshid Beheshti jamshid.beheshti@mcgill.ca School of Information Studies, McGill University, Montreal, CanadaSearch for more papers by this authorDania Bilal, Dania Bilal dania@utk.edu School of Information Studies, University of Tennessee, Knoxville, USASearch for more papers by this authorAllison Druin, Allison Druin allisond@umiacs.umd.edu College of Information Studies, University of Maryland, Maryland, USASearch for more papers by this authorAndrew Large, Andrew Large andrew.large@mcgill.ca School of Information Studies, McGill University, Montreal, CanadaSearch for more papers by this author Jamshid Beheshti, Jamshid Beheshti jamshid.beheshti@mcgill.ca School of Information Studies, McGill University, Montreal, CanadaSearch for more papers by this authorDania Bilal, Dania Bilal dania@utk.edu School of Information Studies, University of Tennessee, Knoxville, USASearch for more papers by this authorAllison Druin, Allison Druin allisond@umiacs.umd.edu College of Information Studies, University of Maryland, Maryland, USASearch for more papers by this authorAndrew Large, Andrew Large andrew.large@mcgill.ca School of Information Studies, McGill University, Montreal, CanadaSearch for more papers by this author First published: 03 February 2011 https://doi.org/10.1002/meet.14504701222Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat PURPOSE The purpose of this Panel is to discuss and elaborate on testing and evaluation challenges facing the designers of new and novel information systems for children. Since 2002, ASIST has hosted 16 papers, panels, and posters on information retrieval systems, digital libraries, and user interfaces for children11 Result of a search on 'children' in the Proceedings of the American Society for Information Science & Technology. , but very little discussion has been devoted to methodological challenges. Many new systems are designed for today's children, the "Digital Natives" (those born after 1989), who may process information differently from their predecessors. Digital natives often are well versed in using computer games, such as those designed for the X-Box 360 and the Nintendo Wii. However, a panel of experts, meeting recently to discuss the information behavior and needs of the new generation of users, concluded that a research agenda is urgently required to investigate the "characteristics and preferences of this tech savvy group that surprisingly lacks basic skills in information evaluation and retrieval" (Radford et al, 2008, p. 2). Developing new and novel information retrieval mechanism for these digital natives, however, require new ways of testing and evaluation of the system. BACKGROUND Research has shown that the overwhelming majority of information retrieval systems (IRS) such as Google are unsuitable for children. Many of the problems typically encountered by children in information retrieval includes determining appropriate concepts, translating these concepts to keywords that the information retrieval system can understand, finding appropriate synonyms, and distinguishing homonyms (Bilal, 2000; Druin et al, 2009; Large, Nesset, Beheshti, 2008). Traditionally, much of the research on IRS has been experimental, where researchers strive to control as many variables as possible in a carefully regulated environment to minimize interference from external factors. Generally, research in the field has followed the Cranfield 2 model (Cleverdon, 1967), which to date remains the main approach for testing, and lead most of the research efforts in Text REtreival Conference (TREC), and Cross-Language Evaluation Forum (CLEF) (Voorhees, 2002). Given the difficulties experienced by children in interacting with conventional IRS, browsing may be an alternative method for searching for children requiring less cognitive load than other conventional techniques such as keyword searching. Children, in particular are more adapt to browsing, and may use this technique frequently to find information for both assigned and self-imposed tasks. Various mechanisms have been proposed to facilitate children's browsing, ranging from alphabetical searching (Large et al, 2006) to conventional and visual subject taxonomies (Druin, 2005; Large et al, 2009), and virtual reality interfaces (Beheshti, Large, Julien, 2005). However, testing and evaluating these systems pose both theoretical and practical challenges for researchers. CHALLENGES Researchers engaged in designing novel IRS for children face new methodological challenges, beyond the traditional Cranfield 2 model: System evaluation: How is a system that is designed primarily for browsing rather than keyword searching tested? Are metrics such as recall and precision appropriate measures for a children IRS, which is designed for browsing? Is the testing framework proposed for interactive information retrieval systems (IIR) (Borlund, 2003) appropriate for a children's system? User-system evaluation Children do not form a homogenous group of users. Many factors such as Socio-economic and cultural differences (Bilal, Sarangthem, Bachir, 2008), and their age (Large, 2005) may affect their interaction with an IRS. Numerous information seeking models have been proposed, including Wilson (1999), Bates (1989), Belkin (1980), Cole (1999), Ellis (1989), Marchionini (1995), and Savolainen (1995), the core of which deals with: information need, searcher's characteristics, the environment, and the interaction of these three entities (Knight & Spink, 2008). What role do these entities play in evaluation of a children IRS, and what are the metrics to measure them? Kuhlthau (1991, 2004), through her observations of school students, developed a six-stage Information Searching Process (ISP) model, designed from the user's perspective under an imposed task (for example a classroom project). She incorporated the concept of time and showed that students' conceptions of information change as they progress through the information problem. She emphasized the affective factor (feelings), and investigated its role on users as they moved through various stages of the ISP, including an uncertainty stage, which she defined as "a cognitive state that commonly causes affective symptoms of anxiety and lack of confidence." Uncertainty, however, is "natural and essential for constructing personal knowledge" in the ISP, which may require intervention that allows students to advance their knowledge. Can a system be constructed to intervene on a need basis during an ISP? And if so, how would such a system be tested and what metrics would be used for evaluation? The panelists will draw on their extensive experience to discuss these and other challenges in testing and evaluating children's information retrieval systems. PANELISTS Each panelist will lead the discussion with a 10 minute presentation, outlining a research topic and the pertinent methodological challenges. The panelist will then invite the audience to discuss for approximately 10 minutes potential resolutions to these challenges. Following the four presentations and the ensuing discussions, the moderator will summarize the deliberations in the last 10 minutes of the session. 1 Andrew Large: Experimental versus operational testing of a children's portal Andrew Large contends that in designing and developing a children's retrieval system both experimental and operational testing pose challenges to researchers, but that each offers a unique insight into children's reactions to information technologies. Large and Beheshti have developed a children's web portal, History Trek (www.historytrek.ca), which has attracted more than 800,000 hits since its public début in September 2007. The portal was created by an intergenerational team comprising adult designers and young students and provides access to more than 2,300 web pages in English and French on Canadian history. The portal has been tested in experimental, and in actual operational (classroom) environments. Bio: Andrew Large is an Associate Dean, Research and Graduate Students, at the Faculty of Education at McGill University. He is a member of the School of Information Studies, where he is the CN-Pratt-Grinstad Professor of Information Studies. He has conducted research and produced numerous articles, books, and book chapters on information retrieval, and children information seeking behavior in many prestigious international publications. 2. Dania Bilal: Testing and evaluating children's information retrieval systems using holistic frameworks. Dania Bilal argues that working with children on research is not only rewarding, but also "unique" and requires "specific" abilities and skills on the part of the researcher, and well as use of "appropriate," and "innovative" research methodologies to reap significant results and benefits. She challenges system designers and will comment on their insufficient efforts in addressing children's behavior, cognition, and affect in the design of interfaces. Bilal has been studying children's holistic information behavior since 1997. She has published several studies including children's interaction with and design of web search engines (e.g., Bilal, 2000, the International Children's Digital Library (e.g., Bilal & Bachir, 2007); and on children's cognitive structures of taxonomies in web directories. She has focused efforts on both children's information behavior in unicultural and multicultural environments. She has employed empirical (quantitative) and qualitative research methods to investigate and test the usability of information retrieval systems from the children's perspectives. She has recently developed models of children's interaction with the International Children' Digital Library (ICDL), the results of which have led to new proposed models for children's information behavior consisting of seven modes of activities (e.g., Bilal, Sarangthem, & Bachir, 2008). Bio: Dania Bilal is a Professor of information sciences at the School of Information Sciences (SIS) at the University of Tennessee, Knoxville. She published research on the cognitive and affective information behavior of children's interaction with web search engines, developed measures for assessing the usability of cross-cultural international web interfaces, studied the design of web search engines' directory structures against children's cognitive structures of taxonomies, and modeled children's information behavior in digital environments. Her recent research includes information behavior of children with cognitive disabilities; and query and browsing analysis of children's web interaction. 3 Allison Druin: Testing the children's information retrieval system on mobile devices Allison Druin maintains that both quantitative and qualitative methodologies are needed to test new systems, such as the ICDL, in various environments. She argues that children, regardless of their diverse cultural backgrounds and their context, use visual tools to search and browse for information, and that systems should be age-sensitive. Druin and her intergenerational and interdisciplinary team at the University of Maryland have developed the International Children's Digital Library (ICDL) (http://en.childrenslibrary.org/), which provides children with several search options, including a 'flatten' hierarchical subject category for easy navigation. ICDL contains more than 4,000 children's books, used monthly by some 100,000 users from 150 countries. Druin and her team have built two ICDL applications for the iPhone and one for the iPad. Bio: Allison Druin is an Associate Professor at the College of Information Studies at University of Maryland, where she is the Director of the Human-Computer Interaction Lab at the Institute for Advanced Computer Studies, and an Affiliate Faculty member at the College of Education. She has received major grants to design and develop the International Children's Digital Library, and conduct research on its use across the world. She has published many articles on the ICDL, as well as on mobile technologies for children. 4. Jamshid Beheshti: Testing a virtual reality children's information retrieval system Jamshid Beheshti claims that when constructing a novel system, such as a virtual reality library, children's participation at the design stage is far more limited than when developing more conventional systems. Testing such a system is equally unconventional. The system should be evaluated by how much information it can transfer to the child regardless of the immediate utility of that information. Beheshti and Large have designed a virtual reality library (VRLibrary), the objective of which is to assist children and young adults in browsing information available on the web for educational projects (Beheshti, Large, Julien, 2005). VRLibrary was constructed using the metaphor of a physical library with rooms, bookcases and books. The system was tested against a conventional portal under experimental conditions, and the data are currently being analyzed. Bio: Jamshid Beheshti is an Associate Professor at the School of Information Studies at McGill University, where he was appointed as the Director of the School for six years, the Associate Dean of the Faculty of Education for four year, and the Interim Dean of the Faculty. He is the principal investigator on a Social Sciences and Humanities Research Council of Canada grant on Virtual Environments as an Intervention Agent in the Information-Seeking Process of Elementary School Students. In collaboration with his colleagues, he has obtained many research grants, has published widely, and has presented his research at national and international academic and professional conferences. SIGs List of all SIGs who might find this panel of interest: Digital Libraries (DL) Human-Computer Interaction (HCI) Information Needs, Seeking and Use (USE) REFERENCES Bates, M. J. The Design of browsing and berrypicking techniques for the online search interface. Online Review, 13 (5): 407– 424. Google Scholar Beheshti, J., Large, A. & Julien, C.A. (2005). Designing a virtual reality interface for children's web portals. Data, Information, and Knowledge in a Networked World. Canadian Association for Information Science 2005 Annual Conference, The University of Western Ontario, London, Ontario, June 2–4, 2005. Google Scholar Belkin, N. J. (1980). Anomalous states of knowledge as a basis for information retrieval. Canadian Journal of Information Science, 5, 133– 143. Web of Science®Google Scholar Bilal, D. (2000). Children's use of the Yahooligans! Web search engine: 1. Cognitive, physical and affective behaviors on fact-based search tasks. Journal of the American Society for Information Science, 51 (7), 646– 665. Wiley Online LibraryWeb of Science®Google Scholar Bilal, Dania, & Bachir, Imad. 2007. Children's interaction with international and multilingual digital libraries. I. Understanding interface system design representations. Information Processing & Management, 43 (1): 47– 64. CrossrefWeb of Science®Google Scholar Bilal, D., Sarangthem, S., & Bachir, I. (2008). Toward a model of children's information seeking behavior in using digital libraries. Proceedings of the Second International Symposium on Information Interaction in Context (IIiX), ACM International Conference Proceeding Series; Vol. 348, October 14–17, London, UK (pp. 145– 151). Google Scholar Brlund, P. (2003). The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research, 8 (3), paper no. 152 [Available at: http://informationr.net/ir/8–3/paper152.html] Google Scholar Cleverdon, C. W. (1967). The Cranfield tests on index language devices. In Aslib Proceedings, volume 19, pages 173–192, 1967. (Reprinted in Readings in InformationRetrieval, K. Sparck-Jones and P. Willett, editors, Morgan Kaufmann, 1997). Google Scholar Cole, C. (1999). Activity of understanding a problem during interaction with an "enabling" information retrieval system: Modeling information flow. Journal of the American Society for Information Science, 50 (6): 544– 552. Wiley Online LibraryWeb of Science®Google Scholar Druin, A. (2005). What children can teach us: developing digital libraries for children with children. Library Quarterly, 75 (1): 20– 41. CrossrefWeb of Science®Google Scholar Druin, A. et al. (2009). How children search the internet with keyword interfaces. IDC 2009, June 3–5, Como, Italy. ACM. Google Scholar Ellis, D. (1989). A behavioural model for information retrieval system design. Journal of Information Science, 15 (4–5), 237. CrossrefWeb of Science®Google Scholar Knight, S. A. & Spink, A. (2008). Toward a Web search information behavior model. In: A. Spink, & M. Zimmer (Eds.) Web Search, Springer Series in Information Science and Knowledge Management 14. Springer-Verlag, 209– 234. Web of Science®Google Scholar Kuhlthau, C. C. (1991). Inside the search process: information seeking from the users' perspective. Journal of the American Society for Information Science, 42 (5): 361– 371. Wiley Online LibraryWeb of Science®Google Scholar Kuhlthau, C. (2004). Seeking meaning: A process approach to library and information processes. Westport, Conn.: Libraries Unlimited. Google Scholar Large, A. (2005). Children, teenagers, and the Web. Annual Review of Information Science and Technology, 2005, 39: 347– 390. Wiley Online LibraryWeb of Science®Google Scholar Large, A., Beheshti, J., Nesset, V. & Bowler, L. (2006). Web portal design guidelines as identified by children through the processes of design and evaluation. Proceedings of the American Society for Information Science and Technology Annual Meeting, Austin, Texas, November 2006. Google Scholar Large A., Nesset, V., Beheshti, J. (2008). Children as information seekers: what researchers tell us. New Review of Children's Literature and Librarianship, 14 (2): 121– 140. CrossrefGoogle Scholar Large, A., Beheshti, J., Clement, I., Tabatabaei, N., Tam, M. (2009a). Visualizing a hierarchical taxonomy in a children's web portal: User evaluations of a prototype. Canadian Journal of Information and Library Science, 33 (3/4), 255– 282. Web of Science®Google Scholar Marchionini, G. (1995). Information seeking in electronic environments. New York, NY: Cambridge University Press. CrossrefGoogle Scholar Radford, M. L., Silipigni Connaway, L., Agosto, D.E., Cooper, L. Z., Reuter, K., Zhou, N. (2008) Behaviors and preferences of digital natives: information a research agenda. Proceedings of the American Society for Information Science and Technology, 44 (1): 1– 15. Google Scholar Savolainen, R. (1995) Everyday life information seeking: Approaching information seeking in the context of way of life. Library & Information Science Research, 17 (3): 259– 294. CrossrefWeb of Science®Google Scholar Voorhees, E. M. (2002). The Philosophy of information retrieval evaluation. Evaluation of Cross-Language Information Retrieval Systems. Lecture Notes in Computer Science, 2406. 355– 370. CrossrefWeb of Science®Google Scholar Wilson, T. D. (1999). Models in information behaviour research. Journal of Documentation, 55 (3), 249– 270. CrossrefWeb of Science®Google Scholar 1 Result of a search on 'children' in the Proceedings of the American Society for Information Science & Technology. Citing Literature Volume47, Issue1November/December 2010Pages 1-4 ReferencesRelatedInformation