Portal de Periódicos da CAPES

PROTOCOL: Interventions for improving learning outcomes and access to education in low‐ and middle‐income countries: a systematic review

2017; The Campbell Collaboration; Volume: 13; Issue: 1 Linguagem: Inglês

10.1002/cl2.176

1891-1803

Birte Snilstveit, Emma Gallagher, Daniel R. Phillips, Martina Vojtkova, John Eyers, Dafni Skaldiou, Jennifer Stevenson, Ami Bhavsar, Philip R. Davies,

Child Nutrition and Water Access

Education is considered essential for sustainable development and is a fundamental human right, as stated by article 26 of the Universal Declaration of Human Rights (UN, 2013a). There is widespread consensus on the importance of education for human well-being (Glewwe and Kremer, 2005). For instance, Sen (1999: 296) argues that education has a "direct relevance to the well-being and freedom of people" as well as an "indirect role through influencing social change" and "economic production". In addition to the intrinsic value of education in and of itself, research suggests positive relationships between education and economic growth and earnings (Barro, 1991; Duflo, 2000; Psacharopoulos & Patrinos, 2004), and this relationship becomes more pronounced in poorer countries (Psacharopoulos, 1985; Mankiw et al., 1992). Moreover, various studies have provided evidence of a link between better education systems and other indicators of human development, including health status, maternal and infant mortality, lower population growth and lowered crime (Glewwe, 2013; Hillman & Jenkner, 2004; Hannum & Buchmann, 2003). In other words, individuals with high levels of education are more likely to be employed, generate higher income, overcome economic shocks and maintain healthier families (World Bank, 2011). Substantial efforts have been made in recent years to improve access to education in in low- and middle-income countries (L&MICs). While there has been significant progress, this has been uneven and challenges remain. For instance, the net enrolment rate for children of primary school increased from 82 to 90 per cent between 1999 and 2010 (UN, 2013b). However, improvements in enrolment rates slowed down considerably after 2004 and in 2010, 61 million children of primary school age were still out of school, more than half of them (33 million) in Sub-Saharan Africa (UNESCO, 2012). While there has been progress in reducing the number of girls excluded from education, from 58 per cent in 1999 to 53 percent in 2010 (UN, 2012), girls are still more likely than boys to miss out on schooling. Girls' participation rates remain lower than those of boys in 53 developing countries, with disparities particularly pronounced in West Asia and Sub-Saharan Africa (UN, 2012). The increase in primary education has increased the demand for secondary education and enrolment in secondary school has risen by almost 10 percentage points during the last ten years (World Bank, 2013). This demand also presents a challenge for many countries however, and 71 million children of lower secondary age (12-15 years) are out of school worldwide (UN, 2012), with three of four out-of-school adolescents residing in either Sub-Saharan Africa or South and West Asia (UNESCO, 2012). Adolescents from disadvantaged backgrounds are more likely to miss out on education, with those from poor and rural households being more likely to be excluded, and girls being more likely to not attend lower secondary school than boys (UN, 2012). Over the last decades, much attention has been focused on addressing issues related to access to education, but more recently attention has shifted towards improving the quality of education. While there has been significant progress towards achieving education for all, in many countries the promise of schooling has failed to translate into learning (Prichett, 2013). Children will not receive a better education just by virtue of being in school if the conditions that enable learning are not also present (Petrosino et al., 2012; Pritchett, 2013). As Glewwe (2013:3) argues, 'enrolment is not the final goal of education policy. The ultimate goal is to prepare children for a better life when they are adults.' Studies measuring learning outcomes among school children across low- and middle-income countries find consistently low levels of learning, with hundreds of millions of children leaving school without basic numeracy and literacy skills (Prichett, 2013; Robinson, 2011; UNESCO, 2012). According to the Education for All Global Monitoring report (UNESCO, 2013) around 250 million children in L&MICs cannot read, write, or do basic maths problems. This number includes over 130 million children who are actually enrolled in primary school and yet have not acquired these basic skills, leading some commentators to suggest there is a global learning crisis (Robinson, 2011). For example, the 2010 annual assessment of the basic reading and arithmetic skills of over 600,000 children in India, conducted by ASER, found that in many states only 53 per cent of children in Grade 5 were able to read a Grade 2 level text (ASER, 2011). Similarly, the Uwezo learning-assessment, a survey aiming to determine the level of literacy and numeracy across Kenya, Tanzania and Uganda, also highlight mayor gaps in children's learning (Uwezo, 2013). The most recent report found that less than one third of Grade 3 children have basic Grade 2 level literacy and numeracy skills, and that two in ten children still have not achieved this level by Grade 7. Assessments conducted by the Southern & Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ) found that around 27 per cent of students who took part in the assessments were functionally illiterate, meaning that they were unable to read short simple texts, nor extract meaning from them (Spaull, 2011). As a response to the importance of education to international development, the proposed systematic review will identify, appraise and analyse the findings of all impact evaluations on educational interventions in low- and middle-income countries (L&MICs) over the past two decades. A range of different interventions are currently being implemented to address the challenges associated with ensuring all children in L&MICs have access to schooling, and that they gain sufficient skills and knowledge to realise the benefits a good education can bring. To structure our review of this broad range of interventions and to specify the types of intervention we will include, we will use a framework for classifying interventions based on institutional setting. This follows a similar approach to the one adopted by Sherman et al. (2002) in their review of the evidence on a range of different crime prevention interventions. The framework, together with a provisional list and description of interventions falling under each category, is outlined below. As depicted in Figure 1, a complex set of determinants influence access to schooling, quality of education and learning, including the individual or school level, the wider socio-cultural context, and economic factors. The household context (presence of parents, work commitments of parents and children, number of children, education and health of household members, language used at home, distance children must travel to school) can have an important impact on the likelihood of children enrolling, attending and learning. Households must also have the ability to deal with direct costs such as school fees, indirect costs such as school uniforms, travel and the opportunity cost of sending a child to school, and be able to cope with income shocks. These factors will also be linked to wider contextual factors, including socio-cultural attitudes toward education in general, gender and so on, the local economic conditions and the safety and security of school attendance. School infrastructure, materials, teaching resources and pedagogical approaches are all fundamental in determining the quality of, and access to, schooling. The educational system, governance and leadership will all determine the institutional setting in which schooling takes place, and these will themselves be a function of wider national policy, budget and expectations. The interventions included in this review aim to influence one or more of these determinants. They draw on a range of theoretical underpinnings and aim to achieve their objectives by means of a series of contrasting pathways and mechanisms. One of our review objectives is to describe and explore how interventions work, and thus we will not provide detailed discussions of programme theories for all interventions in this protocol. Rather, we outline an overall framework for the logic underlying broader intervention categories and the main pathways through which these interventions aim to improve education outcomes. Figure 2 provides an overview of interventions classified according to the settings in which they take place, outlining the main pathways through which they may improve education outcomes. The interventions all work through a series of causal pathways either designed to make attendance/enrolment more desirable or accessible, or to facilitate learning by improving the teaching and learning environment, or by improving student health and therefore directly boosting their ability to learn. For example, a child-centred intervention such as school feeding provides an incentive for parents to send their children to school, but also aims to provide students with the nutrition they need to learn. Better nourished children are less likely to miss school due to illness, and better attendance can have the knock-on effect of improving learning outcomes (Kristjansson et al., 2009). Providing information, either to children or parents, is intended to emphasise the long-term benefits that schooling provides and, in so doing, increase demand for both enrolment and attendance (Krishnaratne & White, 2013). School-based management interventions are intended to improve the efficiency of school administration and leadership by facilitating innovation and allowing parent power to drive up the quality of schooling (Banerjee et al., 2008). Increasing quality of schooling and improved learning outcomes may also have an important impact on enrolment and attendance, and vice-versa. Improving the learning environment should have a direct impact on learning outcomes but may also have the indirect effect of pushing up demand by increasing the perceived benefits of schooling. Greater enrolment and attendance may change the student-teacher ratio, or lead to greater competition for limited resources or the inclusion of more children with a lower educational baseline. However, higher enrolment and attendance is likely to increase the absolute number of students completing school or passing exams and may even improve learning and completion through increased competition and gains in efficiency. There is a relatively large literature of experimental and quasi-experimental evaluations assessing the impact of interventions in the education sector in low- and middle-income countries. Several authors have reviewed this impact evaluation literature in order to draw wider conclusions on the effectiveness of education interventions (Petrosino et al., 2012; Baird et al., 2013; Morgan et al., 2012; Morgan et al., 2013; among others). A comprehensive review of existing systematic reviews identified 19 systematic reviews, 1 protocol, and 13 non-systematic reviews (see the Education Evidence Gap Map of systematic reviews here: http://gapmaps.3ieimpact.org/evidence-maps/education-evidence-gap-map), two of which were using meta-analysis. Less than half of the systematic reviews include meta-analysis (Baird et al., 2013; Kabeer et al., 2012; Kristjansson et al., 2009; Petrosino et al., 2012; Taylor-Robinson et al., 2012), and apart from Petrosino et al. (2012) these reviews focus on a relatively discrete set of interventions. Below we review these studies in brief. The majority of existing reviews focus on interventions to improve school enrolment and attendance. For instance, the impact of CCTs on schooling outcomes has been examined by five systematic reviews (Baird et al., 2013; Bouillon and Tejerina, 2007; Kabeer et al., 2012; Petrosino et al., 2012; Yoong et al., 2012), three of which used meta-analysis (Baird et al., 2013; Kabeer et al., 2012; Petrosino et al., 2012), and one meta-analysis (Saavedra & Garcia, 2013). These reviews all find that CCTs in education contribute to improving enrolment and attendance, though the evidence base on the effects on learning outcomes is limited, with available studies suggesting at best small effects. Two systematic reviews examine the impact of UCTs (Yoong et al., 2012; Baird et al., 2013). Yoong et al. (2012) report positive effects on enrolment, but only when the transfer is received by women. Baird et al. (2013) report positive effects, both on enrolment and attendance, although the size of the effect is lower than for CCTs. School vouchers are another popular intervention implemented to improve access to education, primarily by enabling children from low-income households to access private education. Three systematic reviews include studies assessing the effects of school vouchers (Bouillon & Tejerina, 2007; Morgan et al., 2013; Petrosino et al., 2012), one of which included meta-analysis (Petrosino et al., 2012). Both Bouillon and Tejerina (2007) and Morgan et al. (2013) found positive effects of vouchers on attendance and performance, and an increase in private school enrolment among the poorest income groups respectively, while Petrosino et al., (2012) found no effects of school vouchers. Morgan et al. (2012) reviewed the evidence on interventions that reduce or eliminate schooling costs (fees and uniforms) and found that such interventions strongly increased school enrolment and positively affected other education and non-education outcomes. Petrosino et al. (2012), on the other hand, reported no effect for user fee elimination, but found positive effects for provision of free uniforms. School feeding and school-based health interventions are implemented to improve both school attendance and learning outcomes, and several systematic reviews have assessed the evidence on such interventions. Two systematic reviews found school feeding interventions to have positive effects on attendance as well as on attainment (Kristjansson et al., 2009; Petrosino et al., 2012). Petrosino et al. (2012) found generally positive effects of a range of interventions providing health care (de-worming, vitamin A intake, malaria prevention, and menstruation cups), while Taylor-Robinson et al. (2012) found positive but weak evidence of the impact of de-worming on school attendance. Three reviews examined the effect of infrastructure improvements and new roads on schooling access, one of which used meta-analysis (Petrosino et al., 2012). Petrosino et al. (2012) and Bouillon and Tejerina (2007) reported positive effects on attendance, while Birdthistle et al. (2011) found no studies assessing the effects of separate sanitation facilities on girls' attendance. Ensuring a sufficient number of appropriately trained teachers who are present in classrooms is a key challenge for efforts to improve children's learning, and several systematic reviews have assessed different interventions targeting teachers, although only one review includes meta-analysis. For instance, Orr et al. (2013) examined the effect of teacher training and find mixed effects on educational attainment. Guerrero et al. (2012) found that teacher monitoring in combination with incentives is effective in tackling teacher absenteeism, although they did not find any effects on student achievement. On the other hand three systematic reviews assessing the evidence on effects of wage increases and incentives (Carr & Leggatt-Cook, 2011; Bouillon & Tejerina, 2007; Petrosino et al., 2012) found such interventions can have positive effects on students' attainment. Kingdon et al. (2013) reviewed the evidence on the effects of contract teachers, and concluded that the use of contract teachers is more effective for improving student outcomes than teachers with permanent positions. Finally, decentralisation programmes were reviewed by Bouillon and Tejerina (2007) who found that decision-making at the local level can improve performance in schools, provided that there is sufficient institutional capacity and appropriate human capital to support the intervention, without which effects may be non-existent or even negative. The review was conducted some time ago, and subsequent (non-systematic) reviews (Bruns et al., 2011) include additional studies on a broader range of school management interventions. The above summary of findings from existing reviews reveals that while there is an increasing body of systematic reviews of education interventions of relevance to LMICs, existing reviews are scattered across a wide variety of interventions and outcomes, and reviewers seldom use statistical meta-analysis to synthesise findings, resulting in many reviews with mixed or contradictory results. The education Evidence Gap Map indicates that very few reviews link up the key stages between initial, intermediate and final outcomes. Some reviews focus mainly on enrolment and attendance and others examine effects only on attainment. Fewer reviews cover academic completion or progression outcomes, while only one provides findings on the cost effectiveness of the interventions examined. Several of the existing reviews also suffer from methodological shortcomings and rely on searches completed several years ago. Knowing what works is not sufficient for policy makers who also need to know 'how to make it work in different contexts and environments and with different groups of people' (Davies, 2006:99). Twelve out of seventeen of the identified systematic reviews and meta-analyses focused on the effectiveness of interventions alone and reviewed only quantitative evidence to draw conclusions about what works. Among these reviews, only five performed sub-group analysis in order to explore effects across different groups of participants, quality of studies or geographical regions. The systematic review and meta-analysis conducted by Petrosino et al. (2012) is the most inclusive (in terms of the interventions that it covers) conducted to date. However, the authors focused primarily on outcomes related to enrolment and attendance and reported progression outcomes only when included in those studies, thus excluding studies that evaluated learning outcomes only. Additionally, the search was conducted in 2009 and studies published after that date are not included. Moreover, the review included only experimental and quasi-experimental designs, and did not engage with the theory of change of interventions and qualitative literature. Finally, the review estimated overall intervention effects, pooling different types of interventions. Sub-group-analysis was conducted, but without specifying outcome types. The issues outlined above present drawbacks for policy-makers and donors interested in being able to compare the effectiveness of interventions across outcomes and across sub-groups of participants. The importance attached to the role of education for human development is reflected by the international community's continued focus on access to, and quality of, education, as demonstrated for instance by the Education for All (EFA) initiative (UNESCO, 2013) and the Millennium Development Goals (MDGs) (UN, 2013b). The targets of the education related MDGs is to achieve universal primary education for all boys and girls (MDG 2) and eliminate gender disparities in all levels of education (MDG 3) by 2015, while the Dakar Framework for Action on Education for All (EFA) provides a strategy for achieving those objectives. Education interventions have focussed on getting children into school, whether by increasing enrolment in existing schools or building and staffing new ones where there was no school before. More recently commentators have called for a shift in focus from access to education to learning for all (Prichett, 2013; Robinson, 2011), and this is also increasingly reflected in the education policy of major agencies. For instance, learning is at the core of the education policy of the Department of International Development (DFID) in the United Kingdom (DFID, 2013). Similarly, the World Bank education strategy for the period until 2020 is focused on learning and skills development, with 'Learning for all, beyond schooling' as a primary objective (World Bank, 2011). The Dakar Framework for Action on Education for All (EFA) included a commitment that no country should be left behind in making progress towards the EFA goals due to a lack of resources and significant funding has been dedicated to education over the last decade. For instance, domestic government spending on education increased at high rates in LMICs between 1999 and 2011, despite the global economic crisis and regional food crises (UNESCO, 2012). In low income countries, the average real annual government spending on education grew at a rate of 7.2 per cent, and at a rate of five per cent in Sub-Saharan Africa (ibid), suggesting a commitment to reaching global education targets in many countries. Moreover, between 2002 and 2010, aid to education increased by 77 per cent to US$13.5 billion, with the World Bank, the USA and the UK being the largest donors to the sector (ibid). Nevertheless, the resources available for education in low-income countries still pales in comparison to the resources dedicated to education in high-income countries. In 2010, countries in North America and Western Europe spent $7916 on primary schooling per pupil (constant US$), in contrast to US$134 in Sub-Saharan Africa and US$263 in South and West Asia (UNESCO, 2012). In addition, donor spending trends from the last couple of years indicate a stagnation of aid to education and a general tightening of aid budgets as high-income countries around the world attempt to rein in their spending following the global financial crisis, reversing the aid expenditure trends of the last decade (UNESCO, 2013). These trends come despite significant remaining challenges in ensuring that all children have access to high-quality education, as noted above. To help inform decisions about how to spend limited resources, this review will provide a comprehensive review of the evidence on the relative effectiveness of education interventions in improving education access and learning outcomes for primary and secondary school children in L&MICs. This review aims to build on the work already undertaken by Petrosino et al. (2012), but with inclusion criteria covering learning outcomes more comprehensively. The primary objective of this review is to identify, assess and synthesise evidence on the effects of education interventions on children's access to education and learning in low- and middle-income countries. We will also aim to assess how education interventions affect different sub-groups of participants by incorporating sub-group analyses, and will also include a broader range of evidence to address questions relating to process, implementation and costs. While increasing generalisability of findings from single studies is one of the main arguments for conducting systematic reviews (Petticrew & Roberts, 2003), systematic reviews in international development focus on interventions across a broad range of geographical locations, settings and populations, with heterogeneity in intervention implementation and outcomes. This has raised concerns about the external validity of findings from systematic reviews, and in particular meta-analysis (Prichett & Sandefur, 2013). We aim to address these concerns through careful application of systematic review and synthesis methods as outlined in the section The review will follow the Campbell and Cochrane Collaborations approaches to systematic reviewing (Becker et al., n.d.; Hammerstrøm et al., 2010; Higgins & Green, 2011; Shadish & Myers, 2004; Shemilt et al., 2008). The review will also draw on the concepts of theory-based impact evaluation (White, 2009) and theory-based systematic reviews (Snilstveit, 2012) to provide a systematic review and analysis along the causal chain, reviewing evidence on context, process and implementation to identify barriers and facilitators to improvement of educational outcomes. The review will systematically collect and synthesise quantitative evidence from impact evaluations of education interventions to answer review questions 1a and 1b. If sufficient data is available outcomes will be synthesised along the causal chain, from intermediate to final outcomes. For the review to be more useful for policy-makers and practitioners, we will extend the review of effectiveness (Noyes et al., 2011) by collecting quantitative and qualitative evidence on process and implementation, context and underlying factors that determine or hinder the effectiveness of interventions to address review questions 2a and 2b, using a combination of qualitative synthesis and meta-regression analysis. The review will include studies in two phases (See Figure 3, below). To address questions 1a and 1b, we will include studies meeting the inclusion criteria outlined above. To address questions 2a and 2b, studies that pass these criteria will then be used as the basis for a second phase to identify and include qualitative studies, project documents, process evaluations and other supplementary data on the programmes examined by the studies included to address questions 1a and 1b. To address questions 1a and 1b we will include studies that assess the effects of interventions using experimental and quasi-experimental study designs that allow for causal inference. Quasi-experimental studies may be subject to bias in their estimate of treatment effects, however, studies have shown that if well conducted quasi-experimental studies can provide un-biased estimates of treatment effects (Cook et al., 2008; Shadish, 2011). In setting our inclusion criteria we have aimed to incorporate studies that adopt techniques which empirical research suggest are effective in reducing or removing bias. Including a pre-test measure of the outcome and controlling for appropriate covariates in particular have been found to be important in reducing selection bias in quasi-experimental studies (Steiner et al., 2010; Shadish, 2011). Thus, studies without random allocation to treatment and comparison group that do not include a baseline measure of the outcome variables will be excluded. Similarly, studies without random allocation to treatment and comparison group that do not use matching or other statistical methods to control for selection bias and confounding will be excluded. The selection and measurement of appropriate covariates that are correlated with both the selection or allocation of the treatment and the outcomes is important in reducing selection bias in quasi-experimental studies (Steiner et al., 2010). We will not address this issue at the inclusion stage, but we will do so when assessing the risk of bias in all included studies (details below). To avoid confounding treatment effects with teacher or school effects studies of any classroom and school level educational interventions, whether they are randomised or not, will be excluded if they have less than two teachers or schools in each group. Finally, our interest is in identifying the evidence on the effects of an intervention implemented as part of a program under circumstances that approach 'real-world' practice, so-called effectiveness studies. These types of studies stand in contrast to efficacy trials which test an intervention under ideal and controlled conditions in order to maximise the likelihood of observing an effect, if one exists. Although there exists broad agreement on the type of study design characteristics of effectiveness (pragmatic) trials and efficacy (explanatory) trials, there is currently no validated definition of 'effectiveness studies' (Treweek et al., 2009; Gerthlener et al., 2006; Singal et al., 2014). Furthermore, as Thorpe et al. (2009) note, the distinction between the two types of trials can be regarded as a continuum rather than a dichotomy as very few trials are purely pragmatic or explanatory. In order to distinguish effectiveness from efficacy studies we developed five criteria, drawing on two existing tools (Gartlehner et al., 2006; Thorpe et al., 2009). Studies will be considered efficacy trials and will be excluded if they fulfil at least one of the criteria outlined below: A project document providing information about planned, ongoing or completed interventions. They may describe the background and design of an intervention, or the resources available for a project for instance. As such, these documents do not typically include much analysis of primary evidence, but they provide factual information about interventions. The purpose of including them in our review is to ensure we have sufficient information about the context and interventions in included studies. The review will include interventions targeted at primary school and secondary school1 age children in mainstream education in low- and middle-income countries (L&MICs), as defined by the World Bank, at the point in time that an intervention was carried out. We will exclude studies focusing on refugees, migrants and orphans only. We will also exclude studies of children with special educational needs. All adult education interventions, including those that are university-based, will be excluded. We will also exclude studies from high-income countries as the differences with L&MICs, in terms of policy challenges, resources devoted to education systems, state capacity and broader contextual factors, are such that we consider this evidence to be of limited applicability Broadly defined, interventions will be included in the proposed review if they aim to improve the access and/or the quality of primary and secondary education in low- and middle-income countries. The interventions will cover a broad range of programs and in doing so we acknowledge that there are a wide range of options