Portal de Periódicos da CAPES

Do location-specific forecasts pose a new challenge for communicating uncertainty?

2015; Wiley; Volume: 22; Issue: 3 Linguagem: Inglês

10.1002/met.1487

1469-8080

Shymali Abraham, R. E. Bartlett, Matthew Standage, Alison Black, Andrew Charlton‐Perez, Rachel McCloy,

Decision-Making and Behavioral Economics

Meteorological ApplicationsVolume 22, Issue 3 p. 554-562 RESEARCH ARTICLEOpen Access Do location-specific forecasts pose a new challenge for communicating uncertainty? Shymali Abraham, Shymali Abraham Department of Psychology, University of Reading, UKSearch for more papers by this authorRachel Bartlett, Rachel Bartlett Department of Meteorology, University of Reading, UKSearch for more papers by this authorMatthew Standage, Matthew Standage Centre for Information Design Research, Department of Typography & Graphic Communication, University of Reading, UKSearch for more papers by this authorAlison Black, Alison Black Centre for Information Design Research, Department of Typography & Graphic Communication, University of Reading, UKSearch for more papers by this authorAndrew Charlton-Perez, Corresponding Author Andrew Charlton-Perez orcid.org/0000-0001-8179-6220 Department of Meteorology, University of Reading, UK Correspondence: A. Charlton-Perez, Department of Meteorology, University of Reading, Reading, UK. E-mail: a.j.charlton-perez@reading.ac.ukSearch for more papers by this authorRachel McCloy, Rachel McCloy Department of Psychology, University of Reading, UKSearch for more papers by this author Shymali Abraham, Shymali Abraham Department of Psychology, University of Reading, UKSearch for more papers by this authorRachel Bartlett, Rachel Bartlett Department of Meteorology, University of Reading, UKSearch for more papers by this authorMatthew Standage, Matthew Standage Centre for Information Design Research, Department of Typography & Graphic Communication, University of Reading, UKSearch for more papers by this authorAlison Black, Alison Black Centre for Information Design Research, Department of Typography & Graphic Communication, University of Reading, UKSearch for more papers by this authorAndrew Charlton-Perez, Corresponding Author Andrew Charlton-Perez orcid.org/0000-0001-8179-6220 Department of Meteorology, University of Reading, UK Correspondence: A. Charlton-Perez, Department of Meteorology, University of Reading, Reading, UK. E-mail: a.j.charlton-perez@reading.ac.ukSearch for more papers by this authorRachel McCloy, Rachel McCloy Department of Psychology, University of Reading, UKSearch for more papers by this author First published: 18 May 2015 https://doi.org/10.1002/met.1487Citations: 12AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat ABSTRACT In the last decade, the growth of local, site-specific weather forecasts delivered by mobile phone or website represents arguably the fastest change in forecast consumption since the beginning of television weather forecasts 60 years ago. In the present study, a street-interception survey of 274 members of the public a clear first preference for narrow weather forecasts above traditional broad weather forecasts is shown for the first time, with a clear bias towards this preference for users under 40 years. The impact of this change on the understanding of forecast probability and intensity information is explored. While the correct interpretation of the statement ‘There is a 30% chance of rain tomorrow’ is still low in the cohort, in common with previous studies, a clear impact of age and educational attainment on understanding is shown, with those under 40 and educated to degree level or above more likely to correctly interpret it. The interpretation of rainfall intensity descriptors (‘light’, ‘moderate’ and ‘heavy’) by the cohort is shown to be significantly different to official and expert assessment of the same descriptors and to have large variance amongst the cohort. However, despite these key uncertainties, members of the cohort generally seem to make appropriate decisions about rainfall forecasts. There is some evidence that the decisions made are different depending on the communication format used, and the cohort expressed a clear preference for tabular over graphical weather forecast presentation. 1 Introduction The ways in which weather forecasts are delivered to the general public have undergone a significant change in the last 5 years. In common with other forms of content consumption there has been a shift from broadcast media such as radio, television and daily newspapers towards more flexible and personalized ‘narrow-cast’ consumption. In the narrow-cast style of communication, consumers expect and attain some control over the information they receive (Swatman et al., 2006; Hirst and Harrison, 2007). One example of this process, which is particularly relevant to weather forecasting, stems from the rapid market penetration of smartphones in the United Kingdom. By the middle of 2014, 75% of the population was expected to have access to a smartphone (http://www.marketingmagazine.co.uk/article/1216797/iab-engage-smartphone-penetration-reach-75-2014). Almost all new smartphones come packaged with a weather forecasting application and there is wide uptake of additional enhanced weather forecasting applications. For example, the Met Office launched apps for iPhone and Android phones in 2012, which have been subsequently downloaded more than 5.5 million times (http://www.metoffice.gov.uk/services/iphone). Both smartphone and other web-based applications provide regularly updated and highly location-specific forecasts of weather variables. Many forecast providers give forecasts down to individual post-codes (typically around 0.15 km2), representing a change not only in forecast specificity and availability, but also in resolution (although it is important to be clear that the forecast data used often have much coarser resolution). This study attempts to understand in detail if this change in forecast presentation poses a new challenge to forecasters who seek to communicate the uncertainty inherent in forecasts at this hyper-local scale. The focus is on forecasts of precipitation because this is the most studied variable in the literature on public understanding of forecast uncertainty and is particularly pertinent for consumers in the United Kingdom. For some time there has been interest in understanding how well different methods of communicating uncertainty translate into operational weather forecasts (see, e.g. Murphy et al., 1980). Recent studies have used large surveys to develop a clearer picture of the communication process. End-users typically infer a background level of uncertainty in weather forecasts, even if this is not stated explicitly (Morss et al., 2008) and they have a good heuristic understanding of both the decreasing skill of weather forecasts with increased lead time and in the different levels of skill for different forecast variables (Morss et al., 2008; Joslyn and Savelli, 2010). However, when members of the public are asked to interpret probability of precipitation (PoP) forecasts, they are more likely than not to misinterpret the measure of uncertainty provided by the forecast (Gigerenzer et al., 2005; Morss et al., 2008). It appears that people who do not correctly interpret PoP forecasts may tend to risk averse interpretations (Joslyn et al., 2009), although there is increasing evidence of a complex relationship between probabilistic forecasts and end-user decision making, which does not conform to simple cost-loss models (Morss et al., 2010). One under-explored aspect when considering end-user decision making under uncertainty is the extent to which descriptors of the likely intensity of precipitation are interpreted correctly by the general public, or if these influence decision making under uncertainty. It is hypothesized that the rapid penetration of narrow-cast weather forecast information in the United Kingdom may have had an influence on the way the general public consume PoP forecasts. To test this hypothesis, a street-interception survey of the general public in and around Reading, UK, during July and August 2013 was performed. Two recent studies have also used similar methods to investigate public understanding of forecast uncertainty in a group of undergraduates at the University of Manchester (Peachey et al., 2013) and weather enthusiasts in the Republic of Ireland (O'Hanrahan and Sweeney, 2013). This study builds on these studies to provide a new view of forecast consumption for UK consumers. By using the street-interception method, it was possible to survey a broad demographic range of consumers (see Section 2 for further discussion). The aim of the study is to answer the following questions: do the UK public have the same level of understanding of PoP forecasts as reported in other studies? do the UK public understand descriptive information about rainfall intensity? when combined, how do these factors influence understanding and decision making? how does the way in which forecast information is presented influence perception and decision making? The study methods are presented in Section 2. Section 3.1 presents results showing that narrow-cast technology is a significant source of weather forecast information, particularly amongst those under 40. Section 3.2 then analyses understanding of PoP forecasts both in terms of probability and precipitation intensity in the study sample. Section 3.3 then shows how these factors influence forecast preference and decision making. Finally, Section 4 presents conclusions and suggestions for further work. 2 Methods 2.1 Participants and procedure A total of 274 people responded to the questionnaire (144 females; 128 males; 2 participants did not record their gender). The participants had a mean age of 40.6 years (range 13–92 years). A total of 237 participants identified their nationality as British with 37 participants saying that they were from elsewhere. The survey took place during July 2013 in two main locations: at a range of public events in and around Reading Town Centre and at an open day at the excavations at Silchester Roman Town an archaeological site open to the public, approximately 14 km southwest of Reading. Completion of the survey took 5–10 min and all participants provided informed consent and were told that they were free to withdraw at any time. 2.2 Study tasks and materials The survey was split into six sections. The sections were as follows. Section 1 asked for basic demographic information (age, gender, nationality, educational level). In this section, four further questions on why participants usually consulted weather forecasts, what sources they used and with what frequency, which source they preferred and why were also included. Section 2 presented participants with verbal descriptions of a range of rainfall estimates that varied in their probability (40% chance of rain; 60% chance of rain) and in their intensity (no intensity information; light rain; moderate rain; heavy rain). On the basis of each of these estimates participants were asked to judge how likely they would be to change their plans to attend an outdoor event. Section 3 presented participants with a choice of times at which they could choose to carry out a particular outdoor activity. Each timeslot was presented alongside information about the likelihood of rain and its intensity. Likelihood information was given either as percentages (e.g. 20 and 60%) or as verbal probabilities (low, medium and high). Intensity information was presented either just verbally or with an additional visual cue (in terms of progressively more strongly hatched box indicators on a scale). The information was either presented to participants verbally in a tabular format or graphically using a scale layout (with probability scales arranged either horizontally or vertically). By design, none of these formats directly replicates presentations common in the UK media in order to avoid biases in user preferences towards a familiar presentation, but they do have much in common with typical ways in which forecast organizations present time-based forecasts (see, e.g. temperature ranges at http://www.metoffice.gov.uk/public/weather/forecast). Figure 1 shows the presentation formats used. For each choice, participants were asked to select which of the timeslots they would prefer. Section 4 presented participants with the four different presentation formats used in Section 3 (verbal, verbal numeric, graphic vertical and graphic horizontal) and asked them to choose the one they preferred. Section 5 assessed participants' understanding of verbal intensity levels used in weather forecasts. For each of light, moderate and heavy rain, participants were asked to provide assessments of the amount of rain that was likely to fall (in mm h−1), of the length of time that it would take for puddles to form (in minutes) and of the things they might see or experience with each type of rainfall. Section 6 asked participants to rate their current experience with weather forecasts. They rated their confidence in forecasts in general and their satisfaction with the forecasts that they currently receive. Participants were additionally asked if they had consulted a weather forecast the previous day, and, if so, what source they had used and to rate how accurate they believed the forecast had been. Finally, as part of this section, participants were asked a question to assess their understanding of uncertainty in rainfall forecasts based on that used by Gigerenzer et al. (2005). Participants were asked to select the correct interpretation of the phrase ‘a 30% chance of rain’ from a selection which included the correct interpretation (the ‘days like this’ interpretation), along with two common misunderstandings (the ‘time’ interpretation and the ‘area’ interpretation). Participants were also free to provide their own explanation. Figure 1Open in figure viewerPowerPoint Examples of forecast presentation methods tested: (a) verbal; (b) verbal numeric; (c) graphic vertical; (d) graphic horizontal. Our aim across the different sections of the survey was to get a picture of participants' understanding of and preference for different kinds of uncertainty and intensity information. 2.3 Study design Four variants of the questionnaire were prepared in order to balance question content across the different rainfall forecasts in Section 2 and to allow comparison of the tabular formats (verbal and verbal numeric) with the graphic (vertical and horizontal scales, respectively) in the selection of times for activities in Section 4. Participants were assigned at random to each of the four questionnaire variants. 3 Results 3.1 How do users get forecast information? Before discussing the ways in which end-users in this cohort understand and interpret uncertainty it is important to understand the means by which they access forecast information, the reasons why they make these choices and the trust they place in current forecast information. Table 1 shows the primary source used by respondents for gathering weather information. The clear majority (68%) prefer narrow-cast channels (website and mobile phone) to traditional broadcast media (television and radio). This is a considerable contrast to an analysis of a US sample in 2009 by Lazo et al. which showed that 90% (albeit of a wider population) rarely or never used electronic devices for weather information (Lazo et al., 2009). Splitting the cohort into two age categories shows that 75% of those whose primary source of information is mobile phone were under the age of 37 and no users in this group picked radio as their primary source. In contrast, three times as many users older than 40 as those of 40 or younger picked television as their primary source of weather information. The cohort was split at age 40 because this divided the study group almost equally in two. Table 1. Preferred source of weather forecast information for respondents (sample size: 265) split into respondents whose stated age is 40 or below and whose stated age is above 40 Preferred source for weather forecast information Total Mobile phone Website Television Radio Respondents aged 40 or below 58% (77) 28% (37) 13% (17) 1% (1) 132 Respondents aged above 40 19% (25) 30% (40) 38% (51) 13% (17) 133 Total 38% (102) 29% (77) 26% (68) 7% (18) 265 Raw number of responses is shown in brackets. A chi-squared test shows that the difference in the distribution of source preference for the two age groups is highly significant (df = 3, N = 265, X2 = 57.846, p = 0.00). To get a broader picture of forecast consumption, participants were also asked about frequency of use of a range of different sources. Comparing the behaviour of groups with an expressed preference for phone and web versus television and radio forecasts reveals that forecast use on phones is often supplemented by other sources such as television (>50% of the phone/web group still use television forecasts at least twice per week). In contrast, a clear majority of those preferring television (65%) rarely or never use mobile phone forecast apps. This difference may present a challenge in the future for forecast providers to present a consistent and clear message across different forecast formats. Because of the contrast between age groups and apparent change in behaviour in the last 5 years it is important to try to understand how users choose their primary source. Figure 2 shows responses to question 1.9, which asked users to pick the most desirable aspects of weather forecasts for them. More than 50% of respondents cited ease of access as the most desirable aspect of forecasts. This is consistent with the apparent shift of use towards narrow-cast information, particularly that provided via mobile phone. Participants were asked for their levels of satisfaction with and confidence in current forecasts. A total of 79% of respondents indicated they were satisfied or very satisfied with the current forecasts they use but only 44% had very high or high confidence in the accuracy of their forecasts. This contrast is consistent with earlier studies (Morss et al., 2008, 2010) which indicate that forecast users in the general public have a sophisticated appreciation of the limitations of weather forecasts and match their expectations of forecast performance to this. Both the level of satisfaction and level of confidence in forecasts were similar for respondents who expressed a preference for narrow forecasts and those who preferred broad forecasts. Figure 2Open in figure viewerPowerPoint Reasons cited by respondents for their choice of preferred forecast source. Numbers are expressed as percentages of responses (sample size: 258). 3.2 How do end-users understand precipitation forecasts? 3.2.1 How do users attribute probability in precipitation forecasts? To compare the understanding of probabilistic forecast information for this cohort with previous groups, respondents were asked a standard question (about their interpretation of probabilistic information) that was used by several previous studies (Gigerenzer et al., 2005; Morss et al., 2008; Peachey et al., 2013): Imagine that the weather forecast predicts ‘There is a 30% chance of rain tomorrow’. Please indicate which of the following is the most appropriate interpretation of the forecast? The correct interpretation of the statement is that this will occur on 30% of days like tomorrow. Figure 3 shows the frequency of the different interpretations given by respondents. Figure 3Open in figure viewerPowerPoint Interpretation of the statement ‘There is a 30% chance of rain tomorrow’ by respondents. Possible answers were ‘It will rain in 30% of the region’; ‘It will rain for 30% of the time’; ‘It will rain on 30% of days like tomorrow’; ‘I don't know’ and ‘other’. Responses are expressed as percentage of total number of answers (sample size: 271). In common with previous studies, a majority of respondents did not interpret this statement correctly, and a substantial fraction answered ‘other’ (again in common with Morss et al., 2008; Joslyn et al., 2009; Peachey et al., 2013) which suggests widespread difficulty in interpreting probabilistic forecast statements. In this sample, interestingly, of the three categories (region, time and days) in which respondents could indicate that they understood what the statement meant, the correct interpretation (days) was the most common answer (27%) indicating some understanding of PoP forecasts. One important caveat here, which may also be true in other studies, is that a significant proportion of respondents answered ‘other’ and that often these respondents did potentially demonstrate some understanding of the PoP forecast by restating the question posed (see Table 5). Gigerenzer et al. (2005) hypothesized that increased familiarity with PoP forecasts improved the accuracy with which the public interpreted them (their study showed greater accuracy for respondents from New York compared with several European cities, where PoP forecasts are not commonly employed). In the United Kingdom, provision of PoP forecasts (as opposed to deterministic forecasts) is mixed, but increasingly PoP is provided in narrow-cast forecasting services, such as smartphone apps. To test if differences in forecast consumption might influence the accuracy with which users interpret PoP, the sample was segregated by several different criteria. Comparing interpretation of the PoP forecast by respondents who preferred narrow forecasts with those who preferred broad forecasts showed no significant difference in interpretation between the two groups (Table 2). This suggests that the relatively recent introduction of narrow weather forecasts in the United Kingdom has yet to influence people's comprehension of probabilistic information, although the variety of presentation techniques used by forecast providers shows that not all end-users will have seen probabilistic representations. Table 2. Interpretation of the statement ‘There is a 30% chance of rain tomorrow’ when respondents are segregated by narrow or broad cast preference, for respondents who did not answer ‘other’ or ‘don't know’ Interpretation of the statement ‘There is a 30% chance of rain tomorrow’ Total Region Time Days Narrow cast preference (mobile and Internet) 26% (25) 25% (24) 49% (48) 97 Broad cast preference (television, radio and newspaper) 24% (13) 33% (18) 43% (23) 54 Total 25% (38) 28% (42) 47% (71) 151 Raw number of responses is shown in brackets. There is no significant difference in the pattern of responses using a chi-squared test (df = 2, N = 151, X2 = 1.310, p = 0.519). However, when the respondent group was split into two sub-groups based on age (above and below 40) there was a significant difference in their responses (using a chi-squared test, Table 3), with those in the younger group more likely to give the correct response. Similarly, when the respondent group was split into those with degree-level education and above and those without, the sub-group with the higher level of educational qualification were also more likely to give the correct response (Table 4). Table 3. Interpretation of the statement ‘There is a 30% chance of rain tomorrow’ when respondents are segregated by age (above and below 40) for respondents who did not answer ‘other’ or ‘don't know’ Interpretation of the statement ‘There is a 30% chance of raintomorrow’ Total Region Time Days Respondents aged 40 or below 16% (11) 27% (19) 57% (40) 70 Respondents aged above 40 33% (28) 29% (24) 38% (32) 84 Total 25% (39) 28% (43) 47% (72) 154 Raw number of responses is shown in brackets. There is a significant difference in the pattern of responses using a chi-squared test (df = 2, N = 154, X2 = 7.671, p = 0.02). Table 4. Interpretation of the statement ‘There is a 30% chance of rain tomorrow’ when respondents are segregated by educational attainment (at degree level) for respondents who did not answer ‘other’ or ‘don't know’ Interpretation of the statement ‘There is a 30% chance of rain tomorrow’ Total Region Time Days Respondents with education below first degree level 32% (22) 32% (22) 35% (24) 68 Respondents with education at first degree level and above 22% (18) 24% (20) 54% (45) 83 Total 26% (40) 28% (42) 46% (69) 151 Raw number of responses is shown in brackets. There is a marginally significant difference in the pattern of responses using a chi-squared test (df = 2, N = 151, X2 = 5.450, p = 0.066). These results suggest that there may be an effect of exposure to probabilistic information when users interpret PoP forecasts, but that this is likely related to exposure to thinking about uncertainty, generally, during their educational career. Given the large rise in the proportion of people going on to higher education in the United Kingdom after the age of 16 over the last 40 years (8.4% in 1970 and 33% in 2000, House of Commons library) the results in Tables 3 and 4 are unlikely to be independent. Finally in this section, write-in responses for participants who selected ‘Other’ were explored (see Table 5). For this survey, it was found that the range of write-in responses was smaller than that of Morss et al. (2008) and similar to Peachey et al. (2013). Table 5. Write-in answers for respondents who answered ‘other’ to the question ‘There is a 30% chance of rain tomorrow’ Interpretation Example answer Percentage of ‘other’ responses Restatement: probability ‘There is a 30% likelihood of it raining’ 39% (36) Restatement: probability and reverse ‘There is a 70% chance it will be dry’ 21% (19) Restatement: odds ‘There is a 3 in 10 chance of rain – low rain’ 11% (10) Restatement: worded ‘There is a low chance of it will rain’ 29% (26) Numbers in brackets show total number of responses, sample size, N = 91. As in previous studies, almost all of these answers focussed on re-writing the probabilistic statement in some way rather than specifying what users thought the probability measure referred to. 3.2.2 How do users understand precipitation intensity descriptors? In addition to information about likelihood of rain, forecasts also typically give information about expected intensity of precipitation. This is usually given in verbal descriptors such as ‘light’, ‘moderate’ and ‘heavy’ or equivalent graphical signifiers of these classes (e.g. the number of rain drops below a cloud). How users understand these intensity descriptors and, subsequently, how these two parts of the forecast (intensity and likelihood) influence decision making were investigated. Respondents were asked for their understanding of rainfall intensity descriptors in three different ways: as a numerical estimate of rainfall rate, the amount of time they would expect for puddles to form on road surfaces and a descriptive comparison of what they would expect to see. Because interpretation of verbal descriptors is inherently subjective, a second, small survey of seven academic experts in meteorology was performed at the University of Reading to compare their understanding with that of the general public. Current Met Office practice is to split rainfall verbal descriptors into drizzle, rainfall and rain showers classes and then use additional descriptors (‘slight’, ‘moderate’ and ‘heavy’) within each class. Numerical values for these classes are assigned as follows. For rain (other than in showers), ‘slight’ is <0.5 mm h−1, ‘moderate’ is 0.5–4 mm h−1 and ‘heavy’ is <4 mm h−1. For rain showers, ‘slight’ is <2 mm h−1, ‘moderate’ is 2–10 mm h−1 and ‘heavy’ is <10–50 mm h−1. An additional class (violent) is used for showers but is not discussed further here. Figure 4 shows mean estimates and 95% confidence intervals for numerical estimates of rainfall intensity and the time for puddles to form for the expert and the general public cohort. Figure 4Open in figure viewerPowerPoint Mean estimates of rainfall intensity (mm h−1) and time (minutes) for puddles to form for three rainfall descriptors for general public (light grey) and expert (dark grey) groups (95% confidence interval for each mean estimate is shown by the error bars). Sample sizes, general public group for rainfall amount in millimetres (light = 218, moderate = 216, heavy = 217) and time for puddles to form (light = 241, moderate = 250, heavy = 250). There is a significant difference between experts and the general public in their estimates of both rainfall rate and the time taken for puddles to form in all rain rate categories and in the moderate and heavy time for puddles to form categories (see Table 6). For rainfall rate, the public generally estimate much higher rainfall rates than experts, with the expert estimates consistent with the official definitions for rain showers. Importantly, the standard deviation of estimates from the general public was higher than that of the experts for the light and moderate categories, which indicates wide variance in understanding of rainfall rates, although this is true for both experts and the public for heavy rainfall. Similarly for estimates of the time taken for puddles to form, there is a wide variation amongst members of the public and experts for light rainfall. There were also a large proportion of the public survey returns that did not make any estimate for these categories (∼20% for estimates of rainfall rate in mm h−1 and ∼10% for estimates of the