‘Any prediction is better than none’? A study of the perceptions of fire behaviour analysis users in Australia

Keywords: fire management, decision making, predictive services, risk communication, fire management modelling, planning, Australia.

Fire and land management agencies in many fire-prone countries, such as Australia, Chile, USA, Spain and Canada, have long utilised fire behaviour analysis to inform operational, seasonal and long-term decision-making in wildfire management. In recent decades, this has often taken the form of specific trained practitioners, such as fire behaviour analysts (FBANs), generating forecasts of a given wildfire’s potential spread and behaviour, informing various aspects of incident and hazard management, identifying risks relating to ignition likelihood and potential impacts (Cruz et al. 2014; Gibos et al. 2015; Pacheco et al. 2015). Established and new stakeholders within and outside incident response have increasingly come to rely upon these practitioners and their forecasts to respond to severe wildfire seasons, though the underlying modelling still contains ‘significant knowledge gaps’ (Pacheco et al. 2015: 7; Plucinski et al. 2017). Such changes in use may potentially be driven not only by the increasing frequency and severity of fire seasons (Abatzoglou et al. 2019), and the increasing number of people living in wildland–urban interfaces (Mell et al. 2010), but also by heightened expectations placed on fire and land management agencies (Bhandari et al. 2015) and their forms of predictive analysis (Neale and May 2018).

In Australia, there has been a pronounced increase in the use and prominence of fire behaviour analysis over the past two decades, demonstrated by the swelling numbers of staff trained and employed as FBANs, increased availability of fire prediction software, aligned recommendations from official inquiries and reviews, and the increased dissemination of predictive spread maps within emergency management (Neale and May 2018; Neale and May 2020). Following the landmark 2019–2020 bushfire season in south-east Australia, state and federal inquiries have recommended further investment and training in predictive fire behaviour analysis (e.g. Owens and O’Kane 2020; RCNNDA 2020: 23); some suggesting not only that such capabilities were ‘very valuable’ but that they should also be more available to both incident managers and public audiences (SAIR 2020: 57). During that season, and for the first time, fire agencies in New South Wales (NSW) and the Australian Capital Territory (ACT) released fire spread predictions on social media ahead of high-risk days (Whittaker et al. 2021), suggesting a high level of institutional support for this form of analysis.

FBANs and their predictions are produced within the type of incident management systems that have become ubiquitous internationally and in Australia over the past several decades (e.g. Hayes and Omodei 2011). These systems have multiple origins; however, as Collier and Lakoff (2015) suggest, these systems were significantly shaped by Cold-War-era efforts to establish consistent and hierarchical ‘command and control’ arrangements around vital systems in times of crisis. Many have doubts about the efficiency of these systems (e.g. Quarantelli 2000; Flin and Arbuthnot 2017), and existing studies of both their functioning and their use of predictive analysis have suggested they bear out many of the lessons of other complex sociotechnical systems, particularly in relation to the importance of trust and interactive relationships between systems, practitioners and their contexts (McLennan et al. 2006; Thompson 2014). In short, predictive decision-making tools and platforms are not neutral technologies but rather are socially constructed (e.g. Morss et al. 2005; Demeritt et al. 2013). They are encoded with the social values and preferences of the institutions and actors involved in their development and use.

If this point seems too abstract, the most well studied fire behaviour analysis system provides a clear example of the influence of values and preferences in such contexts. The US Forest Service’s Wildland Fire Decision Support System (WFDSS) – which began development in 2005 before it began being used in 2009 – was partly designed to incorporate recent scientific advances, including in fire behaviour modelling, into fire managers’ decision-making; the intent was to provide a ‘common scale’ for risk-based decision-making during fire events (Finney 2005; Thompson et al. 2011). Ideally, WFDSS would make operational trade-offs apparent to managers before their decisions about strategies, generating ‘optimally balanced’ decisions aligned with government policies (Calkin et al. 2013; Wibbenmeyer et al. 2013) and ensuring chosen strategies had defined outcomes and objectives (Calkin et al. 2014). However, as we explain in the following, studies of the system’s use suggest that it is often used to retrospectively justify decisions rather than optimise them.

As Noble and Paveglio (2020: 157) note, researchers frequently assert the benefits of new wildfire management tools but very rarely demonstrate them, or seek to explain their failures (e.g. FireDST (Australia), Fire Program Analysis (USA), see Pacheco et al. 2015), in part because utilisation is very difficult to research. Although such innovations are typically framed as technical solutions, they are in fact social interventions entering contexts with established dynamics of trust and authority. Thus, as Owen et al. (2012) demonstrated, interpersonal communication and relationships are more influential in the use of predictive analysis than, say, the scientific rigour of that analysis. The ‘trust and credibility’ of a new system needs to be progressively built through social connection, Rapp et al. (2020: 587) argue, given that practitioners often perceive such inventions as challenges to their own authority. Based on interviews with WFDSS users, Rapp et al. (2020) argue that it is most often used to retrospectively justify or document operational decisions that users have already made.

In their review of such decision support systems (e.g. WFDSS, FireDST, etc.), Pacheco et al. (2015: 10–14) note that ‘the path between science and practical implementation, involving multiple stakeholders, is often not clear’ and that, because ‘their adoption by policymakers and managers has been generally feeble,’ we need to better understand the social and institutional constraints of their use. This paper approaches the growing uptake in Australia of FBANs and their predictive outputs as an opportunity for exactly this kind of insight. How are these predictive practitioners and their analysis perceived by their growing number of users, and are there any discernible patterns in these perceptions? To answer this question, we conducted a survey of predictive services users across most Australian jurisdictions, testing hypotheses about users’ values and preferences developed through previous qualitative research. Our results suggest that, indeed, there is a need to foster social connection between these predictive practitioners, their tools and their users.

Over several fire seasons, Neale developed qualitative research projects with FBANs in multiple Australian states, reviewing grey literature (Neale and May 2018), interviewing more than 20 practitioners (Neale and May 2020) and conducting participant observation with FBANs in Victoria and South Australia during the 2019–2020 bushfire season. Through this research, they developed several hypotheses about predictive services users’ values and preferences.

Hypothesis 1: the majority of users prefer timeliness over accuracy in predictive analysis. Our aforementioned prior research suggested that fire managers are under increasing pressure to make fast decisions about suppression resources and tactics and that user needs for spatiotemporal specificity increase as a fire grows and its potential impacts increase.

Hypothesis 2: users with greater seniority (i.e. incident controller, regional controller and above) and experience (i.e. 20 years or more in fire management) are more cautious than other users about releasing FBAN outputs (e.g. fire spread predictions) to the general public.¹ This is based on our prior research (Neale and May 2018; Neale and May 2020) and previous studies (e.g. Rapp et al. 2020) and their common findings that newer technologies and forms of expertise can be perceived by established actors as challenges to their authority.

Hypothesis 3: users with greater seniority and experience differ from others in their assessment of the most important skills for FBANs. This is based on our prior research, which suggested that, owing to differences in training and career path, there are cultural differences between more senior fire managers and their colleagues.

Hypothesis 4: users with greater seniority and experience are more likely to use predictive services more frequently. This is based on our prior research, previous studies (e.g. McLennan et al. 2006), and an analysis of incident management structures (AIDR 2019), according to which more senior roles have greater decision-making responsibilities and therefore greater need for predictive intelligence.

Hypothesis 5: users who consult predictive services more frequently have more positive perceptions of FBANs and their outputs than other users. Our prior research and previous studies (e.g. Owen et al. 2012; Rapp et al. 2020) suggest that there is such a correlation between opinion and exposure to FBANs and their outputs.

Hypothesis 6: the majority of users prefer the advice of a local fire manager to a non-local FBAN. Our prior research and previous studies (e.g. Nyquist 2019) suggest that local knowledge is highly valued within fire agencies, particularly in comparison to knowledge based in scientific tools.

We contacted all Australian emergency management agencies to compile a comprehensive list of individuals who had been in roles identified as predictive services users in the past bushfire season (e.g. incident controllers, planning officers).² Agencies from all Australian states and territories agreed to participate, excluding Queensland and the Northern Territory, and invitations to participate in the 37 question survey (see Supplementary Material) via the online survey platform Qualtrics were emailed directly to 616 contacts at the start of the 5-week survey period on 1 October 2020 (see Table 1).³ At the end of the survey period, 208 valid responses had been received (33.7% response rate), with over 80% of the total response from participants in Victoria, New South Wales or South Australia.⁴ Male respondents (88.6%) made up a significant majority of respondents, which is broadly reflective of the current sector workforce gender diversity of ~80–85% male (AFAC 2020), as did respondents 40–59 years old (75.6%). As might be anticipated, there was a statistically significant association between the seniority of respondents’ roles and length of experience (P = 0.11).

Table 1.  Description of survey respondents by jurisdiction, length of experience and seniority of role

The materials were developed iteratively in collaboration with FBANs with 5 or more years of experience. Neale met with an invited group of 8 FBANs in April–May 2020 to develop the survey instrument, which was piloted with 12 FBANs during July 2020 to seek feedback and refine its structure, length and wording. We collected information from respondents regarding their jurisdiction of employment, current operational role in bushfire response and length of operational experience. Then, we asked questions about the frequency of use and perceived qualities of FBAN services. Subsequently, respondents were asked to rank seven given qualities that, in their view, were ‘most important’ for an FBAN to possess. These skills were as follows: knowledge of fire science, knowledge of computer programming, operational experience of fire behaviour tools, fireground experience, interpersonal communication skills, the ability to improvise based on intuition, the ability to follow official rules and guidelines, and a residual ‘other’ category that we asked respondents to add if they wanted. We assigned a score from 1 to 8 to each quality based on the ranking given by respondents (8 = most important, 1 = least important). We then asked whether the predictive services received from FBANs meet their needs in terms of accuracy, quality, timeliness and clarity on a Likert scale (5 = strongly agree to 1 = strongly disagree).

Subsequently, we presented respondents with hypothetical scenarios of predictive services being used in operational fire response, asking respondents to make and assess decisions within these scenarios. In the first scenario (Scenario A), a fire event has begun and an FBAN, presented with uncertain information about the fire’s origin and current location, refuses to provide users with any predictions until they have reliable fireground intelligence. Respondents had to assess whether the FBAN’s decision was appropriate or not appropriate and then provide an explanation of this assessment.

In the second scenario (Scenario B), there is an existing fire near a town, a bad fire weather day is expected tomorrow, and an FBAN provides a predictive spread map showing it is possible that the town will be affected by the fire. After reading the scenario, respondents were asked ‘If you were the incident controller, how would you likely utilise this map?’ and they were asked to rate their likelihood to do the following: hold a town meeting and show the map to residents, prepare a Watch and Act for the town, publish the prediction map on social media and send it to news media, send the prediction map to firefighting contacts via email and/or text message. Responses were collected on a 5-point Likert scale (1 = extremely unlikely, 2 = moderately unlikely, 3 = neither likely nor unlikely, 4 = moderately likely, 5 = extremely likely). We created a composite measure of the four items (α = 0.54).

In the third scenario (Scenario C), the respondents were receiving conflicting and incompatible analysis of two fires (Fire X and Fire Y), with a local fire manager near the firegrounds saying Fire X has greater potential impact and an FBAN far from the firegrounds saying Fire Y has greater potential impact. In such a situation, we asked respondents, what percentage of their available resources would they devote to each fire and why.

Finally, we asked six further questions about respondents’ experiences and preferences in relation to predictive services, asking respondents to rank the relative importance of different types of skills for FBANs, the trustworthiness of different prediction methods, and the characteristics they prefer FBANs to have. At the end of the survey, two questions collected demographic information on the age and gender of respondents.

To test Hypothesis 1, we ran univariate analyses of the responses to Scenario A. Open field answers were coded iteratively in the qualitative analysis platform NVivo 11 using Thematic Codes developed from Neale’s ethnographic fieldwork and survey responses (Cope 2005).⁵ Where quoting directly from these open field answers we have included the respondent’s jurisdiction. To test Hypothesis 2, we used ANOVA tests to see whether respondents with greater seniority and experience respond differently to Scenario B. To test Hypothesis 3, we used ANOVA tests to see whether respondents with greater seniority and experience valued different skills compared with younger and more junior colleagues. To test Hypothesis 4, we used chi-square tests to examine whether there are differences in the proportion of respondents who are more senior and with more years of experience among those who use FBAN services hourly versus the others. To test Hypothesis 5, we used ANOVA tests to see whether respondents who use FBAN services more frequently have different perceptions of their quality, accuracy, clarity and timeliness. Finally, to test Hypothesis 6 we ran univariate analyses of the responses to Scenario C and, again, coded open field answers iteratively in NVivo 11 using developed Thematic Codes. Table 2 summarises our hypotheses, measures and analytical strategies.

Table 2.  Summary of hypotheses, measures and analytical strategies

Asked to assess the appropriateness of the FBAN’s actions in Scenario A, a majority of respondents thought they were not appropriate (60.6%) and a minority thought they were appropriate or very appropriate (39.4%). Amongst respondents who deemed the FBAN’s actions not appropriate, there were consistent themes in their reasoning. Some (n = 12) suggested that ‘any prediction is better than none’ or ‘better than nothing,’ while others stated that in the early period of a fire incident, such as the hypothetical, they primarily wanted FBANs to produce a ‘quick and dirty’ (n = 7) prediction, several ‘scenarios’ (n = 3), or a ‘worst-case scenario’ (n = 5) that could then be refined or corrected over time. Echoing others, one respondent suggested that ‘a correction can be made easily in the mind’ (WA) of a user if an initial prediction proves to be significantly incorrect. Some even expressed a sense of resentment at the hypothetical FBAN, suggesting that they should ‘get with the program!’ (Vic.) and ‘be more agile and responsive to the community’s needs, not the needs of the FBAN’ (Vic.). Amongst the respondents who deemed the FBAN’s actions appropriate, some (n = 3) noted that it can be hard to reset thinking after an initial prediction (or, in their words, ‘bullshit in, bullshit out’ (WA)).

The responses to this scenario are interesting as they do not support the hypothesis, but rather its antithesis. On average, respondents with over 20 years of experience were significantly more likely to release FBAN outputs to the general public (M = 3.68, s.d. = 0.84) than respondents with less than 20 years of experience (M = 3.25, s.d. = 0.70, F (1, 197) 12.87, P < 0.001). Respondents with more senior positions were on average more likely to release FBAN outputs to the general public (M = 3.64, s.d. = 0.80) than less senior colleagues (M = 3.43, s.d. = 0.83), although the difference between the two groups was not statistically significant (F (1, 197) 3.38, P = 0.07).

This third hypothesis was partially supported but not in the ways expected. The ANOVA tests reported in Table 3 show that respondents with less senior roles tended to value FBANs having knowledge of computer programming and the ability to improvise significantly less than those in senior roles. The latter valued FBANs having interpersonal communication skills significantly less than less senior colleagues. This suggests three differences in skills preferences, based on this evidence. Descriptive statistics show that respondents ranked higher for FBANs to have skills in fire behaviour science, operational experience with relevant tools and fireground experience. Other skills such as interpersonal communication skills and the ability to improvise based on intuition also ranked well, followed by computer programming knowledge, the ability to follow official rules and guidelines and ‘other’ all ranking lower on average. The only statistically significant difference between respondents with more and less than 20 years of experience is their perception of interpersonal communication skills, which is more valued by respondents with less than 20 years of experience (P = 0.03).

Table 3.  Differences in perceived value of FBAN skills between respondents with greater and less senior roles: mean (standard deviation), F value and P value

As reported in Table 4, we found that respondents in more senior roles are significantly more likely to use FBAN services hourly or more frequently, whereas less senior roles are more likely to consult them only daily or less frequently (Chi-square (1, 201) = 4.37, P = 0.04). The differences are not significant between respondents with more than 20 years of experience and less than 20 years of experience (P = 0.28).

Table 4.  Frequency of use of FBAN services among greater and less senior respondents: count and row percentages

As shown in Table 5, respondents who consult FBANs and their outputs more frequently have significantly more positive perceptions of the accuracy, quality and clarity of predictive services than people who use them less frequently. Differences in timeliness are not statistically significant.

Table 5.  Differences in perceived accuracy, quality, timeliness and clarity of FBAN services between respondents who use FBAN services hourly or more frequently, and daily or less frequently: mean (standard deviation), F value and P value

The responses to Scenario C show that on average respondents would devote 51.1% of their resources following the FBAN advice, and 48.9% of their resources following the local fire manager. Amongst respondents who favoured an even split of resources, some insisted that they wanted ‘further’ or ‘more’ information (n = 15), while others suggested that it was best to ‘hedge’ or split resources ‘50/50’ (n = 17) until further intelligence on the fires was available or, alternatively, the FBAN and local fire manager were able to discuss their analyses. Other answers suggested that respondents’ views depended on their opinions of the particular individuals involved as, some suggested, ‘local knowledge is always important but can be biased’ (WA) whereas, to quote another, ‘I have a small crew of trusted FBANs I would back 100% but when they are not known to me, I weigh up all options’ (Vic.).

There were also common themes amongst respondents who favoured either the local fire manager or FBAN in this scenario. Those who put more resources towards Fire X tended to affirm the value of local knowledge, stating it ‘should never be underestimated’ (SA) or ‘cannot be discounted’ (NSW), and many seemed to frame FBAN analysis as an outcome of tools rather than practitioners. This was illustrated by the multiple responses that the knowledge of a local fire manager is ‘more accurate than a computer modelling product’ (Vic.), will ‘always be more informed than remote data sources’ (NSW), and should be preferred ‘over data in a computer’ (NSW) or ‘over modelled scenarios’ (Vic.).

Those who put more resources towards Fire Y often explained the FBAN as more reliable due to having more data and objectivity, stating that the local fire manager ‘may not be across the current weather forecast’ (NSW), ‘will not know the full picture’ (ACT), and ‘may be parochial or biased’ (Vic.). The FBAN will be ‘less influenced by other biases’ (WA), ‘have a better understanding of the bigger picture’ (Tas.), and make ‘more of a science-based conclusion’ (NSW). Interestingly, one respondent suggested that they would favour the FBAN’s analysis because this would represent ‘decision making that would stand up to scrutiny’ in a post-event inquiry (NSW, see Lawson et al. 2017).

Overall, it appears that respondents are quite satisfied with the predictive services that are currently available to them, agreeing or strongly agreeing that the predictive services they receive meet their needs in terms of accuracy (77%), quality (83%), timeliness (63%) and clarity (76%). Many seem to want the service expanded, as a majority (54.27%) responded that they would prefer Incident Management Teams to always be staffed with an FBAN, a level of resourcing that is not officially recommended or required in any jurisdiction.

Further, when asked to rank the five means through which they gain access to predictive services, the top-ranked option amongst respondents was to consult an FBAN in person or via telephone (M = 1.53) rather than consult printed fire spread prediction maps (M = 2.54) or teleconference briefings (M = 3.44). When asked to rank the type of qualities they prefer FBANs to have, academic expertise in fire science (M = 2.0), significant firefighting experience (M = 2.2), and personal familiarity with the user (M = 2.3) ranked both highest and very closely, compared with whether FBANs followed rules and guidelines (M = 3.4).

The results of our survey have created new questions for further research while also refining our understanding of predictive services users in the surveyed jurisdictions. As might be expected, it appears that these users often want a prediction when a fire incident is first reported and are, thereby, willing to forego some level of accuracy and certainty in order to have this intelligence available to them. Such predictions are one source of uncertain intelligence amongst others that users such as incident controllers and operations officers consult in the first hour of an incident. Interestingly, there are several signs in our results that FBANs and their analyses carry substantial esteem with their users, notably in respondents’ positive views of the accuracy, quality, timeliness and clarity of predictive services and the apparent trust many placed in the advice of an FBAN positioned away from a fireground (see ‘Hypothesis 6’). That said, there are also grounds to suggest that users place even greater trust in FBANs who have personal experience of firegrounds and that users have personally interacted with (see ‘Hypothesis 3’ and ‘Other results’).

This evidence, we propose, comes into sharper focus when considered in light of users’ negative views of predictive services, as well as the preferences expressed by veteran users and more senior users. Amongst those respondents who preferred the local fire manager’s advice in Scenario C, many expressed a view of predictive services as the outcome of distant and impersonal analysis or, to quote one respondent, just ‘data in a computer.’ Such user perceptions are a poor reflection of actual FBAN practice – which involves significant analysis and synthesis from diverse sources of partial information, including detailed fireground intelligence, to produce a fire spread map and other predictive outputs (Plucinski et al. 2017; Neale and May 2020) – and may be related to low levels of familiarity between FBANs and these users or a lack of active communication by FBANs about their skills and expertise. In parallel, correlations amongst veteran user respondents between frequent interactions and positive perceptions suggest a possible causal relationship, also implied by other studies (e.g. Rapp et al. 2020), where interactions with FBANs increase familiarity and trust in predictive services. Further research into the drivers of user trust and distrust in FBANs is warranted; however, the current typical modes and means of delivery of predictive services do not seem to be well aligned with user preferences, with FBANs often working from centralised control centres, with limited opportunities to visit firegrounds or interact with users outside these contexts.

Our results also provide inspiration for further research when considered in light of two recent trends within the Australian fire management sector. The first of these is the sector’s increased investment in digital tools and systems to deliver forms of operational intelligence, including predictive analysis, as well as the growing use of automation and machine learning in these contexts (e.g. Anonymous 2021). The proliferation of such tools and systems, and increases in users’ dependency on them, may lead to even less interpersonal familiarity between FBANs, whose analyses are inputs, and users, whose decisions are informed by their outputs. Our findings suggest that fire agencies should seek to better understand not only how such tools and systems affect relationships between FBANs and users but also how increased understanding of the social dynamics of utilisation can support learning and adaptive capacity in operational response (see Pahl-Wostl et al. 2013). We cannot say whether greater use of predictive services leads to better decision-making, but we can say that interpersonal trust is, amongst other factors, important to greater use and operational effectiveness (McLennan et al. 2006; Thompson 2014).

The second trend relates to the rising instances of fire agencies publicly releasing predictive maps during fire events, including on social media, as occurred in NSW and the ACT during the 2019–2020 bushfire season (e.g. Whittaker et al. 2021: 8–12). These maps typically have different designs and spatial and temporal scope to those used within incident management but nonetheless originate in predictive services units and their analyses.⁶ Respondents did not appear to be universally enthusiastic about such release strategies; however, there is no evidence from our survey that more senior or experienced users are more conservative or cautious on such matters. The latter seemed more amenable than others did to sharing information with affected communities, particularly in organised briefings. Considering that, supported by aligned recommendations from post-event inquiries (e.g. Owens and O’Kane 2020: xix; SAIR 2020: 92), more fire agencies appear to be publishing both incident warnings and predictive services outputs during fire incidents, it is vital that more research effort is directed towards understanding how established (e.g. fire managers) and emerging (e.g. relevant publics) ‘predictive services users’ would each prefer to see these outputs developed and disseminated. It is also important that the evidence of this study is tested against user views in subsequent years given that the 2019–2020 bushfire season was a landmark season in several Australian jurisdictions.

As Pacheco et al. (2015) conclude, the path to implementation in fire management is rarely clear and requires a robust understanding of the relevant social context: the people and institutions. This study, like others before it, provides further evidence to undermine ‘linear’ models of innovation in fire management, that is, models that assume that new systems, tools or even roles will be embraced by fire managers and fire management agencies because they are scientifically-supported and available (see Godin 2006). Examining the growing uptake in Australia of FBANs and their predictive outputs, we argue both that user perceptions shape how (and whether) these predictive practitioners and their analysis are utilised and, more specifically, that there are patterned relationships between user demographics and utilisation. These patterns require extensive empirical research, though our survey results indicate not only that most users would like more access to FBANs but also that through personal interaction trust can be progressively built between analysts and users and outputs can be honed to users’ actual needs. A trusted FBAN, from the perspective of this study, is one who regularly meets users and visits firegrounds. If investment in predictive services in Australia continues apace, it is therefore important that support is not simply devoted to tools and systems but, rather, fosters social connection between FBANs, their tools and their existing and emerging users.

The data collected in this study have not been published due to the conditions of its institutional human research ethics approval.

The authors declare no conflicts of interest.

This research was supported by the Australian Government through the Australian Research Council’s Discovery Early Career Researcher Award funding scheme (DE190100233, ‘Pyrosecurity: understanding and managing bushfires in a changing climate’).