An evaluation of wildland fire simulators used operationally in Australia

Introduction

Fire is integral to the Australian landscape (Russell-Smith et al. 2007) but poses a hazard to life and property (Scott et al. 2013; Doerr and Santín 2016; EM-DAT 2018). Each year, fire burns part of the Australian landscape (Carmona-Moreno et al. 2005; Giglio et al. 2013), and occasional fire disasters have caused widespread destruction and many deaths (e.g. Stretton 1939; Cheney 1976; Brotak 1980; Miller et al. 1984; Mills 2005; Teague et al. 2010; Owens and O’Kane 2020; Tomašević et al. 2022). Fire and land management agencies attempt to manage fire in the landscape. Over the last two decades in particular, fire behaviour simulators, which model the spread and some other properties of fires, have become a tool to predict fire behaviour for that management (Finney 2004; Opperman et al. 2006; Haas et al. 2013). Simulator applications include estimating the likelihood of fire ignitions becoming difficult to control (Finney et al. 2011), predicting direction and rate of growth of going fires (Tymstra et al. 2010), assessing the impact of fuel reduction strategies on future fires (Ager et al. 2010; State Fire Management Council 2014) and identifying safe evacuation routes from wildfires (Ozaki et al. 2019). Owing to their quite recent introduction, wildland fire simulators lack standardisation, and new simulator versions have sometimes been introduced to operations without clear and comprehensive assessment of their suitability. Several simulators, and sometimes different versions of the same simulator, are used in wildfire management in Australia, often within one agency. Fire agencies have become concerned that they do not know whether the best simulator is used for any given application (Cruz et al. 2014).

Simulator evaluations have been reported from fire-prone regions globally. These studies, although often including multiple case studies, have tended to focus on individual simulators (e.g. Arca et al. 2007; Duff et al. 2013; Kelso et al. 2015; Miller et al. 2015; Giannaros et al. 2020), though there are exceptions. Opperman et al. (2006) examined simulators then available for their applicability in Australia and New Zealand, and Duff et al. (2018) compared versions of the Phoenix simulator. Some simulator evaluations have incorporated investigation of simulator input uncertainty (e.g. Bachmann and Allgöwer 2002; Fujioka 2002; Finney et al. 2011; Benali et al. 2016; Allaire et al. 2020, 2022; DeCastro et al. 2022) and variability (e.g. Hilton et al. 2015), whereas Cruz and Alexander (2013) dealt with modelling uncertainty in the fire behaviour models that underpin simulators. Pinto et al. (2016) used an ensemble of FARSITE instances to represent input uncertainty when generating probabilistic fire spread predictions for a fire in Portugal. Plucinski et al. (2017) presented a package, Amicus, aiming to standardise fire behaviour prediction by including an understanding of simulator limitations and expert knowledge. They also discussed a framework for uncertainty analysis. Penman et al. (2020) focussed on the impact of weather forecast error on two simulators, Phoenix and Spark, in order to avoid the potential for model bias to influence their results. The authors found considerable sensitivity on the part of simulators to variability in input weather parameters.

Approaches to simulator evaluation have used numerous performance metrics, some of which we note here. Filippi et al. (2014) addressed previous approaches to evaluation, including the Sørenson similarity index, Jaccard’s coefficient and Kappa statistics, but also advancing Arrival Time and Shape Agreement, in an effort to accommodate imprecision in underlying validation data. Duff et al. (2016) assessed Shape Deviation Index and Area Difference Index, as well as Jaccard’s coefficient and Sørenson similarity.

Efforts to formalise evaluation of fire simulation systems extend back at least to Rothermel and Rinehart (1983) for the Rothermel (1983) wildland fire spread model. Filippi et al. (2014) introduced a formal evaluation protocol that targeted dynamic properties of fires and fire simulations but noted that no single measure of simulator performance was likely to comprehensively capture simulator error characteristics. Recently, again, Duff et al. (2018) proposed an approach to evaluate relative performance of simulator versions using Area Difference Index.

To our knowledge, there has not been an attempt to date to evaluate multiple simulators for a large variety of case studies, taking uncertainty in input variables into account and visualising the results for different end-users. Here, we present a new evaluation that achieves those aims, using a framework that caters for a wide range of test cases and visualises simulator performance using methods new to this field. By ‘framework’, we mean a suite of software that automates the evaluation of fire simulators against a defined set of documented fire events. The framework acknowledges that different user groups may have different decision requirements in any evaluation process. Such users will include high level fire agency managers looking to improve agency predictive capability, fire behaviour analysts attempting to obtain the best possible simulations of fire behaviour during individual incidents, developers attempting to enhance simulator performance by improved algorithms and potentially other groups. Thus, a range of simulator performance metrics were used to accommodate the range of user requirements.

Such evaluation is important for fire authorities to understand the effectiveness of predictive tools at their disposal, and the circumstances in which simulators can be expected to perform well or poorly. Simulator evaluation, and documentation of the evaluation process, is important not only for fire behaviour analysts using simulators as a routine part of their work, but for senior fire agency managers making decisions about which simulators their organisations will use or support in development. It is also important for simulator developers to assist them in identifying the extent to which changes in simulator design improve on earlier efforts.

As noted, the framework developed to carry out evaluations consists of software for organising the execution of simulators in a manner mimicking operational use, including presenting them with input data with uncertainty estimates to assess the degree of simulator sensitivity to sometimes uncertain real-world input data, and computing metrics of simulator performance against fire spread observations. Finally, investigation of (relative) simulator performance was also identified as an important consideration for users. Plots were developed to assist in the analysis of evaluation metrics and to cater for different user needs, as noted above.

The work described in this paper was commissioned by the Australasian Fire and Emergency Service Authorities Council, AFAC, with the intention of better understanding the accuracy and sensitivities of fire simulators used in Australia at the time. It was also anticipated that the work undertaken would provide a mechanism for assessing fire simulators and simulator versions developed in the future.

AFAC requested the Australian Bureau of Meteorology undertake the assessment of fire simulator performance. The Bureau has a long involvement in fire weather forecasting and has worked closely with fire managers over many decades. In addition, fire managers wanted to make use of the Bureau’s extensive experience in running models of physical systems – albeit weather and environmental systems rather than fire, largely – and in verifying the performance of those models.

An initial workshop involving researchers and senior operational managers from fire and land management agencies around Australia established key requirements that users wanted addressed concerning simulator performance. The workshop also agreed on a broad approach to simulator assessment, and identified a number of test cases of fire events against which simulator performance would be evaluated.

Ten historical fires were used to test the performance of four simulators used in Australia: Australis (Johnston et al. 2008), Phoenix (Tolhurst et al. 2008), Prometheus (Tymstra et al. 2010) and the Spark simulator framework (Hilton et al. 2015).

Details of the fire behaviour models and fire propagation methods used in each simulator are provided in Table 1. Description of the implementation is available in Bureau of Meteorology (2017).

This paper focuses on describing the results obtained in one case study event, and illustrates the application of the evaluation framework with example data and plots from that case study. Also, we offer some observations on the effectiveness of the framework. Full results of the evaluation are available in Bureau of Meteorology (2017), together with greater detail of the evaluation framework.

Evaluation method

As noted above, at the commencement of the project, senior operational fire managers from most Australian state and territory fire authorities attended a workshop with project staff to discuss project implementation. Participants agreed that evaluation of a range of test cases was an appropriate way of assessing simulator performance. Fire agency staff selected 10 historical fire events for analysis, across a range of climates and vegetation types, in which fire spread was not suppressed during the analysis period. The latter constraint was added to ensure that the simulators were assessed simply and fairly, i.e. without having to engage suppression sub-models that may have confounded the analysis of their basic operation. Case study baseline conditions are summarised in Table 2. Limited availability of observed fire behaviour data constrained the number of possible case studies. To approximate conditions agreed in the initial workshop, a fairly unsuppressed fire run of 6–12 h was required, from either a known ignition point or perimeter, along with the corresponding final fire perimeter for the simulation period. For each case study, fire agencies supplied data layers required to run simulators, including fuel and topography, as well as fire observations. Fire agency staff were also asked during the initial project scoping workshop which aspects of fire behaviour simulators were important to them. Their answers informed the development of the simulator evaluation framework. The fire agency staff required knowledge on:

Simulator input sensitivity is important for several reasons. Most data available from firegrounds are necessarily imprecise and sparse and may not reflect conditions across an entire active fire. It is important for users to understand how such imprecision and inaccuracy may affect simulator behaviour, and whether, for example, it might be advisable, or even necessary, to run several simulator instances to capture output uncertainty, given known uncertainties in input parameters. Note that such simulator sensitivity may be appropriate if fire behaviour is similarly sensitive to those inputs. These requirements dictated the specific methods used to investigate simulator performance. Archived official weather forecast grids from the Australian Digital Forecast Database (ADFD, Bureau of Meteorology 2015) were used as weather inputs for all cases. ADFD grids are produced from numerical weather predictions edited by forecasters and are available throughout Australia, providing a representation of average weather parameters over 6 km grid cells (3 km over Victoria and Tasmania) at the times of the case studies. To produce hourly input weather time series, ADFD grids were sampled at the ignition point or, if a case involved prediction from an existing fire, the centroid of the observed fire perimeter. This was undertaken to provide the most representative weather for the initial fire point or perimeter. The derived weather stream dataset for each case study comprised on-the-hour forecast values of temperature, relative humidity, wind speed and wind direction, together with 3-hourly drought factor (DF, an indication of the proportion of forest fuel available to burn; McArthur 1967) and weekly updated observations of grassland curing, depending on the nature of the vegetation burnt. Data from the three nearest automatic weather stations (AWS) were compared against ADFD grids to estimate weather parameter uncertainty.

Simulators were tested using data from the selected events, using baseline data provided by agencies; no attempt was made to tune results by modifying local fuel or weather inputs as would usually occur when using simulators operationally, except in one case study to explore the effects of changing clearly incorrect forecast weather. Where included, suppression modules were deactivated.

This approach was used to best represent the real-world usage of simulators. Users wanted, for example, to know which simulator(s) best reproduce fire behaviour, irrespective of inputs that may not be completely accurate. To have provided ‘perfect’ inputs would not have achieved this aim, even though it would have isolated potential model errors from the possibility of input errors affecting simulator outputs.

Input variables were perturbed independently to explore sensitivity of simulators to each input. Simulator outputs were compared against available fire observations using evaluation metrics that relate to spatial verification of the fire boundary, targeting area, bearing and distance of forward spread, and overlap between observed and simulated burnt area. Uncertainties in fuel type, fuel load and weather were quantified using a data reliability assessment defined by Cruz et al. (2012).

We estimated uncertainties in weather data from weather forecast errors determined around the time and place of each case study, using AWS observations as noted above. This local ‘error climatology’ approach matched ADFD grid cells with AWS observations (Bridge 2015) in the vicinity of each case study (e.g. Fig. 1) to create error histograms from which perturbations were determined (Pinto et al. 2016).

Fig. 1.

(a) The Blue Mountains, New South Wales, showing the best-guess ignition location for the State Mine fire and the location of three AWSs. (b) ADFD temperature forecast ‘error climatology’ derived from AWSs in the vicinity of the State Mine fire of October 2013.

Perturbations were also applied to baseline fuel load for each case study using linear sampling in the range of ±20%, based on NSW RFS estimation. Cloud, DF and curing perturbations were made by linearly sampling in the range ±10%, based on fire weather meteorologist estimation. Ignition time perturbations were made by linearly sampling within a ±30 min window of best guess ignition time, based on NSW RFS estimation. Ignition location perturbations were made by randomly sampling within a 200 m radius of the ‘best guess’ ignition location; this range was determined arbitrarily. Some 19 perturbations were applied, yielding 20 input values, including the baseline value, for each input variable. In our application of the framework, Spark and Phoenix simulators were run 201 times for each case study (20 simulations for each of the 10 input variables perturbed plus a baseline simulation) but Prometheus and Australis simulations had to be performed manually, resulting in fewer perturbed runs (16 for each case study for Australis, 16 for one – Tasmanian – case study for Prometheus). Also, fewer input variables could feasibly be perturbed for those simulators. The alternative was not to include Australis and Prometheus in the evaluation, which was not desirable.

Details of the computing infrastructure built to run the evaluation framework are available in the project final report (Bureau of Meteorology 2017). Briefly, the evaluation was divided into four general reproducible workflows to be applied for each experiment: input data preparation, comparison data (observation) normalisation, simulator execution and summarisation.

Two simulators (Phoenix and Spark) were run automatically using the Simulator execution workflow. Prometheus (in the Tasmanian case, where fuel state data were available) and Australis were run manually outside of the Simulator execution workflow, as noted, with outputs returned and ingested into the summarisation workflow to allow evaluation using standard metrics.

Simulators ran using the standardised inputs and produced simulated fire boundaries. Metrics were generated to assess simulator accuracy and sensitivity. Parameters considered in the sensitivity analysis included wind speed, wind direction, temperature, humidity, ignition location, ignition time, DF, curing, cloud and fuel layer, where simulators permitted varying these parameters.

Given user interest expressed in the initial workshop in accurate prediction of fire spread, bearing and area burnt, we adopted four metrics in the evaluation framework. Jaccard’s coefficient is commonly known as the threat score (TS) in weather warning evaluation and is defined as:

where H (hits) is the area correctly identified as burnt, FA (false alarms) the area incorrectly identified as burnt and M (misses) the area not predicted to burn but that was burnt (Fig. 2). A good TS is closer to 1 whereas a poor TS is near 0.

Fig. 2.

Schematic diagram showing how the evaluation metrics are calculated. The diagram depicts an idealised observed (red) and simulated (blue) fire. A common ignition location is indicated by the black cross. Overlapping (H, hits) and non-overlapping (M, misses; FA, false alarms) components of the simulated and observed fires are used for calculation of the threat score. Bearing of the simulated and observed forward spread of the fire is determined using a binned circular weighting approach (described in the text). Bearing error is the difference between simulated bearing bin (blue sector) and observed bearing bin (red sector). Forward spread error is determined from the difference between the observed and simulated maximum fire extent in the direction of the observed and simulated bearing bins, respectively. Burnt area error is the difference between the total burnt area within the simulated fire perimeters and the total burnt area within the observed fire perimeter.

The bearing error is computed as the simulated bearing minus observed head fire direction (e.g. Duff et al. 2012). The range is [0, 180] degrees with a good bearing error being closer to zero. Bearing error may point to errors in simulator inputs including wind direction, fuel mapping, topography or disruptions.

The average forward rate of head fire spread is given by the distance travelled by the head fire during a defined time interval. Various metrics have been employed to measure the difference between simulated and observed rates of spread (e.g. Cruz and Alexander 2013; Cruz et al. 2015). Here, we use forward spread error. This is the difference between the simulated and observed head fire spread – a distance rather than a speed. Converting between these two metrics is trivial as they differ only by a scaling factor. Forward spread errors can be expected to relate to the one-dimensional fire models underpinning the simulators and the weather and fuel inputs that drive them. However, unless the fuel and topography in the fire landscape are homogeneous, this depends on an accurate fire bearing. A good forward (rate of) spread error is close to zero.

The burnt area error is simulated minus observed total burnt area (compared graphically by Cui and Perera 2010). Positive (negative) burnt area errors represent over (under)-prediction. Small values are desirable. Absolute values of error can be important operationally because they correlate to damage, CO₂ emissions and suppression effort. However, absolute errors may be misleading when comparing fires that differ greatly in size. An alternative, relative error measure is the ratio of simulated to observed total burnt area. However, a 100% overprediction has different operational implications if the real fire is 3 ha or 3000 ha. The choice of metric depends on user need. Burnt area errors can arise from underlying model errors, and input fuel- and weather-related errors. These evaluation metrics are summarised for convenience in Table 3.

Discriminating between false alarms and misses, and hence information on simulator over- and under-prediction, can be achieved with two auxiliary measures:

These are, respectively, identical to the definitions of ‘recall’ and ‘precision’ in Duff et al. (2016). In addition, SR is equal to 1– FA. TS, PoD and SR can be represented on a categorical performance diagram, described later. Of these metrics, TS and SR penalise false alarms, TS and PoD penalise misses, and all are insensitive to correct negatives. This is desirable, as the metrics are then independent of fire domain area, which can be defined arbitrarily. This problem besets Cohen’s kappa coefficient (Finney 2000; Filippi et al. 2014; Duff et al. 2016).

High-level summaries of relative simulator accuracy for the baseline runs are obtained by comparing outputs from each metric using a technique developed for evaluation of climate models (Gleckler et al. 2008). Simulator performance relative to other simulators is colour-coded so that poor performance of a simulator is displayed in these ‘Gleckler plots’ as progressively deeper shades of pink as performance deteriorates and progressively deeper shades of green as performance improves. White denotes performance close to the median of simulator performance.

Results

First, we present results for one case study, the State Mine Fire (NSW), to illustrate evaluation outputs and demonstrate how these metrics are used to address the user requirements identified in the Evaluation Methods section.

Of the 10 case studies in the evaluation project, the State Mine Fire was chosen because quite frequent fire boundary observations were available. This was not always the case with other fire case study events owing to firefighting resource limitations. Additionally, simulator performance was broadly similar for the State Mine Fire. Across all cases studies, no simulator performed markedly better than others but, in some cases, one or other simulator appeared superior on account of the specific details of the case. We wanted to avoid any perceptions that one simulator was clearly superior to others in the case study we presented. We also present summary results and plots for Spark and Phoenix simulators to illustrate the capacity of the evaluation framework to provide an overview and intercomparison of simulator performance. Comprehensive results and interpretation for all case studies are available in Bureau of Meteorology (2017).

Figs 3–5 address the user requirements for better understanding of simulator relative performance and accuracy. Baseline results for the State Mine Fire case for each simulator (Fig. 3, yellow) compared with the observed fire boundary at the same time (orange) permit subjective visual assessment of baseline simulation results, providing users with an indication of relative simulator performance and accuracy compared with observed conditions. Rate of spread is overestimated by all simulators, and forecast bearing differs from observed fire behaviour for most simulators, resulting in little overlap. The uniform overestimation of rate of spread suggests that the input weather forecast data were inaccurate. This is perhaps unsurprising. Forecast data were obtained from routine grids at 6 km horizontal resolution and the fire occurred in complex terrain. Such routine data were used to ensure consistency between case studies, as noted above. Also of interest is the fire aspect ratio, with a range of length to width ratios between simulators.

Fig. 3.

Baseline simulations for the State Mine Fire, showing the predicted fire perimeter from different simulators in yellow and the observed fire perimeter in orange.

Fig. 4.

Example relative comparisons for the State Mine baseline fire case study for (a) Threat score, and (b) Bearing error, showing the performance of each simulator coloured by the distance from median simulator performance (in white). Green (pink) colours indicate relatively better (worse) accuracy in comparison with other simulators. Absolute metric values for the worst, median and best-performing simulator are also provided in inset text in the colour legends.

Fig. 5.

Categorical performance diagram for the State Mine fire case study, providing a direct visual comparison of simulator performance. Dashed lines show constant values of Threat Score, with better performance towards the top right corner. Overprediction of fire spread appears above the diagonal and underpredicted results appear below the diagonal.

Fig. 4 displays a Gleckler plot for the State Mine fire simulations, specifically addressing the user question of which simulator performs best relative to others. Phoenix versions show similar performance for TS while Australis has the best TS (Fig. 4a), in part because the total area burnt is similar between modelled and observed cases (Fig. 3), but potentially also because the Australis version tested had wind directions fixed to compass points. However, the Bearing Error for Australis is relatively poorer than that of Spark and later Phoenix versions.

A categorical performance diagram (Roebber 2009) displays absolute, rather than relative, TS values (Fig. 5), providing users with an overview of objective simulator performance against observed fire behaviour, in addition to an indication of relative performance. The vertical axis shows PoD, the horizontal axis shows SR and dashed lines the TS, with the latter increasing towards the top right corner of the diagram. Overprediction of fire spread appears above the diagonal and underpredicted results appear below the diagonal. The superior TS of Australis in this example is clear, as is the substantial area error above the diagonal representing overprediction. The high PoD but relatively lower SR for all other simulators objectively highlight the false alarms evident from Fig. 3.

To address the third user requirements specified in the evaluation Methods section, better understanding of simulator sensitivity to variability and uncertainty in input parameters, a visual assessment can again be used to explore sensitivity across all input variables amenable to perturbation. For example, maps for each simulator in Fig. 6 display the effect of varying wind direction within the estimated error distributions for ADFD weather forecast grids, based on the error climatologies derived using data from nearby AWS. Again, baseline results for each simulator are in yellow, compared with the observed fire boundary at the final time step shown in orange. The area burnt in the simulations by 1, 10 and 19 or more of the 20 simulations with perturbed wind direction inputs is indicated by the dotted, dashed-dotted and solid thick black boundaries, respectively. These results show high sensitivity to uncertainty in the wind direction, giving a wide range of forward spread bearings. For most tested simulators, however, the solid black line includes most of the observed fire boundary; thus, 19 of the 20 perturbed simulations included most of the observed burnt area. Note, again, that the requirement to manually input data into Australis did not permit its evaluation in this fashion. Its baseline results are included in Fig. 6 for reference.

Fig. 6.

As in Fig. 3 but also showing effect of perturbing the wind direction input to each simulator. The simulated area burnt by at least 1, 10 and 19 of the 20 simulations with different wind direction inputs is indicated by the dotted, dashed-dotted and solid thick black lines, respectively. No wind perturbations were applied to the Australis simulator.

These sensitivity results can also be viewed for each input variable in terms of relative simulator performance. Display of relative performance specifically addresses user requirements to identify the best simulator for any case study. Fig. 7 shows the relative inter-comparison of simulators by varying input parameters, using a modified Hinton diagram. Box size indicates the level of sensitivity of the simulator output for each variable. Colours again identify the relative accuracy of simulators. Here, the median metric value of the perturbed simulations for each simulator is compared with the median metric value for the perturbed simulations for all simulators. White indicates a near-median result, pink indicates results worse than 25th percentile and green indicates a simulator performed better than 75th percentile of assessed simulators.

Fig. 7.

Modified Hinton diagram for the State Mine Fire case study, showing the sensitivity and accuracy of simulators to variations of input parameters. The box size indicates the sensitivity of the Threat Score to perturbations in corresponding parameters. Green (pink) colours indicate relatively better (worse) accuracy in comparison with other simulators. Absolute metric values for the worst, median and best-performing simulator are also provided in parentheses.

Sensitivity results for Australis should be interpreted with care. Instance simulations were run manually, then inserted into the evaluation with a smaller sample size and fewer variables, as noted. Manual operation was required because Australis codebase was not available for automation. Although far fewer instances (17) could be run than for Phoenix and Spark versions (201 each), it was considered valuable to include Australis results here to highlight how simulator differences affected the operation of this evaluation framework and how it was sufficiently flexible to include diverse simulators, and to provide an indication of the relative performance of simulators used operationally in Australia.

Absolute sensitivity performance can also be visualised using the categorical performance diagram shown in Fig. 5. Given the larger number of runs when including the perturbations for each input variable, there are more results to display, often resulting in clusters or ‘snakes’ of points for each simulator. Although users did not specifically request an indication of absolute performance in the evaluation of simulators, as noted above, this type of evaluation permits an assessment of the degree to which simulators differ in their performance, as well as their departure from a perfect (i.e. identical to that observed) output. Fig. 8 shows an example for perturbations to wind direction input, again for the State Mine Fire.

Fig. 8.

Performance diagram for the State Mine fire for all simulators including perturbed wind direction inputs as well as the baseline input.

To provide a measure of overall simulator performance, i.e. across all case studies, metrics must be aggregated. We explored further extending the performance diagram. Whereas Fig. 8 shows, for all simulators, the results of simulations from one case study and perturbing one input variable (wind direction), Fig. 9 shows for Spark (left) and Phoenix (right) simulators the TS values for all simulations for all 10 case studies and all 10 perturbed input variables. In contrast to the performance diagram in Fig. 8 where data are represented as points, in Fig. 9, the larger number of data points are instead summarised by allocating them to bins. Interpretation of the diagram remains the same insofar as, ideally, most simulations would fall in the upper right bin, and a tendency to over (under)-predict fire spread results in high relative frequencies above (below) the diagonal.

Fig. 9.

A visual meta-analysis approach based on the performance diagram. The two diagonal lines enclose the ±50% over- and under-prediction region, i.e. where prediction bias is more acceptable. Left: all Spark simulators, all case studies. Right: all Phoenix simulators, all case studies. Darker (lighter) colours show higher (lower) proportions of the simulations.

The above results indicate that different simulators, and different versions of the same simulator, perform better than others as conditions vary. This result carried through across case studies, with no simulator performing well against all, or even most, case studies. Given the varied and extreme nature of available case studies, and the small number available for this study, it is not possible to draw conclusions regarding simulator accuracy for specific fire types or weather conditions from this evaluation.

Discussion and summary

Evaluation is important both in absolute terms, for users to know the extent to which simulators can be expected to reproduce observed fire conditions, and in relative terms, to enable assessments of whether and by how much successive simulator versions improve over their predecessors. Our evaluation, described fully in Bureau of Meteorology (2017), highlights substantial differences in performance between simulator versions. It is also valuable to know how well simulators compare with other simulator implementations. The evaluation framework presented in association with case study results provides a tool for testing and verification of new simulator releases to assess operational impacts and risks as well as improvements.

We describe results of one case study, the State Mine Fire in NSW in October 2013. We also, however, demonstrate the information that can be gained from aggregating results of multiple case studies (Fig. 9). For the 10 cases we examined, no one simulator was universally superior to the others. The framework designed for this evaluation is sufficiently flexible to accommodate available case study verification data from a wide variety of perspectives. Such flexibility is valuable to account for differing requirements for simulator evaluation from a variety of user groups, and from the same users over time as their needs change and as simulators develop.

A set of spatial metrics based on simulated and observed fire perimeters was selected to answer the questions related to fire spread, bearing and area. These consisted of a summary metric, threat score and three diagnostic metrics (head fire spread error, head fire bearing area and burnt area error) enabling more detailed characterisation of simulation errors. Our choice of metrics is not prescriptive or comprehensive – additional metrics have been defined and used in the literature (e.g. Filippi et al. 2014; Kelso et al. 2015; Duff et al. 2016). Threat Score does not, for example, explicitly distinguish misses and false alarms. Were this a primary evaluation requirement, an alternative might be a combination of ‘probability of detection’ and ‘success ratio’ that form the axes of the categorical performance diagram (e.g. Fig. 8).

Also important was the gathering of input data (weather, fuel, topography) required to run simulators and observations of fire behaviour from historical wildland fires. Ideally, many wildfire cases would be available for this purpose. In reality, this is generally not the case. Although indicative, our dataset of 10 fire case studies was inadequate for drawing robust conclusions about simulator performance. High-quality field observations required for simulator evaluation are currently limited in type and quantity (Kelso et al. 2015; Filkov et al. 2018). Our observations were exclusively snapshots in time of fire perimeters obtained from fire agencies.

Uncertainty estimation for simulator inputs guides confidence in the results. We paid particular attention to uncertainty in weather data. The uncertainty estimates applied here and the one-at-a-time approach to sensitivity analysis are one way of catering for simulator input uncertainty. An alternative might employ a high-resolution weather ensemble forecasting system to provide a more dynamic and internally consistent picture of weather uncertainty. Such an approach may be favoured by fire behaviour analysts to estimate the probability of impact (e.g. Louis and Matthews 2015; Miller et al. 2015) and better inform risk-based decisions. The effect of (knowledge about) uncertainty in other inputs is also important. In particular, fuels may not be well characterised and variation in fuel between modelled and actual landscapes may substantially affect simulator outputs, as will variations in fuel load between modelled and actual fires, even if the fuel is well specified. We did not attempt to characterise errors arising from this source, and merely note that it is an additional contributing factor to background error in simulations.

Simulators were run for the historical fires and evaluation metrics calculated by comparing simulator predictions against observed fire behaviour. Our software of the framework developed for this purpose was implemented in the Python programming language with repeatability, reuse, flexibility and extensibility as key features. The intent of the framework development was that it would be available for simulator testing and comparison by the fire management community in Australia.

Visualisation of simulator evaluation metrics for analysis was the final component in the developed framework. We used several novel approaches to present results, including techniques drawn from weather and climate science, to address aspects of simulator performance important to a range of simulator users. Simple shaded plots provided a summary view in relative terms while performance diagrams provided greater detail. A modified Hinton diagram showed sensitivity to various perturbed simulator input variables and could be used as a means of comparing simulator performance across all case studies. As noted above, sensitivity to input parameters, or lack of it, is not in itself a useful characteristic of a simulator, unless actual fires are similarly sensitive. As users indicated initially, though, it is important to know whether a simulator is sensitive to sometimes imprecise or potentially inaccurate input data.

The current set of metrics and case studies formed an initial test bed for the fire research community, representing an example of a framework that could in future be routinely used to evaluate fire simulators prior to operational implementation. The metrics used address the questions about simulator performance initially posed by users, but other questions may arise over time, or on inspection of these and other case studies. In particular, users may wish to further investigate what constitutes a useful or good simulation. The currently used set of metrics provide simple, robust measures of aspects of simulator performance, such as the degree to which simulators burn the correct area, and simulate the rate at which fires progress. However, these measures were not able – using the available case study data – to distinguish whether any one simulator was superior to others. There may be other metrics that could be used in this framework to provide more nuanced information about simulator performance now or in future. In addition, it may be of value to weight particular measures more highly than others for some applications. Of course, a greater range of case study events would also be helpful in this respect.

Effort is required to collect routine measurements of fire behaviour to create a larger set of case studies to support future simulator development, testing and ongoing verification across a range of fire conditions and scenarios. This will permit more robust conclusions to be made and stratification of results by factors including fire danger rating or fuel type (Filkov et al. 2018). Extending observations to parameters including flame height and spotting distance would enable additional verification to be done using relevant metrics.

Ideally, all tested simulators should be subject to identical perturbations in this framework. We were unable to achieve this, as some tested simulators (Prometheus, Australis) could not be automated, and were therefore not subject to as extensive a range of testing as the remaining simulators.

Although not identified as key questions or outcomes in this evaluation project, establishing performance criteria and defining national data standards for simulator inputs and fire observation data would clearly be highly beneficial for future simulator development. As evaluation becomes more standard in simulator implementation, it is likely that the sorts of questions users ask of the evaluation process will become more focussed on particular aspects of performance, leading to further improvement, analogous to the development and refinement of numerical weather prediction in recent decades.