Live trial performance of the Australian Fire Danger Rating System – Research Prototype^†

Research prototype specifications

Eight operationally available fire spread models were selected as the core for the AFDRS_RP, as described in detail by Kenny et al. (2024). For convenience, they are listed here: (1) Forest; (2) Grassland; (3) Northern Grassland (Savanna); (4) Spinifex; (5) Mallee-Heath; (6) Shrubland; (7) Buttongrass; and (8) Pine. In addition, a non-combustible land cover type was included to account for water bodies, for example. The selection was made based on consultation with experts from the Australian fire community at a science workshop (Sydney, 7 June 2018). Local vegetation types from the Australian mainland and Tasmania were grouped into eight broad fuel types to align with the fire spread models. Since many vegetation types do not have a fire spread model specifically designed for them (e.g. rainforests, arid shrublands, wetlands, rural and urban areas), these vegetation types were assigned to the model with the most similar fuel structure (as per Cruz et al. 2018). Fire Behaviour Indices were derived from each of the fire behaviour models above, mostly based on modelled fire intensity. The exception to this approach was Buttongrass, for which no intensity calculation was available. Instead, a Fire Behaviour Index for Buttongrass was obtained from fire rate of spread calculations.

Taking fire behaviour, operational response and potential consequences into account, thresholds (transition points) for fire danger categories were identified. This is outlined in detail in (Hollis et al. 2024a). For Forest, Grassland, Buttongrass and Pine, six categories of fire danger were identified. Mallee-Heath and Shrubland were assigned five categories, while Northern Grassland (savannah) and Spinifex had four. The categories ranged from ‘mostly self-extinguishing, trouble-free fires’ (Category 1), to fires with ‘rapid fire growth; a high probability of loss of life and property; initial attack success is critical to prevent large fire development; and elevated risk to firefighter safety’ (Category 6). Please note that the descriptions of these categories are, as is fire behaviour, specific to fuel types.

Fire danger rating definition tables were developed that define the characteristics of the fire danger categories (Hollis et al. 2024a). Importantly, these tables were used for the live trial assessment (further discussed below), rather than estimates of other fire characteristics such as fireline intensity or flame length. Calculations were computed daily by the Bureau of Meteorology using gridded forecast weather at hourly intervals at a 1.5-km  resolution across Australia. Each grid cell was assigned to one of the eight broad fuel types (Kenny et al. 2024) and with the weather forecast and fuel characteristics combined, fire behaviour metrics such as rate of spread, intensity, flame height and spotting distance were calculated by using the corresponding fire behaviour model. These metrics were subsequently classified into the rating categories as defined by Hollis et al. (2024a). Ratings were then displayed on (1) a static website, updated daily, showing maps and tables of the daily maximum ratings for each jurisdiction and Fire Weather District, and (2) an interactive website that displayed the ratings and fire behaviour metrics as hourly outputs on maps, and as time series plots.

Participants and training

To assess the performance of the AFDRS_RP, 71 experts from fire and land management agencies across Australia (Appendix 1) were nominated to take part in the live trial. The intention was to have the experts evaluate the fire danger rating of fire incidents during the 2017–2018 southern fire season and compare their observed ratings with the forecast ratings as calculated by the system. The ratings used in this assessment were those employed more broadly in the AFDRS_RP, repeated here for convenience: Category 1–6 for the AFDRS_RP, and Low-Moderate, High, Very High, Severe, Extreme and Catastrophic for the FFDI/GFDI. Four members of the AFDRS project team processed the observations received from participants.

Prior to the commencement of the live trial, participants were introduced to the project through a webinar or face-to-face training session that (1) explained the purpose of the project and the design of the AFDRS_RP, and (2) demonstrated the use of the interactive visualisation systems. The evaluation of the fire incidents, and the performance assessment of the AFDRS_RP, relied critically on the expert opinion of the live trial participants. Because of this, there was potential for bias or variation in opinion depending on the participant’s role, background, experience and training (Plucinski et al. 2012). To quantify the extent of any such variation in expert opinion of fire activity and of AFDRS_RP performance, live trial participants were asked to complete evaluations for two fire event scenarios. Details of these two scenarios can be found in Appendix 2. For convenience, the two scenarios are summarised as: (1) an easily managed grass fire with a Category 2 or Low-Moderate rating (as assessed by the project team); and (2) a more complex forest fire with limited available background information with a Category 4 or High fire danger rating. No attempt was made to calibrate or modify expert ratings in light of the results of this scenario exercise, as this was felt likely to introduce the potential for greater error.

Data collection

Live trial participants were asked to put their incident evaluations in a data collection spreadsheet (Appendix 3), which helped to streamline the collation of an observation database. The evaluations included information on the incident such as location, time and date, fire size, fuel type and fire behaviour, but also information such as suppression strategies and potential consequences. Ideally, evaluations were completed within 24 h of a fire incident; otherwise, participants could download the forecast for the current and previous day, which allowed the completion of evaluations at a later time.

In addition to characterising the incident, participants were asked to estimate the performance of the rating systems by comparing the forecast rating with what they observed. Participants provided rating assessments for both the AFDRS_RP (Category 1–6) and the McArthur system (Low-Moderate, High, Very High, etc.). This was subsequently used for the performance assessment.

Rating forecasts were used in the evaluation process, rather than observations, because gridded forecast ratings were available at 1.5 km horizontal and hourly temporal resolution across Australia. Observational data, such as from the BARRA reanalysis (Sauvage et al. 2024), was only generally available at a lower horizontal resolution of 12 km. It was felt by the project team that the forecast data in general would provide a better estimate of actual conditions at fire observation locations.

Since limited numbers of observations in Categories 5 and 6 were expected, the database was supplemented with well-documented case studies from significant historical fires. This included incidents at the higher end of fire danger, as required to assess the rating systems at the full potential spectrum of fire danger.

Case studies were chosen from as wide a variety of fuel types as possible, noting the requirement for good documentation of fire, fuel and weather conditions. Thus, in addition to forest and grass fires, case studies included, for example, pine, mallee-heath and heathland fires (Matthews et al. 2018, ch. 9). Case study fires were characterised by both over- and under-prediction at times. The latter occurred especially in periods of extreme fire behaviour outside the bounds of predictability by fire behaviour models (e.g. Waroona fire, Peace et al. (2022)). There is a bias in case selection, which should be noted. Well-documented cases were generally those where fire activity caused significant impacts, whether on communities or assets, and which therefore justified the considerable efforts required for data collection, analysis and documentation.

Data analysis

Database entries were audited to ensure that each incident was consistently reported and that the recorded locations and fuel types were accurate. Where errors were identified, corrections were made where possible or entries were excluded from the analysis.

Multi-category contingency tables were produced to summarise system performance. Apart from the raw numbers in tables, the following five statistical measurements were calculated to evaluate the forecasts:

where K = the number of categories

N = the total number of forecasts

n(F_i, O_j) = the number of forecasts in category i that had observations in category j

N(F_i) = the total number of forecasts in category i

N(O_j) = the total number of observations in category j

where s_ij are elements of a scoring matrix given by

(for i = j, diagonal)

(for i ≠ j, off-diagonal)

with the sample probabilities given by

*For a multi-category contingency table:

To assist with the calculations of these measurements, the ‘verification’ package in R was used (Weather Forecast Verification Utilities, R package ver. 1.42. 2015, R Core Team (2018)).

The analysis, as described above, was used to evaluate overall system performance on different versions of the datasets (e.g. raw data, audited data, with/without case studies). Also, we evaluated the system performance per fuel type, and examined fires with initial attack success. A selection of the results is shown in this paper, the full analysis can be found in Matthews et al. (2019).

Potential observation bias (participants)

Of the 71 registered live trial participants, 21 participants completed scenario 1, and 22 completed scenario 2. Details on these two scenarios are in Appendix 2. There was very little variation in the response to scenario 1. With the exception of two observations, all participants rated the AFDRS_RP rating as a Category 2 (Fig. 1a), in agreement with the forecast. Similarly, participants consistently rated the fire as Low-Moderate using the FFDI/GFDI system (Fig. 1b).

Fig. 1.

Boxplots showing the variation in participant response for two scenarios, (a) for the AFDRS_RP, (b) for the FFDI/GFDI system. The boxes represent the inter-quartile range around the median (black bar), the whiskers indicate the range of scores within the first (0–25%) and fourth (75–100%) quartile. Points indicate outliers. For the FFDI/GFDI system: L-M, Low-Moderate; H, High; VH, Very High; S, Severe; E, Extreme; C, Catastrophic.

In contrast, there was considerable variation in response to scenario 2 (Fig. 1). While the median response for the AFDRS_RP was Category 4 (in agreement with the rating forecast), the observations ranged between 2 and 4 (Fig. 1a). This suggests that participants described the fire of scenario 2 as anything between ‘typical prescribed burning conditions, fires generally easy to suppress’ to a fire where ‘initial attack success is critical to prevent large fire development and defensive strategies are used’ (following Hollis et al. (2024a)). There was even more variation in response to scenario 2 when using the FFDI/GFDI system. Observations varied from Low-Moderate to Extreme, with the median being Very-High, which was a category higher than forecast.

The evaluation of these two scenarios suggests that fairly simple fires, supported with sufficient background information, are likely to have provided consistent observed ratings of the AFDRS_RP and FFDI/GFDI system across the dataset. However, more complex fires or fires with minimal supporting information are likely to have resulted in considerable variation in observed ratings, particularly for the FFDI/GFDI system. The clearly defined descriptors of the AFDRS_RP, taking fire behaviour, suppression/containment and consequences into account (Hollis et al. 2024a), are likely to have reduced the amount of variation that can be attributed to subjective interpretation.

Summary statistics

During the 2017–2018 southern Australian fire season, observations were collected covering all jurisdictions, all broad fuel types, and all rating categories (Table 1). A total of 264 live trial observations were collected between October and March, most of which came from New South Wales, Western Australia and Victoria (Fig. 2). There were fewer observations from northern Australia (i.e. Northern Territory and Queensland) because the period between October and March spans mostly the wet season there.

Fig. 2.

Map of observations that were used in the performance assessment. The dataset included 264 live trial observations (black) and 72 case study observations (grey).

One of the aims of the AFDRS_RP was to assess the fire danger across its full categorical spectrum. Therefore, evaluations from nine well-documented and 11 other high impact fires outside the live trial period were added. This provided a larger number of fire danger ratings at the higher end (e.g. Categories 5 and 6). Overall, this supplemented the database with 72 case study observations, leading to a total number of 336 data points for the AFDRS_RP performance assessment (Table 1 and Fig. 2).

Many observations came from Forests (n, 1878) and Grasslands (n, 77); while Pine, Buttongrass and Spinifex were less well represented (n ≤ 10, Table 1). As was expected, most fires had an observed fire danger rating of Category 2, 3, or 4, with a lower number of observations at the lower and higher end.

Performance assessment

The overall performance of the AFDRS_RP was superior to the FFDI/GFDI system, with 56% of the observations correctly predicted by the AFDRS_RP vs 43% for the FFDI/GFDI system (Fig. 3). The AFDRS_RP over-predicted more (35 vs 16%), but under-predicted less (9 vs 41%) than the FFDI/GFDI system. While the observations were fairly evenly distributed across the categories (and diagonal axis) for the AFDRS_RP (Fig. 3a), in contrast, most of the observations were grouped in the top left corner of the contingency table for the FFDI/GFDI system (Fig. 3b). This is also reflected in the Peirce and Gerrity skill scores, which take into account the distribution of observations, which were higher for the AFDRS_RP (0.45 vs 0.23) than the FFDI/GFDI system (0.66 vs 0.42).

Fig. 3.

Forecast vs observed ratings for all observations (n = 336) for (a) the Research Prototype, and (b) the FFDI/GFDI. Cell values represent the number of observations in each category. Values along the diagonal line (green) represent cases where forecast values matched the observed ratings, and the system was considered to perform correctly. Values under the diagonal represent under-predictions (in purple), and values above the diagonal represent over-predicted fire danger (in orange). L-M, low-moderate; H, high; VH, very high; S, severe; E, extreme; C, catastrophic.

Broad fuel type specific performance

Forest

Most of the observations came from forest fires, with 187 observations in total. The percentage of correctly predicted fire danger ratings was higher for the AFDRS_RP than for the FFDI/GFDI system (54 vs 46%, medium grey colour in Fig. 4). The AFDRS_RP had a higher Peirce skill score as well, 0.43 vs 0.28. Whilst the AFDRS_RP tends to over-predict, which may lead to unnecessary total fire bans or crew preparedness, the FFDI/GFDI system under-predicted 35% of the time. This may lead to dangerous conditions, under-preparedness and can put life and properties at risk.

Fig. 4.

System comparison per fuel type. Percentage under-predicted, correct and over-predicted are given as stacked bar graphs for (a) = AFDRS_RP (left bar of the cluster) and (b) = FFDI/GFDI system (right bar of the cluster). Black diamonds represent the Peirce skill score. Number of observations per fuel type are given in brackets.

Case studies only

Case studies were included in our performance assessment, next to the observations made by the trial participants, to capture the higher end of the rating system. Eighteen case studies were evaluated and this led to 72 fire danger rating observations. The performance assessment showed that the AFDRS_RP is better at predicting the fire danger rating at higher ends of the scale, with correct rating predictions in 61% of the observations compared to 33% for the FFDI/GFDI system (medium grey colour in Fig. 5). This was also reflected in a higher Peirce skill score for the AFDRS_RP, 0.48 vs 0.22. The FFDI/GFDI system under-predicted the fire danger rating in 58% of the observations.

Fig. 5.

System comparison for case studies only (n = 72). Percentage under-predicted, correct, and over-predicted are given as stacked bar graphs for (a) = AFDRS_RP (left bar of the cluster) and (b) = FFDI/GFDI system (right bar of the cluster). Black diamonds represent the Peirce skill score.

Improved performance

The AFDRS_RP was critically assessed using results from the live trial with observations from all jurisdictions for the duration of the 2017/2018 southern Australian fire season. This demonstrated that the aim of a national system was achieved and that the build of a fire danger rating system based on multiple fire behaviour models for various fuel types, coupled with fuel type mapping and high resolution weather forecasting, is a feasible approach. Small computational changes were made throughout the live trial without affecting the system performance, thereby confirming its modular approach and allowing for continuous improvement – an important design criterion for system development (Jones et al. 2001).

Assessed against expert opinion, the AFDRS_RP outperformed the McArthur system overall, for the case study observations, and for most broad fuel types. The observations from the case studies included fire danger ratings at the higher end of the classification; e.g. Category 5 and 6. The assessment of fires with rapid fire growth is crucial because under these conditions the risks of loss and damage to life and property are greatest (Blanchi et al. 2010, 2012). As the AFDRS_RP correctly predicted 61% of the case studies, compared to 33% with the FFDI/GFDI system, we affirm the framework and use of the system as a viable replacement.

Despite that the FFDI was specifically designed for forests (McArthur and Dwyer 1958; McArthur 1967) and has been in use for decades, the AFDRS_RP more accurately characterised fire danger with the majority of the live trial observations coming from this fuel type (187 out of 336 observations). One of the reasons for this is that the FFDI system assumes a standard fuel load of 1.25 kg m⁻², while the AFDRS_RP allows for regional variation in fuel inputs. However, there is still room for improvement, given that 36% of the observations of Forest fuels were over-predicted. This could partly be explained by the use of the DEFFM (Vesta) forest fire spread model (Cheney et al. 2012) in conditions it was not designed for, such as the initial build-up phase of fires, or for vegetation types classified as wet sclerophyll forests (Kenny et al. 2024). Ongoing research into improving fire spread models for Australian forests is promising in this regard (e.g. Cruz et al. 2021). The DEFFM/Vesta model does not include fuel availability or rainfall effects, therefore, a drought factor modification was used for the AFDRS_RP (Matthews et al. 2019). Including a detailed forest fuel moisture model (e.g. Matthews 2006) could also improve the accuracy of the AFDRS in these fuel types.

The AFDRS_RP also outperformed the GFDI, again as judged by expert participants in the project, reflecting the improvements in applying a newer fire spread model (Cheney et al. 1998) as well as an updated curing function (Cruz et al. 2015b). This also demonstrates the value of a modular system that can be adapted to incorporate new research.

Limitations

Recommendations

The build of the AFDRS_RP and the results of the live trial identified areas for future improvement. Apart from the suggestions mentioned above, the live trial period was extended so that the dry season in northern Australia could be included. Also, evaluations of the underrepresented fuels (especially Spinifex, Buttongrass and Pine) were targeted during the extension of the live trial throughout the 2018–2019 southern Australian fire season. Evaluation of the AFDRS is an ongoing process and the findings will be used to fine-tune the AFDRS. This work provides a basis for the documentation of these and subsequent upgrades.

To address the over-prediction bias, it will be necessary to revisit the application of some fire behaviour models. Extra attention should be given to the fire build-up phase and initial attack response, topographic effects such as slope and aspect, the role of spotting and application of the fire spread models and AFDRS_RP definition tables for less common fuel types. This is further discussed in (Kenny et al. 2024). Any potential biases related to weather inputs are closely monitored by fire weather meteorologists who, as noted above, curate AFDRS forecast output.

The AFDRS_RP is intrinsically linked to fire spread models for specific fuel types with each model requiring, to varying degrees of sensitivity (Matthews et al. 2018), a range of fuel input parameters. Consequently, the dependence on consistent and accurate spatial representation of fuel type and characteristics means that continued investment is needed to ensure correct fire danger forecasting. To date, sub-seasonal to seasonal updates of fuel state have occurred to maintain currency of the system and to correct any errors detected.

The AFDRS_RP live trial demonstrated promising results. Subsequent developments of the AFDRS_RP have also showed improvements for forecasting fire danger across Australia, whether as part of the project work or as new science such as fire behaviour models or system input availability. The AFDRS project has thereby helped, and continues to help, to create a safer environment for the Australian community.