Performance of operational fire spread models in California

Introduction

Wildfires are a natural disturbance that shape the ecosystems and landscapes (Pausas and Keeley 2009) although they are a growing threat to the environment, economical assets and population in many regions worldwide (Molina-Terrén et al. 2019). In fact, substantial amounts of financial resources have been invested in fire management aiming to reduce the damage associated with unintended consequences of wildfires and improve safety for the population (Liang et al. 2008; Cardil and Molina 2015; Stocks and Martell 2016). Fire agencies rely on suites of wildfire analysis products that are designed to meet the needs of operational response and mitigation planning. Given the challenges that first responders and decision-makers were faced with to adequately predict fire spread and potential impacts, and dispatch the most appropriate resources every time a wildfire is detected in California, California Department of Forestry and Fire Protection (CAL FIRE) (2019), through a request for innovative ideas in March 2019, selected Wildfire Analyst Enterprise (WFA-e) to evaluate initial attack fires and calculate wildfire risk based on automatic simulations. In this project, automatic wildfire spread simulations are conducted aiming to (1) forecast risk days in advance to support operational decision making (readiness levels, resource allocation and release of timely and effective public warnings for disaster management (Cruz et al. 2018; Monedero et al. 2019; Ramirez et al. 2019)); (2) quantify asset and territory risk to inform on the most effective mitigation and asset hardening options. However, fire analysts can also generate on-demand spread predictions in response to active incidents for safe fire suppression strategies and tactics. These simulations are usually performed manually, adjusting the input data and calibrating the models in real time to optimise the model’s performance (Cardil et al. 2019).

The wildfire rate of spread (ROS) is determined by fuel availability, topography and weather (Barrows 1951). Several mathematical approaches have been developed in the last decades aiming at predicting fire behaviour. However, semi-empirical approaches such as Rothermel’s model (Rothermel 1972) have been the most widely used and evaluated fire spread models on Earth owing to simplicity in understanding, use and computational efficiency (Cruz and Alexander 2013; Cruz et al. 2018; Ascoli et al. 2022; Minsavage-Davis and Davies 2022). The capability of accurately predicting the fire spread is directly linked to the model’s inherent inaccuracy derived from its limitations and assumptions (Albini 1976; Ascoli et al. 2015; Vacchiano and Ascoli 2015) and input data quality, which are all factors that may undermine the utility of such predictions for decision-making (Cruz and Alexander 2013; Benali et al. 2017; Ramirez et al. 2019). Therefore, it seems essential to quantify the model’s performance and identify factors leading to unsatisfactory biases and errors. Cruz and Alexander (2013) conducted a comprehensive survey of studies, gathering a database of 1278 paired predicted vs observed ROS values and analysed the accuracy of ROS models. Other studies in several fire-prone areas assessed how new improvements in current fire spread models may enhance the performance relative to the older counterparts (Anderson et al. 2015; Cruz et al. 2018). More recently, Ascoli et al. (2022) used the Rothermel package in R (Vacchiano and Ascoli 2015) to test standard and custom fuel models against a global fire behaviour dataset (BONFIRE) and analysed model performance in different fuel types and ecosystems.

Recent advances in technology have allowed monitoring the fire progression of most wildfires every 15 min in the United States of America (USA) through the National Fireguard Detections product (FG). These data, when available for use on a fire, provide unprecedented capabilities to analyse factors influencing fire behaviour and compare the observed and predicted ROS in fires distributed across different and complex landscapes. Despite California being a major fire hotspot in the Americas, there is no extensive scientific analysis of operational fire spread models allowing analysing of their performance and drivers leading to model inaccuracies. This work assesses the predictive accuracy of the fire spread models currently used in California automatically in operational settings under different environmental conditions using 1853 fires from 2019 to 2021. The analysis identifies in what conditions the models may over- or underestimate the ROS and, subsequently, the burned area and associated fire impacts on buildings and other assets. We highlight that the simulations were performed automatically with no manual refinement of input data or model adjustments (Cardil et al. 2019). The analysis is based on well-known error metrics and statistical approaches (Cruz et al. 2018) aiming to compare the predicted with observed ROS values.

Methods

Study area

This study was developed in California (Fig. 1), a region dominated by mediterranean climatic conditions, known to foster recurrent large wildfires (Pyne et al. 1998). Fire-prone weather situations such as long and dry summers with thunderstorm episodes, low relative humidity and strong dry winds are typical of this region (Sugihara and Barbour 2006). Specifically, from September to May, the dry Diablo and Santa Ana winds arriving from the Great Basin and the Mojave Desert, with high desiccating potential and high wind speed (Li and Banerjee 2021) boost high ROS. The main vegetation types are shrubs, coniferous forest and herbaceous fine fuels according to the United States National Land Cover Database (39.03, 18.59 and 13.47%, respectively) (Jin et al. 2019). Fuel load, specifically dead materials, has increased in forests since 2010 in large areas of the State owing to successful fire suppression, timber harvesting practices and recent insect disturbances, potentially leading to fast and intense wildfires (van Wagtendonk et al. 2018).

Fig. 1. 

Wildfire ignitions retrieved from FireGuard in California from October 2019 to November 2021. Each fire has an associated rate of spread (ROS) for the first burning period (up to 8 h) obtained by averaging the ROS of all FG polygons by incident. The zoomed-scale box shows three fires with the associated FG polygons for the burning period (the largest fire is the Mountain View fire).

Fire progression dataset (FireGuard)

The National Fireguard Detections product (FG) provides wildfire detection and monitoring across the USA (US Department of Defense 2021). The program is operated in conjunction with the National Geospatial Intelligence Agency’s Firefly algorithm, which collects data from multiple sources and sensors, including imagery from satellites and unmanned aerial vehicles (National Guard Association 2021). It allows the detection of new wildfires as well as areas of significant fire growth, which is especially useful in remote regions and during the first 24–48 h when this information is usually scarce. Updates on fire progression are often provided every 15 min. The product provides a set of independent polygons representing sections of the fire front. To obtain a quantitative fire behaviour dataset from the raw data, polygons are classified into incidents and perimeters are analysed to obtain relevant metrics like burned area, maximum ROS, weather, fuels and terrain conditions.

To classify polygons into individual incidents (fires), we use a grid-growing clustering algorithm (Fig. 2; Zhao et al. 2015). All polygon centroids are projected into a 3D grid covering the spatio-temporal dimensions of the dataset using a 3 km in space and 24 h in time cell size resolution. This process leads to a density 3D matrix representing the number of elements at each cell. Clusters are defined as those groups of interconnected cells with at least a given number of centroids (islands of connected valid cells within the grid). To identify these clusters, we loop through the 3D matrix searching for a seed cell not belonging to any previously found cluster and assign it a new cluster ID (identity). That initial cell is then expanded (grown) among all connecting valid cells around it using a fast-marching method with eight degrees of freedom, and assigning the same cluster ID to all visited cells. This process is done iteratively until the loop is completed and all valid cells have an ID. The method assures that any FG polygon in the cluster has at least one neighbouring polygon within 2√2 cell size distance, which is ~67 h in time and 8.4 km in space.

Fig. 2. 

Extraction of ROS vectors from the FireGuard (FG) dataset: (a) retrieval and storing of the FG data in a database; (b) projection of FG polygons into a 3D matrix representing space and time. Clustering identification of hotspot islands inside the grid. (c) Example of a wildfire that shows the evolution in time of a set of FG polygons with their associated maximum ROS vector (bold line), the secondary spread vectors, and the overall perimeter (dashed yellow line). Vector colours are based on their ROS.

Once all clusters are identified, polygons in each incident are ordered in time, increased in resolution (number of vertices), and trimmed down to remove overlapping sections with previous polygons in time. Then a parent–child hierarchy of polygons is constructed based on the Hausdorf distance of each polygon with respect to the surrounding ones (Fig. 2). More explicitly, we compute and identify the vertices leading to the maximum distance from a point in a polygon to the closest point in the surrounding ones. If the child polygon is in contact with other polygons, the Hausdorf distance is computed only with respect to those shared vertices. If the child polygon is not in contact with any other polygon, it is assumed that it was created by spotting, and the Hausdorf distance is computed with respect to all polygons within the last 8 h. Polygons that have no valid distance vector are considered to be independent ignition sources not caused by any other polygon. The maximum ROS vector of a polygon is given by the Hausdorf distance vector divided by the arrival time difference between the parent and child polygon. Once the parent–child hierarchy is established, we can also compute all secondary spread vectors by computing the minimum distance from each vertex of the source polygon to the target one divided by the elapsed time between them. The number of these vectors will be equal to the number of vertices in the child polygon. Finally, to compute the overall perimeter length and burn area during the incident, a multi-polygon geometry representing the overall fire perimeter of the incident at each time step is created by computing the dissolved union of all previous polygons in time (the union of polygons without internal vertices, Fig. 2).

Through the Integrated Reporting of Wildland Fire Information (IRWIN) framework, which integrates different dispatch systems across the US, we automatically evaluated more than 30 000 fires in the study period (October 2019 to November 2021) from the reported locations. Of those incidents, a total of 1853 wildfires occurred in California and had FG data associated with their unique IRWIN IDs. We removed prescribed fires and wildfires with a duration less than 1 h and selected the first FG polygon later used as ignition source to simulate the fire growth. We used the first burning period (up to 8 h from the fire start) to compare the simulated ROS and observed ROS obtained with FG. In Fig. 3, we show four wildfires with associated ROS in each FG polygon and the fire simulation performed with WFA-e. To complete the fire behaviour dataset, fuel, terrain characteristics and weather conditions (see section 2.3 for further details) were assigned to each FG spread polygon through a polygon-based approach with zonal statistics.

Fig. 3. 

Fire progression (FireGuard data) and simulation of four wildfires through Wildfire Analyst Enterprise (WFA-e) in California: (a) Mountain View fire (lat.  38.515; lon.  −119.465; 2020/11/17); (b) Chaparral fire (lat.  33.485; lon.  −117.399; 2021/08/28); (c) Bridge fire (lat.  38.921; lon.  −121.037; 2021/09/05); (d) French fire (lat.  35.687; lon.  −118.55; 2021/08/18); fote that the FG polygons and WFA-e simulated fire progression have the same time duration.

Results

The average ROS measured from FG data for all analysed fires was 0.55 km/h, with extreme fires with a sustained ROS of 1.45 and 2.91 km/h (quantile 95 and 99% respectively, Fig. 1). The fires spreading on timber areas were the fastest on average (0.66 km/h) compared with Grass (0.55 km/h), Grass–Shrub (0.48 km/h) and Shrub fires (0.42 km/h). The comparison between the observed and predicted ROS and the influence of the different environmental variables on ROS are evaluated in this section and Figs 4 and 5. Binning the mean MBE and MAE by fuel type and environmental factors exposed which factors a given simulation was most sensitive to. Fuel type was the key contributor to ROS bias and wind speed impacts were secondary. Both live and dead fuel moisture had noticeably smaller contributions to MBE, which implies improvements to fuel mapping and wind forecasts are the most critical pieces to a successful implementation of the Rothermel spread model.

Fig. 4. 

Fire spread model’s performance analysis for 1853 wildfires that occurred in California in the 2019–2021 period: (a) histogram of ROS residuals by fuel type based on Scott and Burgan (2005); (b) ROS mean absolute error (MAE); (c) ROS mean bias error (MBE); (d) ROS mean absolute percentage error (MAPE). GR: Grass Fuel Types 101–109; GS: Grass–Shrub Fuel Types 121–124; SH: Shrub Fuel Types 141–149; TL: Timber–Litter–Understorey Fuel Types 161–189.

Fig. 5. 

Scatterplot of the observed and predicted ROS and mean bias error (MBE) by fuel type (Scott and Burgan 2005), wind speed, dead fuel moisture content and herbaceous live fuel moisture content for 1853 wildfires that occurred in California in the 2019–2021 period. GR: Grass Fuel Types 101–109; GS: Grass–Shrub Fuel Types 121–124; SH: Shrub Fuel Types 141–149; TL: Timber–Litter–Understorey Fuel Types 161–189.

Discussion

Fire agencies rely on semi-empirical models to daily predict fire behaviour aiming at assessing fire risk and improving the safety and effectiveness of fire suppression operations (Cardil et al. 2019). Thus, it is a priority to better understand the model’s performance under different environmental conditions to prove the feasibility and accuracy of outputs and further improve the current fire models. In fact, Rothermel’s (1972) model is the most widely evaluated fire spread model on Earth (Cruz and Alexander 2013; Cruz et al. 2018). In the present paper, the observed (FG) and predicted (WFA-e) ROS were compared analytically with a dataset of 1853 fires based on well-known error and bias metrics (Cruz et al. 2018). The amount, nature and quality of fire behaviour observations that FG data provide with extremely high temporal resolution have allowed a better understanding of the model’s performance and quantification of the effect of model input variables (i.e. fuel types, wind speed, fuel moisture content, etc.) on the model’s accuracy. The major caveats of previous research studies were associated with the availability of fire behaviour observations as well as inadequate input data (Cruz and Alexander 2013). Having overcome those obstacles, this analysis provides strong insights on the ROS predictions in operational environments.

The errors of fire behaviour predictions found in this work may be derived from intrinsic limitations and assumptions of fire spread models as well as external factors such as the quality of input data or local processes not captured by models (i.e. pyroconvection, local wind fields influenced by relief, etc.) (Ramirez et al. 2019) and the FG data. Although FG data provide unprecedented capabilities to validate the fire spread models, we recognise that small inaccuracies in the fire progression maps may occur, subsequently affecting the comparison between the observed and predicted ROS and associated errors. We used improved input data (fuels and weather) to run the simulations aiming to minimise the effect of input data quality in the model’s performance. In fact, improvements with simulating fire behaviour have arisen mostly from these external factors such as better mapping of fuel characteristics and moisture (Sullivan and Matthews 2013; Perrakis et al. 2014; Pimont et al. 2016), significant enhancements in weather forecasting with the operational implementation of mesoscale weather models and physically based downscaling techniques that can predict wind speed and direction over complex topography (Forthofer et al. 2009; Wagenbrenner et al. 2016), and the integration of fire behaviour models into software packages (Finney 1998, 2006; Ramírez et al. 2011; Monedero et al. 2019). Unfortunately, our study was not able to disaggregate errors resulting from all of these factors. Also, it is worth noting that all the fire simulations were performed automatically, exposing a new source of error given that the fire analyst can often adjust input data and calibrate the simulations in real time to adjust the simulated fire growth to the real fire progression (Cardil et al. 2019). For instance, it is well known how wind fields may change spatially owing to topography and how this effect may dramatically impact the fire simulations. This is a forecast error that may be easily addressed using observational data such as from weather stations near a given fire, and such input corrections did not occur with the simulations analysed herein. In this sense, Cruz and Alexander (2013) found that the lowest errors were associated with prescribed fires where fuel and weather inputs would have been measured on site. Therefore, the errors shown in this manuscript would strongly decrease with the knowledge of advanced users and integration of other data sources such as weather station data (Rothermel and Rinehart 1983; Stratton 2006; Alexander and Cruz 2013; Andrews 2014).

Overall, the fire spread model had moderate MAE and MAPE and low bias for Shrub (SH), Grass (GR) and Grass–Shrub (GS) Scott and Burgan (2005) fuel types, similarly to other studies (Cruz et al. 2018). Conversely, we found the highest MAE, MBE and MAPE in timber fuel types (Fig. 4) with a MAPE value of 67% with a preponderant ROS underprediction bias, an error type leading to potentially negative consequences in operational decision making (Cheney and Gould 1995). Previous studies found the lowest MAE values in timber and logging slash fuel types but also the largest MAPE (76%) values. These metrics were derived from the range of observed ROS values in these fuel types, which were substantially lower than in grasslands and shrublands fuel types (Cruz et al. 2018). However, in our study, we found the highest MAE and MBE values in timber fuel types because the database contained fires with high ROS that were not accurately predicted. This may be related to convective fire behaviour leading to local winds in the fire front not considered by the weather models or the difficulty of predicting active crown fire behaviour with the current fire spread models. Nevertheless, the effect of forest understorey fuel types on the ROS underprediction could probably be associated with the oversensitivity of the model to fuel bed depth and compactness (Catchpole et al. 1993). Thus, new custom fuel families adapted to California are needed to improve the accuracy of fire simulations. In this sense, we optimised custom timber fuel types for California based on genetic algorithms (Ascoli et al. 2015) using the data presented in this work, minimising the model bias and error substantially (MAPE 37%). The model’s accuracy was also influenced by other input variables beyond fuel types. Low wind speeds and high fuel moisture (dead and live) led to ROS underpredictions. This would probably be related to limitations of current wind field models to capture local patterns such as diurnal winds, abrupt landscapes driving turbulence, changes in wind direction or even pyroconvection that could increase the wind speed on the surface and prediction of crown fire behaviour in timber fuel types. Although attempts to reformulate and improve the Rothermel model have occurred in the past (Catchpole et al. 1993; Sandberg et al. 2007), the reality is that the model has yet to be replaced despite past recommendations for enhancing predictive fire models (Andrews 2014).

Our work supports the need for improvements proposed by Andrews (2014) and suggests that new enhancements (custom fuel families, improved input data including high-resolution wind fields, operational coupled fire–atmosphere models to capture pyroconvection, etc.) are still needed to decrease the error and bias of fire spread models, especially in timber areas. However, this research work provides promising results on the use of wildfire modelling for assessing fire spread potential during the initial attack in operational environments. In this sense, fire agencies can improve their preparedness and response in real time by systematically simulating all fire alerts and calculating fire risk metrics across the landscape. This procedure supports readiness and dispatching levels and drives the prioritisation of prevention activities. Also, electrical utilities can evaluate the fire risk and consequences, mitigate and prevent catastrophic wildfires. Finally, the fire spread models are also used to evaluate the extended attack incidents. Assessment of the model’s performance for this use case would require further work given that fire analysts can further calibrate the models in real time and the complexity of the fire environment is exacerbated owing to the aforementioned phenomena, such as pyroconvection, that may play a key role in explaining fire behaviour.

Conclusions

This work concludes that the fire spread model’s performance for California is in line with previous studies developed in other regions and the models are accurate enough to be used in real-time operations, especially with the use of adjustment modes that allow the calibration of predictions using field data (Artès et al. 2015; Cardil et al. 2019). Note that Cruz and Alexander (2013) highlighted that a ±35% error would be a reasonable and conservative standard for model performance of fire spread rate for research studies and could, understandably, result in wider error intervals in operational environments when simulations are performed automatically.

The accuracy of fire behaviour outputs is modulated by environmental conditions, especially fuel types and wind speed, which may bias ROS predictions. However, we also recognise that there are challenges regarding the effect of pyroconvection on local wind fields and the estimation of ROS in timber areas. The results of this evaluation suggest that the accuracy of fire simulations may be improved with newer models aiming to address the spread of fires in timber areas, crown fire behaviour modelling and the effect of pyroconvection in weather conditions and fire behaviour. The FG data and the algorithms developed in this research to calculate ROS vectors represent a crucial enhancement to better analyse the fire spread model’s performance for calibration efforts, developing new fire spread models and creating new fuel families. Our approach addresses issues arising from the use of long fire runs, encompassing at times variations in fuel types, the estimation of fuel characteristics across the landscape, and the averaging of wind speed over broad spatial and temporal scales.

Supplementary material

Supplementary material is available online.

Data availability

The data that support the findings of this study are classified. Restrictions apply to the availability of these data, which were used under licence for this study.

Conflicts of interest

The authors declare no conflicts of interest.

Declaration of funding

This research did not receive any specific funding.