Guidelines for effective evaluation and comparison of wildland fire occurrence prediction models
Nathan Phelps A B and Douglas G. Woolford A C
+ Author Affiliations
- Author Affiliations
A Department of Statistical and Actuarial Sciences, University of Western Ontario, London N6A 3K7, Canada.
B Department of Computer Science, University of Western Ontario, London N6A 3K7, Canada.
C Corresponding author. Email: dwoolfor@uwo.ca
International Journal of Wildland Fire 30(4) 225-240 https://doi.org/10.1071/WF20134
Submitted: 28 August 2020 Accepted: 16 December 2020 Published: 29 January 2021
Journal Compilation © IAWF 2021 Open Access CC BY-NC-ND
Abstract
Daily, fine-scale spatially explicit wildland fire occurrence prediction (FOP) models can inform fire management decisions. Many different data-driven modelling methods have been used for FOP. Several studies use multiple modelling methods to develop a set of candidate models for the same region, which are then compared against one another to choose a final model. We demonstrate that the methodologies often used for evaluating and comparing FOP models may lead to selecting a model that is ineffective for operational use. With an emphasis on spatially and temporally explicit FOP modelling for daily fire management operations, we outline and discuss several guidelines for evaluating and comparing data-driven FOP models, including choosing a testing dataset, choosing metrics for model evaluation, using temporal and spatial visualisations to assess model performance, recognising the variability in performance metrics, and collaborating with end users to ensure models meet their operational needs. A case study for human-caused FOP in a provincial fire control zone in the Lac La Biche region of Alberta, Canada, using data from 1996 to 2016 demonstrates the importance of following the suggested guidelines. Our findings indicate that many machine learning FOP models in the historical literature are not well suited for fire management operations.
Keywords: area under curve (AUC), Brier score, logarithmic score, mean absolute error (MAE), model selection, precision-recall curve, receiver operating characteristic curve, visual diagnostics, wildfire occurrence.
References
Alexander ME, Taylor SW, Page WG (2015) Wildland firefighter safety and fire behavior prediction on the fireline. In ‘Proceedings of the 13th international wildland fire safety summit & 4th human dimensions wildland fire conference’. Boise, Idaho, USA. pp. 20–24. (International Association of Wildland Fire, Missoula, Montana, USA)Alonso-Betanzos A, Fontenla-Romero O, Guijarro-Berdiñas B, Hernández-Pereira E, Andrade MIP, Jiménez E, Soto JLL, Carballas T (2003) An intelligent system for forest fire risk prediction and firefighting management in Galicia. Expert Systems with Applications 25, 545–554.
| An intelligent system for forest fire risk prediction and firefighting management in Galicia.Crossref | GoogleScholarGoogle Scholar |
Bar Massada A, Syphard AD, Stewart SI, Radeloff VC (2013) Wildfire ignition-distribution modelling: a comparative study in the Huron–Manistee National Forest, Michigan, USA. International Journal of Wildland Fire 22, 174–183.
| Wildfire ignition-distribution modelling: a comparative study in the Huron–Manistee National Forest, Michigan, USA.Crossref | GoogleScholarGoogle Scholar |
Benedetti R (2010) Scoring rules for forecast verification. Monthly Weather Review 138, 203–211.
| Scoring rules for forecast verification.Crossref | GoogleScholarGoogle Scholar |
Bickel JE (2007) Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Analysis 4, 49–65.
| Some comparisons among quadratic, spherical, and logarithmic scoring rules.Crossref | GoogleScholarGoogle Scholar |
Boyd K, Eng KH, Page CD (2013) Area under the precision-recall curve: point estimates and confidence intervals. In ‘Joint European conference on machine learning and knowledge discovery in databases’, 23–27 September 2013, Prague, Czech Republic. (Eds H Blockeel, K Kersting, S Nijssen, F Železný), pp. 451–466. (Springer: Berlin, Germany)
Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Review 78, 1–3.
| Verification of forecasts expressed in terms of probability.Crossref | GoogleScholarGoogle Scholar |
Brillinger DR, Preisler HK, Benoit JW (2003) Risk assessment: a forest fire example. Lecture Notes -Monograph Series 40, 177–196.
| Risk assessment: a forest fire example.Crossref | GoogleScholarGoogle Scholar |
Chawla NV, Japkowicz N, Kotcz A (2004) Special issue on learning from imbalanced data sets. SIGKDD Explorations 6, 1–6.
| Special issue on learning from imbalanced data sets.Crossref | GoogleScholarGoogle Scholar |
Chollet F, Allaire JJ (2017) R interface to keras. Available at https://keras.rstudio.com/index.html [Verified 17 December 2020]
Costafreda-Aumedes S, Comas C, Vega-Garcia C (2017) Human-caused fire occurrence modelling in perspective: a review. International Journal of Wildland Fire 26, 983–998.
| Human-caused fire occurrence modelling in perspective: a review.Crossref | GoogleScholarGoogle Scholar |
Cunningham AA, Martell DL (1973) A stochastic model for the occurrence of man-caused forest fires. Canadian Journal of Forest Research 3, 282–287.
| A stochastic model for the occurrence of man-caused forest fires.Crossref | GoogleScholarGoogle Scholar |
Dal Pozzolo A, Caelen O, Johnson RA, Bontempi G (2015) Calibrating probability with undersampling for unbalanced classification. In ‘2015 IEEE symposium series on computational intelligence’, 7–10 December 2015, Cape Town, South Africa. pp. 159–166. (IEEE)
Davis J, Goadrich M (2006) The relationship between Precision-Recall and ROC curves. In ‘Proceedings of the 23rd international conference on machine learning’, 25–29 June 2006, Pittsburgh, USA. pp. 233–240. (ACM)
Ecological Stratification Working Group (1995) A national ecological framework for Canada. Report and national map at 1 : 7 500 000 scale. Agriculture and Agri-Food Canada, Research Branch, Centre for Land and Biological Resources Research and Environment Canada, State of the Environment Directorate, Ecozone Analysis Branch. (Ottawa/Hull, Canada).
Géron A (2017) ‘Hands-on machine learning with Scikit-Learn and TensorFlow: concepts, tools, and techniques to build intelligent systems’. (O’Reilly Media, Inc.: Sebastopol, CA, USA)
Grau J, Grosse I, Keilwagen J (2015) PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics 31, 2595–2597.
| PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R.Crossref | GoogleScholarGoogle Scholar | 25810428PubMed |
Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29–36.
| The meaning and use of the area under a receiver operating characteristic (ROC) curve.Crossref | GoogleScholarGoogle Scholar | 7063747PubMed |
Harrell FE, Jr (2015) ‘Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis.’ (Springer: Berlin, Germany)
Hosmer DW, Jr, Lemeshow S, Sturdivant RX (2013) ‘Applied logistic regression.’ (John Wiley & Sons)
Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. Journal of Machine Learning Research 37, 448–456.
Jeni LA, Cohn JF, De La Torre F (2013) Facing imbalanced data – recommendations for the use of performance metrics. In ‘2013 Humaine Association conference on affective computing and intelligent interaction’, 2–5 September 2013, Geneva, Switzerland. pp. 245–251. (IEEE)
Johnston LM, Flannigan MD (2018) Mapping Canadian wildland fire interface areas. International Journal of Wildland Fire 27, 1–14.
| Mapping Canadian wildland fire interface areas.Crossref | GoogleScholarGoogle Scholar |
Johnston LM, Wang X, Erni S, Taylor SW, McFayden CB, Oliver JA, Stockdale C, Christianson A, Boulanger Y, Gauthier S, Arseneault D, Wotton BM, Parisien MA, Flannigan MD (2020) Wildland fire risk research in Canada. Environmental Reviews 28, 164–186.
| Wildland fire risk research in Canada.Crossref | GoogleScholarGoogle Scholar |
Keilwagen J, Grosse I, Grau J (2014) Area under precision-recall curves for weighted and unweighted data. PLoS One 9, e92209
| Area under precision-recall curves for weighted and unweighted data.Crossref | GoogleScholarGoogle Scholar | 24651729PubMed |
Kingma D, Ba J (2014) Adam: a method for stochastic optimization. In ‘Proceedings of the 3rd international conference on learning representations’, 7–9 May 2015, San Diego, USA.
Kourtz P, Todd B (1991) Predicting the daily occurrence of lightning-caused forest fires. Forestry Canada, Petawawa National Forest Institute, Information Report PI-X-112. (Chalk River, ON).
Kuhn M (2008) Building predictive models in R using the caret package. Journal of Statistical Software 28, 1–26.
| Building predictive models in R using the caret package.Crossref | GoogleScholarGoogle Scholar |
Liaw A, Wiener M (2002) Classification and regression by randomForest. R News 2, 18–22.
Magnussen S, Taylor SW (2012) Prediction of daily lightning-and human-caused fires in British Columbia. International Journal of Wildland Fire 21, 342–356.
| Prediction of daily lightning-and human-caused fires in British Columbia.Crossref | GoogleScholarGoogle Scholar |
Martell DL (2007) Forest fire management: current practices and new challenges for operational researchers. In ‘Handbook of operations research in natural resources’. (Eds A Weintraub, C Romero, T Bjørndal, R Epstein) pp. 489–509. (Springer: Berlin, Germany)
Martell DL, Otukol S, Stocks BJ (1987) A logistic model for predicting daily people-caused forest fire occurrence in Ontario. Canadian Journal of Forest Research 17, 394–401.
| A logistic model for predicting daily people-caused forest fire occurrence in Ontario.Crossref | GoogleScholarGoogle Scholar |
Martell DL, Bevilacqua E, Stocks BJ (1989) Modelling seasonal variation in daily people-caused forest fire occurrence. Canadian Journal of Forest Research 19, 1555–1563.
| Modelling seasonal variation in daily people-caused forest fire occurrence.Crossref | GoogleScholarGoogle Scholar |
McFayden CB, Woolford DG, Stacey A, Boychuk D, Johnston JM, Wheatley MJ, Martell DL (2020) Risk assessment for wildland fire aerial detection patrol route planning in Ontario, Canada. International Journal of Wildland Fire 29, 28–41.
| Risk assessment for wildland fire aerial detection patrol route planning in Ontario, Canada.Crossref | GoogleScholarGoogle Scholar |
Merkle EC, Steyvers M (2013) Choosing a strictly proper scoring rule. Decision Analysis 10, 292–304.
| Choosing a strictly proper scoring rule.Crossref | GoogleScholarGoogle Scholar |
Nadeem K, Taylor SW, Woolford DG, Dean CB (2020) Mesoscale spatio-temporal predictive models of daily human and lightning-caused wildland fire occurrence in British Columbia. International Journal of Wildland Fire 29, 11–27.
| Mesoscale spatio-temporal predictive models of daily human and lightning-caused wildland fire occurrence in British Columbia.Crossref | GoogleScholarGoogle Scholar |
Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In ‘Proceedings of the 27th international conference on machine learning’, 21–24 June 2010, Haifa, Israel. (Eds J Fürnkranz, T Joachims) pp. 807–814. (Omnipress: Madison, WI, United States)
Natural Resources Canada (2020) Canadian Forest Fire Danger Rating System (CFFDRS) summary. Available at https://cwfis.cfs.nrcan.gc.ca/background/summary/fdr [Verified 7 December 2020]
Orriols-Puig A, Bernadó-Mansilla E (2009) Evolutionary rule-based systems for imbalanced data sets. Soft Computing 13, 213–225.
| Evolutionary rule-based systems for imbalanced data sets.Crossref | GoogleScholarGoogle Scholar |
Paul RK (2006) Multicollinearity: causes, effects and remedies. IASRI 1, 58–65.
Platt J (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers 10, 61–74.
Plucinski MP (2012) A review of wildfire occurrence research. Bushfire Cooperative Research Centre. (Melbourne, Vic., Australia)
Plucinski MP, McCaw WL, Gould JS, Wotton BM (2014) Predicting the number of daily human-caused bushfires to assist suppression planning in south-west Western Australia. International Journal of Wildland Fire 23, 520–531.
| Predicting the number of daily human-caused bushfires to assist suppression planning in south-west Western Australia.Crossref | GoogleScholarGoogle Scholar |
Prechelt L (1998) Early stopping – but when? In ‘Neural networks: tricks of the trade’. (Eds GB Orr, K-R Müller) pp. 55–69. (Springer: Berlin, Germany)
Preisler HK, Brillinger DR, Burgan RE, Benoit JW (2004) Probability-based models for estimation of wildfire risk. International Journal of Wildland Fire 13, 133–142.
| Probability-based models for estimation of wildfire risk.Crossref | GoogleScholarGoogle Scholar |
Preisler HK, Westerling AL, Gebert KM, Munoz-Arriola F, Holmes TP (2011) Spatially explicit forecasts of large wildland fire probability and suppression costs for California. International Journal of Wildland Fire 20, 508–517.
| Spatially explicit forecasts of large wildland fire probability and suppression costs for California.Crossref | GoogleScholarGoogle Scholar |
R Core Team (2017) R: A language and environment for statistical computing. (R Foundation for Statistical Computing: Vienna, Austria). Available at https://www.R-project.org/ [Verified 17 December 2020]
Rodrigues M, de la Riva J (2014) An insight into machine-learning algorithms to model human-caused wildfire occurrence. Environmental Modelling & Software 57, 192–201.
| An insight into machine-learning algorithms to model human-caused wildfire occurrence.Crossref | GoogleScholarGoogle Scholar |
Saito T, Rehmsmeier M (2015) The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One 10, e0118432
| The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets.Crossref | GoogleScholarGoogle Scholar | 26469698PubMed |
Sakr GE, Elhajj IH, Mitri G, Wejinya UC (2010) Artificial intelligence for forest fire prediction. In ‘2010 IEEE/ASME international conference on advanced intelligent mechatronics’. pp. 1311–1316. (IEEE)
Sakr GE, Elhajj IH, Mitri G (2011) Efficient forest fire occurrence prediction for developing countries using two weather parameters. Engineering Applications of Artificial Intelligence 24, 888–894.
| Efficient forest fire occurrence prediction for developing countries using two weather parameters.Crossref | GoogleScholarGoogle Scholar |
Sherry J, Neale T, McGee TK, Sharpe M (2019) Rethinking the maps: a case study of knowledge incorporation in Canadian wildfire risk management and planning. Journal of Environmental Management 234, 494–502.
| Rethinking the maps: a case study of knowledge incorporation in Canadian wildfire risk management and planning.Crossref | GoogleScholarGoogle Scholar | 30641360PubMed |
Stocks BJ, Lynham TJ, Lawson BD, Alexander ME, Wagner CV, McAlpine RS, Dube DE (1989) Canadian Forest Fire Danger Rating System: an overview. Forestry Chronicle 65, 258–265.
| Canadian Forest Fire Danger Rating System: an overview.Crossref | GoogleScholarGoogle Scholar |
Stojanova D, Panov P, Kobler A, Džeroski S, Taskova K (2006) Learning to predict forest fires with different data mining techniques. In ‘Conference on data mining and data warehouses’. pp. 255–258. (SIKDD: Ljubljana, Slovenia)
Stojanova D, Kobler A, Ogrinc P, Ženko B, Džeroski S (2012) Estimating the risk of fire outbreaks in the natural environment. Data Mining and Knowledge Discovery 24, 411–442.
| Estimating the risk of fire outbreaks in the natural environment.Crossref | GoogleScholarGoogle Scholar |
Taylor SW, Woolford DG, Dean CB, Martell DL (2013) Wildfire prediction to inform management: statistical science challenges. Statistical Science 28, 586–615.
| Wildfire prediction to inform management: statistical science challenges.Crossref | GoogleScholarGoogle Scholar |
Todd JB, Kourtz PH (1991) Predicting the daily occurrence of people-caused forest fires. Forestry Canada, Petawawa National Forestry Institute, Information Report PI-X-103. (Chalk River, ON, Canada)
Tymstra C, Stocks BJ, Cai X, Flannigan MD (2020) Wildfire management in Canada: review, challenges and opportunities. Progress in Disaster Science 5, 100045
| Wildfire management in Canada: review, challenges and opportunities.Crossref | GoogleScholarGoogle Scholar |
Van Beusekom AE, Gould WA, Monmany AC, Khalyani AH, Quiñones M, Fain SJ, Andrade-Núñez MJ, González G (2018) Fire weather and likelihood: characterizing climate space for fire occurrence and extent in Puerto Rico. Climatic Change 146, 117–131.
| Fire weather and likelihood: characterizing climate space for fire occurrence and extent in Puerto Rico.Crossref | GoogleScholarGoogle Scholar |
Vasconcelos MJP, Silva S, Tome M, Alvim M, Pereira JC (2001) Spatial prediction of fire ignition probabilities: comparing logistic regression and neural networks. Photogrammetric Engineering and Remote Sensing 67, 73–81.
Vega-Garcia C, Woodard PM, Titus SJ, Adamowicz WL, Lee BS (1995) A logit model for predicting the daily occurrence of human caused forest-fires. International Journal of Wildland Fire 5, 101–111.
| A logit model for predicting the daily occurrence of human caused forest-fires.Crossref | GoogleScholarGoogle Scholar |
Vega-Garcia C, Lee BS, Woodard PM, Titus SJ (1996) Applying neural network technology to human-caused wildfire occurrence prediction. AI Applications 10, 9–18.
Vilar L, Woolford DG, Martell DL, Martín MP (2010) A model for predicting human-caused wildfire occurrence in the region of Madrid, Spain. International Journal of Wildland Fire 19, 325–337.
| A model for predicting human-caused wildfire occurrence in the region of Madrid, Spain.Crossref | GoogleScholarGoogle Scholar |
Wang X, Wotton BM, Cantin AS, Parisien MA, Anderson K, Moore B, Flannigan MD (2017) cffdrs: an R package for the Canadian Forest Fire Danger Rating System. Ecological Processes 6, 5
| cffdrs: an R package for the Canadian Forest Fire Danger Rating System.Crossref | GoogleScholarGoogle Scholar |
Wilcoxon F (1992) Individual comparisons by ranking methods. In ‘Breakthroughs in statistics’. (Eds S Kotz, NL Johnson) pp. 196–202. (Springer: New York, NY, USA)
Willmott CJ, Matsuura K (2005) Advantages of the mean absolute error (MAE) over the root-mean-square error (RMSE) in assessing average model performance. Climate Research 30, 79–82.
| Advantages of the mean absolute error (MAE) over the root-mean-square error (RMSE) in assessing average model performance.Crossref | GoogleScholarGoogle Scholar |
Wood SN (2011) Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models. Journal of the Royal Statistical Society. Series B, Statistical Methodology 73, 3–36.
| Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models.Crossref | GoogleScholarGoogle Scholar |
Woolford DG, Bellhouse DR, Braun WJ, Dean CB, Martell DL, Sun J (2011) A spatiotemporal model for people-caused forest fire occurrence in the Romeo Malette forest. Journal of Environmental Statistics 2, 2–16.
Woolford DG, Wotton BM, Martell DL, McFayden C, Stacey A, Evens J, Caputo J, Boychuk D, Kuyvenhoven R, Leonard D, Leroux G, McLarty D, Welch F (2016) Daily lightning- and person-caused fire prediction models used in Ontario. Poster presented at Wildland Fire Canada 2016 Conference, Kelowna, BC, Canada. Available at http://www.wildlandfire2016.ca/wp-content/uploads/2019/11/McFayden-Fire-Occurence-Prediction-Poster-Ontario-2016-10-17V2Final.pdf [Verified 22 May 2020]
Woolford DG, Martell DL, McFayden C, Evens J, Stacey A, Wotton BM, Boychuk D (2020) The development and implementation of a human-caused wildland fire occurrence prediction system for the province of Ontario, Canada. Canadian Journal of Forest Research
| The development and implementation of a human-caused wildland fire occurrence prediction system for the province of Ontario, Canada.Crossref | GoogleScholarGoogle Scholar |
Wotton BM (2009) Interpreting and using outputs from the Canadian Forest Fire Danger Rating System in research applications. Environmental and Ecological Statistics 16, 107–131.
| Interpreting and using outputs from the Canadian Forest Fire Danger Rating System in research applications.Crossref | GoogleScholarGoogle Scholar |
Wotton BM, Martell DL (2005) A lightning fire occurrence model for Ontario. Canadian Journal of Forest Research 35, 1389–1401.
| A lightning fire occurrence model for Ontario.Crossref | GoogleScholarGoogle Scholar |
Xi DD, Taylor SW, Woolford DG, Dean CB (2019) Statistical models of key components of wildfire risk. Annual Review of Statistics and Its Application 6, 197–222.
| Statistical models of key components of wildfire risk.Crossref | GoogleScholarGoogle Scholar |
