FireFormer: an efficient Transformer to identify forest fire from surveillance cameras
Yuming Qiao A B # , Wenyu Jiang A B # , Fei Wang
A B * , Guofeng Su A , Xin Li C and Juncai Jiang A B
+ Author Affiliations
- Author Affiliations
A Department of Engineering Physics, Tsinghua University, Beijing, 100084, China.
B Institute of Safety Science and Technology, Tsinghua Shenzhen International Graduate School, Shenzhen, 518000, China.
C Foshan Urban Safety Research Center, Foshan, 528000, China.
International Journal of Wildland Fire 32(9) 1364-1380 https://doi.org/10.1071/WF22220
Submitted: 26 November 2022 Accepted: 18 July 2023 Published: 14 August 2023
© 2023 The Author(s) (or their employer(s)). Published by CSIRO Publishing on behalf of IAWF. This is an open access article distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND)
Abstract
Background: An effective identification model is crucial to realise the real-time monitoring and early warning of forest fires from surveillance cameras. However, existing models are prone to generate numerous false alarms under the interference of artificial smoke such as industrial smoke and villager cooking smoke, therefore a superior identification model is urgently needed.
Aims: In this study, we tested the Transformer-based model FireFormer to predict the risk probability of forest fire from the surveillance images.
Methods: FireFormer uses a shifted window self-attention module to extract similarities of divided patches in the image. The similarity in characteristics indicated the probability of forest fires. The GradCAM algorithm was then applied to analyse the interest area of FireFormer model and visualise the contribution of different image patches by calculating gradient reversely. To verify our model, the monitoring data from the high-point camera in Nandan Mountain, Foshan City, was collected and further constructed as a forest fire alarm dataset.
Key results: Our results showed that FireFormer achieved a competitive performance (OA: 82.21%, Recall: 86.635% and F1-score: 74.68%).
Conclusions: FireFormer proves to be superior to traditional methods.
Implications: FireFormer provides an efficient way to reduce false alarms and avoid heavy manual re-checking work.
Keywords: deep learning, forest fire identification, GradCAM, Interpretability analysis, self-attention mechanism, smoke detection, Transformer, wildland–urban interface.
References
Ahmed MR, Rahaman KR, Hassan QK (2018) Remote Sensing of Wildland Fire-Induced Risk Assessment at the Community Level. Sensors 18, 1570| Remote Sensing of Wildland Fire-Induced Risk Assessment at the Community Level.Crossref | GoogleScholarGoogle Scholar |
Amit Singh (2020) Automatic detection of hand hygiene using computer vision technology. Journal of the American Medical Informatics Association 27(8), 1316–1320.
| Automatic detection of hand hygiene using computer vision technology.Crossref | GoogleScholarGoogle Scholar |
Azim MR, Keskin M, Do N, Gül M (2022) Automated classification of fuel types using roadside images via deep learning. International Journal of Wildland Fire 31, 982–987.
| Automated classification of fuel types using roadside images via deep learning.Crossref | GoogleScholarGoogle Scholar |
Ba LJ, Kiros JR, Hinton GE (2016) Layer Normalization. CoRR. abs/1607.06450.
Ba R, Chen C, Yuan J, Song W, Lo S (2019) SmokeNet: Satellite smoke scene detection using convolutional neural network with spatial and channel-wise attention. Remote Sensing 11, 1702
| SmokeNet: Satellite smoke scene detection using convolutional neural network with spatial and channel-wise attention.Crossref | GoogleScholarGoogle Scholar |
Brock A, De S, Smith SL, Simonyan K (2021) High-performance large-scale image recognition without normalization. In ‘International Conference on Machine Learning’, 18–24 July 2021, ICML-22. pp. 1059–1071. Available at https://proceedings.mlr.press/v139/brock21a.html
Cortes C, Mohri M, Rostamizadeh A (2009) L2 regularization for learning kernels. In ‘Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence’, 18–21 June 2009, UAI-09. pp. 109–116. (AUAI Press)
Devlin J, Chang MW, Lee K, Toutanova K (2018) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In ‘The North American Chapter of the Association for Computational Linguistics’, New Orleans, Louisiana, USA. (Association for Computational Linguistics)
Diakakis M, Xanthopoulos G, Gregos L (2016) Analysis of forest fire fatalities in Greece: 1977–2013. International Journal of Wildland Fire 25, 797–809.
| Analysis of forest fire fatalities in Greece: 1977–2013.Crossref | GoogleScholarGoogle Scholar |
Ding Z, Zhao Y, Li A, Zheng Z (2021) Spatial–Temporal Attention Two-Stream Convolution Neural Network for Smoke Region Detection. Fire 4, 66
| Spatial–Temporal Attention Two-Stream Convolution Neural Network for Smoke Region Detection.Crossref | GoogleScholarGoogle Scholar |
Ding Y, Deng W, Zheng Y, Liu P, Wang M, Cheng X, Bao J, Chen D, Zeng M (2022) IR-Net: Intra- and Inter-Human Relation Network for Multi-Person Pose Estimation. In ‘Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence’, IJCAI-22. pp. 855–862. (International Joint Conference on Artificial Intelligence)
Donida Labati R, Genovese A, Piuri V, Scotti F (2013) Wildfire smoke detection using computational intelligence techniques enhanced with synthetic smoke plume generation. IEEE Transactions on Systems, Man, and Cybernetics: Systems 43, 1003–1012.
| Wildfire smoke detection using computational intelligence techniques enhanced with synthetic smoke plume generation.Crossref | GoogleScholarGoogle Scholar |
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S (2021) An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ‘International Conference on Learning Representations’, 3–7 May 2021, ICLR-21. Available at https://openreview.net/forum?id=YicbFdNTTy
Dufour D, Le Noc L, Tremblay B, Tremblay MN, Généreux F, Terroux M, Vachon C, Wheatley MJ, Johnston JM, Wotton M, Topart P (2021) A Bi-Spectral Microbolometer Sensor for Wildfire Measurement. Sensors 21, 3690
| A Bi-Spectral Microbolometer Sensor for Wildfire Measurement.Crossref | GoogleScholarGoogle Scholar |
Elfadel IM, Wyatt Jr JL (1993) The “softmax” nonlinearity: Derivation using statistical mechanics and useful properties as a multiterminal analog circuit element. In ‘Advances in Neural Information Processing Systems. Vol. 6’. (Morgan-Kaufmann)
Fernández-Berni J, Carmona-Galán R, Martínez-Carmona JF, Rodríguez-Vázquez Á (2012) Early forest fire detection by vision-enabled wireless sensor networks. International Journal of Wildland Fire 21, 938–949.
| Early forest fire detection by vision-enabled wireless sensor networks.Crossref | GoogleScholarGoogle Scholar |
Glorot X, Bordes A, Bengio Y (2011) Deep Sparse Rectifier Neural Networks. In ‘Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics’, 11–13 April 2011, AISTATS. pp. 315–323. Available at https://proceedings.mlr.press/v15/
Göltaş M, Demirel T, Çağlayan İ (2017) Visibility Analysis of Fire Watchtowers Using GIS; A Case Study in Dalaman State Forest Enterprise. European Journal of Forest Engineering 3, 66–71.
Guede-Fernández F, Martins L, de Almeida RV, Gamboa H, Vieira P (2021) A Deep Learning Based Object Identification System for Forest Fire Detection. Fire 4, 75
| A Deep Learning Based Object Identification System for Forest Fire Detection.Crossref | GoogleScholarGoogle Scholar |
Gutmacher D, Hoefer U, Wöllenstein J (2012) Gas sensor technologies for fire detection. Sensors and Actuators B: Chemical 175, 40–45.
| Gas sensor technologies for fire detection.Crossref | GoogleScholarGoogle Scholar |
He K, Zhang X, Ren S, Sun J (2016) Deep Residual Learning for Image Recognition. In ‘IEEE Conference on Computer Vision and Pattern Recognition’, 27–30 June 2016, CVPR-16. pp. 770–778. (IEEE Computer Society)
He K, Chen X, Xie S, Li Y, Dollár P, Girshick R (2021) Masked Autoencoders Are Scalable Vision Learners. In ‘IEEE Conference on Computer Vision and Pattern Recognition’, 18–24 June 2021, CVPR-21. pp. 15979–15988. (IEEE Computer Society)
He L, Gong X, Zhang S, Wang L, Li F (2021) Efficient attention based deep fusion CNN for smoke detection in fog environment. Neurocomputing 434, 224–238.
| Efficient attention based deep fusion CNN for smoke detection in fog environment.Crossref | GoogleScholarGoogle Scholar |
Hendrycks D, Gimpel K (2016). Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR. arxiv.org, abs/1606.08415.
Howard A, Sandler M, Chen B, Wang W, Chen LC, Tan M, Chu G, Vasudevan V, Zhu Y, Pang R, Adam H, Le Q (2019) Searching for MobileNetV3. In ‘IEEE International Conference on Computer Vision’, 27 October–2 November 2019, ICCV-19. pp. 1314–1324. (IEEE Computer Society)
Hu L, Wang S, Li L, Huang Q (2018) How Functions Evolve in Deep Convolutional Neural Network. In ‘14th IEEE International Conference on Signal Processing’, 15–20 April 2018, ICSP. pp. 1133–1138. (IEEE press)
Huang G, Liu Z, Laurens V, Weinberger KQ (2017) Densely Connected Convolutional Networks. In ‘IEEE Conference on Computer Vision and Pattern Recognition’, 21–26 July 2017, CVPR-17. pp. 2261–2269. (IEEE Computer Society)
Jiang W, Wang F, Fang L, Zheng X, Qiao X, Li Z, Meng Q (2021) Modelling of wildland-urban interface fire spread with the heterogeneous cellular automata model. Environmental Modelling & Software 135, 104895
| Modelling of wildland-urban interface fire spread with the heterogeneous cellular automata model.Crossref | GoogleScholarGoogle Scholar |
Jiang W, Wang F, Su G, Li X, Wang G, Zheng X, Wang T, Meng Q (2022a) Modeling Wildfire Spread with an Irregular Graph Network. Fire 5, 185
| Modeling Wildfire Spread with an Irregular Graph Network.Crossref | GoogleScholarGoogle Scholar |
Jiang W, Wang F, Su G, Li X, Meng Q, Wang G (2022b) Key technologies of emergency management informatization for forest fires. China Safety Science Journal 32, 182–191.
Hu J, Shen L, Sun G (2019) Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 7132–7141.
Johnston JM, Johnston LM, Wooster MJ, Brookes A, McFayden C, Cantin AS (2018) Satellite Detection Limitations of Sub-Canopy Smouldering Wildfires in the North American Boreal Forest. Fire 1, 28
| Satellite Detection Limitations of Sub-Canopy Smouldering Wildfires in the North American Boreal Forest.Crossref | GoogleScholarGoogle Scholar |
Kalatzis N, Avgeris M, Dechouniotis D, Papadakis-Vlachopapadopoulos K, Roussaki I, Papavassiliou S (2018) Edge computing in IoT ecosystems for UAV-enabled early fire detection. In ‘2018 IEEE international conference on smart computing’, 18–20 June 2018, SMARTCOMP-18. pp. 106–114. (IEEE Computer Society)
Khan S, Muhammad K, Hussain T, Ser JD, Cuzzolin F, Bhattacharyya S, Akhtar Z, de Albuquerque VHC (2021) DeepSmoke: Deep learning model for smoke detection and segmentation in outdoor environments. Expert Systems with Applications 182, 115125
| DeepSmoke: Deep learning model for smoke detection and segmentation in outdoor environments.Crossref | GoogleScholarGoogle Scholar |
Khan F, Xu Z, Sun J, Khan FM, Ahmed A, Zhao Y (2022) Recent Advances in Sensors for Fire Detection. Sensors 22, 3310
| Recent Advances in Sensors for Fire Detection.Crossref | GoogleScholarGoogle Scholar |
Ko B, Park J, Nam J-Y (2013) Spatiotemporal bag-of-features for early wildfire smoke detection. Image and Vision Computing 31, 786–795.
| Spatiotemporal bag-of-features for early wildfire smoke detection.Crossref | GoogleScholarGoogle Scholar |
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In ‘Advances in Neural Information Processing Systems’, 3–6 December 2012, NIPS-12. pp. 1106–1114. (Morgan-Kaufmann)
Larkin NK, Raffuse SM, Strand TM (2014) Wildland fire emissions, carbon, and climate: U.S. emissions inventories. Forest Ecology and Management 317, 61–69.
| Wildland fire emissions, carbon, and climate: U.S. emissions inventories.Crossref | GoogleScholarGoogle Scholar |
Liu Z, Chen H, Runyang F, Wu S, Ji S, Yang B, Wang X (2021a) Deep Dual Consecutive Network for Human Pose Estimation. In ‘IEEE Conference on Computer Vision and Pattern Recognition’, 19–25 June 2021. CVPR-21. pp. 525–534. (IEEE Computer Society)
Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021b) Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In ‘IEEE International Conference on Computer Vision’, 10–17 October 2021, ICCV-21. pp. 9992–10002. (IEEE Computer Society)
Loshchilov I, Hutter F (2017) Decoupled Weight Decay Regularization. In ‘7th International Conference on Learning Representations’, 6–9 May 2017, ICLR-17. Available at https://openreview.net/pdf?id=Bkg6RiCqY7
Luo S, Yan C, Wu K, Zheng J (2015) Smoke detection based on condensed image. Fire Safety Journal 75, 23–35.
| Smoke detection based on condensed image.Crossref | GoogleScholarGoogle Scholar |
Luo W, Li Y, Urtasun R, Zemel R (2016) Understanding the effective receptive field in deep convolutional neural networks. In ‘Advances in Neural Information Processing Systems’, 5–10 December 2016, NIPS-16. pp. 4898–4906. Available at https://proceedings.neurips.cc/paper_files/paper/2016/file/c8067ad1937f728f51288b3eb986afaa-Paper.pdf
Manzello SL, Almand K, Guillaume E, Vallerent S, Hameury S, Hakkarainen T (2018) FORUM position paper: The growing global wildland urban interface (WUI) fire Dilemma: Priority needs for research. Fire Safety Journal 100, 64–66.
| FORUM position paper: The growing global wildland urban interface (WUI) fire Dilemma: Priority needs for research.Crossref | GoogleScholarGoogle Scholar |
Masoom SM, Zhang Q, Dai P, Jia Y, Zhang Y, Zhu J, Wang J (2022) Early Smoke Detection Based on Improved YOLO-PCA Network. Fire 5, 40
| Early Smoke Detection Based on Improved YOLO-PCA Network.Crossref | GoogleScholarGoogle Scholar |
Mell WE, Manzello SL, Maranghides A, Butry D, Rehm RG (2010) The wildlandurban interface fire problem current approaches and research needs. International Journal of Wildland Fire 19, 238–251.
| The wildlandurban interface fire problem current approaches and research needs.Crossref | GoogleScholarGoogle Scholar |
Milecki A, Rybarczyk D (2020) The Gas Fire Temperature Measurement for Detection of an Object’s Presence on Top of the Burner. Sensors 20, 2139
| The Gas Fire Temperature Measurement for Detection of an Object’s Presence on Top of the Burner.Crossref | GoogleScholarGoogle Scholar |
Ovadia Y, Fertig E, Ren J, Nado Z, Sculley D, Nowozin S, Snoek J (2019) Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In ‘Advances in Neural Information Processing Systems’, 8–14 December 2019. pp. 13969–13980. Available at https://doi.org/10.48550/arXiv.1906.02530
Podur J, Wotton M (2010) Will climate change overwhelm fire management capacity? Ecological Modelling 221, 1301–1309.
| Will climate change overwhelm fire management capacity?Crossref | GoogleScholarGoogle Scholar |
Qiang X, Zhou G, Chen A, Zhang X, Zhang W (2021) Forest fire smoke detection under complex backgrounds using TRPCA and TSVB. International Journal of Wildland Fire 30, 329–350.
| Forest fire smoke detection under complex backgrounds using TRPCA and TSVB.Crossref | GoogleScholarGoogle Scholar |
Research F. A. (2017). Transforming and augmenting images: CenterCrop in Pytorch. Available at https://pytorch.org/vision/stable/transforms.html
Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B (2022) High-resolution image synthesis with latent diffusion models. In ‘Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition’, 17–21 June 2022, CVPR-2022. pp. 10684–10695. (IEEE Computer Society)
Ryu J, Kwak D (2022) A Study on a Complex Flame and Smoke Detection Method Using Computer Vision Detection and Convolutional Neural Network. Fire 5, 108
| A Study on a Complex Flame and Smoke Detection Method Using Computer Vision Detection and Convolutional Neural Network.Crossref | GoogleScholarGoogle Scholar |
Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2020) Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. International Journal of Computer Vision 128, 336–359.
| Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization.Crossref | GoogleScholarGoogle Scholar |
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. In ‘3rd International Conference on Learning Representations’, 7–9 May 2015, ICLR-15. Available at https://arxiv.org/abs/1409.1556
Song L, Wang B, Zhou Z, Wang H, Wu S (2014) The research of real-time forest fire alarm algorithm based on video. In ‘2014 Sixth International Conference on Intelligent Human-Machine Systems and Cybernetics’, 26–27 August 2014, IHMSC-14. pp. 106–109. (IEEE press)
Sousa MJ, Moutinho A, Almeida M (2020) Thermal Infrared Sensing for Near Real-Time Data-Driven Fire Detection and Monitoring Systems. Sensors 20, 6803
| Thermal Infrared Sensing for Near Real-Time Data-Driven Fire Detection and Monitoring Systems.Crossref | GoogleScholarGoogle Scholar |
Tan M, Le Q (2019) EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks In ‘Proceedings of the 36th International Conference on Machine Learning’, 9–15 June 2019, ICML-2019. pp. 6105–6114. Available at https://proceedings.mlr.press/v97/tan19a.html
Tomkins L, Benzeroual T, Milner A, Zacher JE, Ballagh M, McAlpine RS, Doig T, Jennings S, Craig G, Allison RS (2014) Use of night vision goggles for aerial forest fire detection. International Journal of Wildland Fire 23, 678–685.
| Use of night vision goggles for aerial forest fire detection.Crossref | GoogleScholarGoogle Scholar |
Stanford University (2020) CS231n Convolutional Neural Networks for Visual Recognition Visualizing what ConvNets learn. Available at http://cs231n.stanford.edu/2021
Varghese AO, Suryavanshi AS, Jha CS (2022) Geospatial Applications in Wildlife Conservation and Management. In ‘Geospatial Technologies for Resources Planning and Management’. (Eds CS Jha, A Pandey, V Chowdary, V Singh) pp. 727–750. (Cham: Springer International Publishing)
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł and Polosukhin I (2017) Attention is all you need. In ‘Advances in Neural Information Processing Systems’, 4–9 December 2017. pp. 5998–6008. Available at https://doi.org/10.48550/arXiv.1706.03762
Villacrés J, Arevalo-Ramirez T, Fuentes A, Reszka P, Auat Cheein F (2019) Foliar Moisture Content from the Spectral Signature for Wildfire Risk Assessments in Valparaíso-Chile. Sensors 19, 5475
| Foliar Moisture Content from the Spectral Signature for Wildfire Risk Assessments in Valparaíso-Chile.Crossref | GoogleScholarGoogle Scholar |
Wang S, Xiao X, Deng T, Chen A, Zhu M (2019) A Sauter mean diameter sensor for fire smoke detection. Sensors and Actuators B: Chemical 281, 920–932.
| A Sauter mean diameter sensor for fire smoke detection.Crossref | GoogleScholarGoogle Scholar |
Welling M, Kipf TN (2016) Semi-supervised classification with graph convolutional networks. In ‘5th International Conference on Learning Representations’, 24–26 April 2017, ICLR-17. (Openreview.net)
Wu XX, Liu JG (2009) A new early stopping algorithm for improving neural network generalization. In ‘2009 Second international conference on intelligent computation technology and automation’, 10–11 October 2009, IEEE press, ICITCA-09. pp. 15–18. (IEEE press)
Xu G, Zhang Y, Zhang Q, Lin G, Wang Z, Jia Y, Wang J (2019) Video smoke detection based on deep saliency network. Fire Safety Journal 105, 277–285.
| Video smoke detection based on deep saliency network.Crossref | GoogleScholarGoogle Scholar |
Ye Bai BW, Wu Y, Liu X (2022) 2021 Global Forest Fire Roundup. Fire Science and Technology S762, 705–709.
Yuan F, Zhang L, Xia X, Huang Q, Li X (2020) A wave-shaped deep neural network for smoke density estimation. IEEE Transactions on Image Processing 29, 2301–2313.
| A wave-shaped deep neural network for smoke density estimation.Crossref | GoogleScholarGoogle Scholar |
Zhou Z, Shi Y, Gao Z, Li S (2016) Wildfire smoke detection based on local extremal region segmentation and surveillance. Fire Safety Journal 85, 50–58.
| Wildfire smoke detection based on local extremal region segmentation and surveillance.Crossref | GoogleScholarGoogle Scholar |
Zuo X (2022) Gather the powerful forces of fire fighting and rescue. China Emergency Management News 2, 1–3.
