Enhancing wildfire detection: a novel algorithm for controllable generation of wildfire smoke images
Yinuo Huo A B , Qixing Zhang A * , Chong Wang A , Haihui Wang A and Yongming Zhang AA
B
Abstract
The lack of wildfire smoke image data is one of the most important factors hindering the development of image-based wildfire detection. Smoke image generation based on image inpainting techniques is a solution worthy of study. However, it is difficult to generate smoke texture with context consistency in complex backgrounds with current image inpainting methods.
This work aims to provide a wildfire smoke image database for specific scenarios.
We designed an algorithm based on generative adversarial networks (GANs) to generate smoke images. The algorithm includes a multi-scale fusion module to ensure consistency between the generated smoke and backgrounds. Additionally, a local feature-matching mechanism in the discriminator guides the generator to capture real smoke’s feature distribution.
We generated 13,400 wildfire smoke images based on forest background images and early fire simulation from the Fire Dynamics Simulator (FDS).
A variety of advanced object detection algorithms were trained based on the generated data. The experimental results confirmed that the addition of the generated data to the real datasets can effectively improve model performance.
This study paves a way for generating object datasets to enhance the reliability of watchtower or satellite wildfire monitoring.
Keywords: controllable smoke image generation, deep learning, Fire Dynamics Simulator, generative adversarial network (GAN), image inpainting, image smoke detection, numerical simulation.
References
Akhloufi MA, Tokime RB, Elassady H (2018) Wildland fires detection and segmentation using deep learning. In ‘Pattern Recognition and Tracking XXIX’. pp. 86–97. (SPIE) 10.1117/12.2304936
Alkhatib R, Sahwan W, Alkhatieb A, Schütt B (2023) A brief review of machine learning algorithms in forest fires science. Applied Sciences 13(14), 8275.
| Crossref | Google Scholar |
Banerjee S, Scheirer W, Bowyer K, Flynn P (2020) On hallucinating context and background pixels from a face mask using multi-scale gans. In ‘Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision’. pp. 300–309. (IEEE) 10.1109/wacv45572.2020.9093568
Bochkovskiy A, Wang CY, Liao HYM (2020) Yolov4: optimal speed and accuracy of object detection. arXiv Preprint arXiv:2004.10934.
| Crossref | Google Scholar |
Bowman DM, Balch JK, Artaxo P, Bond WJ, Carlson JM, Cochrane MA, d’Antonio CM, DeFries RS, Doyle JC, Harrison SP, Johnston FH, Keeley JE, Krawchuk MA, Kull CA, Marston JB, Moritz MA, Prentice IC, Roos CI, Scott AC, Swetnam TW, van der Werf GR, Pyne SJ (2009) Fire in the Earth system. Science 324(5926), 481-484.
| Crossref | Google Scholar | PubMed |
Chakraborty T, Reddy U, Naik SM, Panja M, Manvitha B (2024) Ten years of generative adversarial nets (GANs): a survey of the state-of-the-art. Machine Learning: Science and Technology 5(1), 011001.
| Crossref | Google Scholar |
Cheng HY, Yin JL, Chen BH, Yu ZM (2019) Smoke 100k: a database for smoke detection. In ‘2019 IEEE 8th Global Conference on Consumer Electronics (GCCE)’. pp. 596–597. (IEEE) 10.1109/GCCE46687.2019.9015309
Duan K, Bai S, Xie L, Qi H, Huang Q, Tian Q (2019) Centernet: keypoint triplets for object detection. In ‘Proceedings of the IEEE/CVF International Conference on Computer Vision’. pp. 6569–6578. (IEEE) 10.1109/iccv.2019.00667
Genovese A, Labati RD, Piuri V, Scotti F (2011) Virtual environment for synthetic smoke clouds generation. In ‘2011 IEEE International Conference on Virtual Environments, Human-Computer Interfaces and Measurement Systems Proceedings’. pp. 1–6. (IEEE) 10.1109/VECIMS.2011.6053841
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In ‘Advances in Neural Information Processing Systems. Vol. 27’. (Eds Z Ghahramani, M Welling, C Cortes, N Lawrence, KQ Weinberger) pp. 139–144. (Curran Associates, Inc.) 10.1145/3422622
Hui Z, Li J, Wang X, Gao X (2020) Image fine-grained inpainting. arXiv Preprint arXiv:2002.02609.
| Crossref | Google Scholar |
Iizuka S, Simo-Serra E, Ishikawa H (2017) Globally and locally consistent image completion. ACM Transactions on Graphics (ToG) 36(4), 1-14.
| Crossref | Google Scholar |
Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In ‘Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition’. pp. 1125–1134. (IEEE) 10.1109/cvpr.2017.632
Labati RD, Genovese A, Piuri V, Scotti F (2013) Wildfire smoke detection using computational intelligence techniques enhanced with synthetic smoke plume generation. IEEE Transactions on Systems, Man, and Cybernetics: Systems 43(4), 1003-1012.
| Crossref | Google Scholar |
Li J, He F, Zhang L, Du B, Tao D (2019) Progressive reconstruction of visual structure for image inpainting. In ‘Proceedings of the IEEE/CVF international conference on computer vision’. pp. 5962–5971. (IEEE) 10.1109/iccv.2019.00606
Liu H, Jiang B, Song Y, Huang W, Yang C (2020) Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. In ‘Computer Vision–ECCV 2020: 16th European Conference’, 23–28 August 2020, Proceedings, Part II 16. pp. 725–741. (Springer International Publishing: Glasgow, UK) 10.1007/978-3-030-58536-5_47
Lopez-Paz D, Oquab M (2016) Revisiting classifier two-sample tests. arXiv Preprint arXiv:1610.06545.
| Crossref | Google Scholar |
Mameli F, Bertini M, Galteri L, Del Bimbo A (2021) A NoGAN approach for image and video restoration and compression artifact removal. In ‘2020 25th International Conference on Pattern Recognition’. pp. 9326–9332. (IEEE) 10.1109/ICPR48806.2021.9413095
Mao J, Zheng C, Yin J, Tian Y, Cui W (2021) Wildfire smoke classification based on synthetic images and pixel-and feature-level domain adaptation. Sensors 21(23), 7785.
| Crossref | Google Scholar | PubMed |
Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv Preprint arXiv:1411.1784.
| Crossref | Google Scholar |
Mohapatra A, Trinh T (2022) Early wildfire detection technologies in practice – a review. Sustainability 14(19), 12270.
| Crossref | Google Scholar |
Moritz MA, Batllori E, Bradstock RA, Gill AM, Handmer J, Hessburg PF, Leonard J, McCaffrey S, Odion DC, Schoennagel T, Syphard AD (2014) Learning to coexist with wildfire. Nature 515(7525), 58-66.
| Crossref | Google Scholar | PubMed |
Namozov A, Im Cho Y (2018) An efficient deep learning algorithm for fire and smoke detection with limited data. Advances in Electrical and Computer Engineering 18(4), 121-128.
| Crossref | Google Scholar |
Nikolenko SI (2021) ‘Synthetic data for deep learning. Vol. 174.’ (Springer Nature) 10.1007/978-3-030-75178-4
Quan W, Zhang R, Zhang Y, Li Z, Wang J, Yan DM (2022) Image inpainting with local and global refinement. IEEE Transactions on Image Processing 31, 2405-2420.
| Crossref | Google Scholar | PubMed |
Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
| Crossref | Google Scholar |
Sagong MC, Shin YG, Kim SW, Park S, Ko SJ (2019) Pepsi: fast image inpainting with parallel decoding network. In ‘Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition’. pp. 11360–11368. (IEEE) 10.1109/cvpr.2019.01162
Sheikh HR, Sabir MF, Bovik AC (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on Image Processing 15(11), 3440-3451.
| Crossref | Google Scholar | PubMed |
Shuai L, Bo W, Ranran D, Zhiqiang Z, Sun L (2016) A novel smoke detection algorithm based on fast self-tuning background subtraction. In ‘2016 Chinese control and decision conference (CCDC)’. pp. 3539–3543. (IEEE) 10.1109/CCDC.2016.7531596
Song Y, Yang C, Shen Y, Wang P, Huang Q, Kuo CCJ (2018) SPG-Net: Segmentation prediction and guidance network for image inpainting. arXiv preprint arXiv:1805.03356.
| Crossref | Google Scholar |
Tan M, Pang R, Le QV (2020) Efficientdet: scalable and efficient object detection. In ‘Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition’. pp. 10781–10790. (IEEE) 10.1109/cvpr42600.2020.01079
Varghese R, Sambath M (2024) YOLOv8: a novel object detection algorithm with enhanced performance and robustness. In ‘2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS)’. pp. 1–6. (IEEE) 10.1109/ADICS58448.2024.10533619
Vinay K, Jain C (2022) Fire and smoke detection with deep learning: a review. i-Manager’s. Journal on Digital Signal Processing 10(2), 22-32.
| Crossref | Google Scholar |
Wan Z, Zhang J, Chen D, Liao J (2021) High-fidelity pluralistic image completion with transformers. In ‘Proceedings of the IEEE/CVF International Conference on Computer Vision’. pp. 4692–4701. (IEEE) 10.1109/iccv48922.2021.00465
Wang C, Xu C, Wang C, Tao D (2018a) Perceptual adversarial networks for image-to-image transformation. IEEE Transactions on Image Processing 27(8), 4066-4079.
| Crossref | Google Scholar | PubMed |
Wang TC, Liu MY, Zhu JY, Tao A, Kautz J, Catanzaro B (2018b) High-resolution image synthesis and semantic manipulation with conditional gans. In ‘Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition’. pp. 8798–8807. (IEEE) 10.1109/cvpr.2018.00917
Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Qiao Y, Change Loy C (2018c) Esrgan: Enhanced super-resolution generative adversarial networks. In ‘Proceedings of the European Conference on Computer Vision (ECCV) Workshops’. pp. 1–23. (IEEE) 10.48550/arXiv.1809.00219
Wang Q, Wu B, Zhu P, Li P, Zuo W, Hu Q (2020) ECA-Net: Efficient channel attention for deep convolutional neural networks. In ‘Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition’. pp. 11534–11542. (IEEE) 10.1109/cvpr42600.2020.01155
Wang Y, Wang G, Chen C, Pan Z (2019) Multi-scale dilated convolution of convolutional neural network for image denoising. Multimedia Tools and Applications 78, 19945-19960.
| Crossref | Google Scholar |
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4), 600-612.
| Crossref | Google Scholar | PubMed |
Wang Z, Wu L, Li T, Shi P (2022) A smoke detection model based on improved YOLOv5. Mathematics 10(7), 1190.
| Crossref | Google Scholar |
Xie C, Tao H (2020) Generating realistic smoke images with controllable smoke components. IEEE Access 8, 201418-201427.
| Crossref | Google Scholar |
Xu G, Zhang YM, Zhang QX, Lin GH, Wang JJ (2017) Deep domain adaptation based video smoke detection using synthetic smoke images. Fire Safety Journal 93, 53-59.
| Crossref | Google Scholar |
Yang C, Lu X, Lin Z, Shechtman E, Wang O, Li H (2017) High-resolution image inpainting using multi-scale neural patch synthesis. In ‘Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition’. pp. 6721–6729. (IEEE) 10.1109/cvpr.2017.434
Yang S, Xiao W, Zhang M, Guo S, Zhao J, Shen F (2022) Image data augmentation for deep learning: a survey. arXiv preprint arXiv:2204.08610.
| Crossref | Google Scholar |
Yu F, Koltun V (2015) Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122.
| Crossref | Google Scholar |
Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In ‘Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition’. pp. 5505–5514. (IEEE) 10.1109/cvpr.2018.00577
Yuan F, Zhang L, Xia X, Huang Q, Li X (2019a) A wave-shaped deep neural network for smoke density estimation. IEEE Transactions on Image Processing 29, 2301-2313.
| Crossref | Google Scholar | PubMed |
Yuan F, Zhang L, Xia X, Wan B, Huang Q, Li X (2019b) Deep smoke segmentation. Neurocomputing 357, 248-260.
| Crossref | Google Scholar |
Zeng Y, Lin Z, Lu H, Patel VM (2021) Cr-fill: generative image inpainting with auxiliary contextual reconstruction. In ‘Proceedings of the IEEE/CVF International Conference on Computer Vision’. pp. 14164–14173. (IEEE) 10.1109/iccv48922.2021.01390
Zhang QX, Lin GH, Zhang YM, Xu G, Wang JJ (2018a) Wildland forest fire smoke detection based on faster R-CNN using synthetic smoke images. Procedia Engineering 211, 441-446.
| Crossref | Google Scholar |
Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018b) The unreasonable effectiveness of deep features as a perceptual metric. In ‘Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition’. pp. 586–595. (IEEE) 10.48550/arXiv.1801.03924
Zhao Y, Lv W, Xu S, Wei J, Wang G, Dang Q, Liu Y, Chen J (2024) Detrs beat yolos on real-time object detection. In ‘Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition’. pp. 16965–16974. (IEEE) 10.48550/arXiv.2304.08069
Zheng CX, Song GX, Cham TJ, Cai JF, Phung D, Luo LJ (2022) High-quality pluralistic image completion via code shared VQGAN. arXiv preprint arXiv:2204.01931.
| Crossref | Google Scholar |