Enhancing wildfire detection: a novel algorithm for controllable generation of wildfire smoke images

Yinuo Huo; Qixing Zhang; Chong Wang; Haihui Wang; Yongming Zhang

doi:10.1071/WF24068

RESEARCH ARTICLE (Open Access)

Previous Next Contents Vol 33(11)

Enhancing wildfire detection: a novel algorithm for controllable generation of wildfire smoke images

Yinuo Huo ^A ^B , Qixing Zhang

^A ^* , Chong Wang ^A , Haihui Wang ^A and Yongming Zhang ^A

+ Author Affiliations

- Author Affiliations

^A State Key Laboratory of Fire Science, University of Science and Technology of China, 96 Jinzhai Road, Hefei, Anhui 230026, China.

^B Hefei Institute for Public Safety Research, Tsinghua University, Hefei, Anhui 230601, China.

^* Correspondence to: qixing@ustc.edu.cn

International Journal of Wildland Fire 33, WF24068 https://doi.org/10.1071/WF24068

Submitted: 11 April 2024 Accepted: 10 October 2024 Published: 11 November 2024

© 2024 The Author(s) (or their employer(s)). Published by CSIRO Publishing on behalf of IAWF. This is an open access article distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND)

Abstract

Background

The lack of wildfire smoke image data is one of the most important factors hindering the development of image-based wildfire detection. Smoke image generation based on image inpainting techniques is a solution worthy of study. However, it is difficult to generate smoke texture with context consistency in complex backgrounds with current image inpainting methods.

Aims

This work aims to provide a wildfire smoke image database for specific scenarios.

Methods

We designed an algorithm based on generative adversarial networks (GANs) to generate smoke images. The algorithm includes a multi-scale fusion module to ensure consistency between the generated smoke and backgrounds. Additionally, a local feature-matching mechanism in the discriminator guides the generator to capture real smoke’s feature distribution.

Key results

We generated 13,400 wildfire smoke images based on forest background images and early fire simulation from the Fire Dynamics Simulator (FDS).

Conclusions

A variety of advanced object detection algorithms were trained based on the generated data. The experimental results confirmed that the addition of the generated data to the real datasets can effectively improve model performance.

Implications

This study paves a way for generating object datasets to enhance the reliability of watchtower or satellite wildfire monitoring.

Keywords: controllable smoke image generation, deep learning, Fire Dynamics Simulator, generative adversarial network (GAN), image inpainting, image smoke detection, numerical simulation.

References

Akhloufi MA, Tokime RB, Elassady H (2018) Wildland fires detection and segmentation using deep learning. In ‘Pattern Recognition and Tracking XXIX’. pp. 86–97. (SPIE) 10.1117/12.2304936

Alkhatib R, Sahwan W, Alkhatieb A, Schütt B (2023) A brief review of machine learning algorithms in forest fires science. Applied Sciences 13(14), 8275.
| Crossref | Google Scholar |

Banerjee S, Scheirer W, Bowyer K, Flynn P (2020) On hallucinating context and background pixels from a face mask using multi-scale gans. In ‘Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision’. pp. 300–309. (IEEE) 10.1109/wacv45572.2020.9093568

Bochkovskiy A, Wang CY, Liao HYM (2020) Yolov4: optimal speed and accuracy of object detection. arXiv Preprint arXiv:2004.10934.
| Crossref | Google Scholar |

Bowman DM, Balch JK, Artaxo P, Bond WJ, Carlson JM, Cochrane MA, d’Antonio CM, DeFries RS, Doyle JC, Harrison SP, Johnston FH, Keeley JE, Krawchuk MA, Kull CA, Marston JB, Moritz MA, Prentice IC, Roos CI, Scott AC, Swetnam TW, van der Werf GR, Pyne SJ (2009) Fire in the Earth system. Science 324(5926), 481-484.
| Crossref | Google Scholar | PubMed |

Chakraborty T, Reddy U, Naik SM, Panja M, Manvitha B (2024) Ten years of generative adversarial nets (GANs): a survey of the state-of-the-art. Machine Learning: Science and Technology 5(1), 011001.
| Crossref | Google Scholar |

Cheng HY, Yin JL, Chen BH, Yu ZM (2019) Smoke 100k: a database for smoke detection. In ‘2019 IEEE 8th Global Conference on Consumer Electronics (GCCE)’. pp. 596–597. (IEEE) 10.1109/GCCE46687.2019.9015309

Duan K, Bai S, Xie L, Qi H, Huang Q, Tian Q (2019) Centernet: keypoint triplets for object detection. In ‘Proceedings of the IEEE/CVF International Conference on Computer Vision’. pp. 6569–6578. (IEEE) 10.1109/iccv.2019.00667

Genovese A, Labati RD, Piuri V, Scotti F (2011) Virtual environment for synthetic smoke clouds generation. In ‘2011 IEEE International Conference on Virtual Environments, Human-Computer Interfaces and Measurement Systems Proceedings’. pp. 1–6. (IEEE) 10.1109/VECIMS.2011.6053841

Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In ‘Advances in Neural Information Processing Systems. Vol. 27’. (Eds Z Ghahramani, M Welling, C Cortes, N Lawrence, KQ Weinberger) pp. 139–144. (Curran Associates, Inc.) 10.1145/3422622

Hui Z, Li J, Wang X, Gao X (2020) Image fine-grained inpainting. arXiv Preprint arXiv:2002.02609.
| Crossref | Google Scholar |

Iizuka S, Simo-Serra E, Ishikawa H (2017) Globally and locally consistent image completion. ACM Transactions on Graphics (ToG) 36(4), 1-14.
| Crossref | Google Scholar |

Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In ‘Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition’. pp. 1125–1134. (IEEE) 10.1109/cvpr.2017.632

Labati RD, Genovese A, Piuri V, Scotti F (2013) Wildfire smoke detection using computational intelligence techniques enhanced with synthetic smoke plume generation. IEEE Transactions on Systems, Man, and Cybernetics: Systems 43(4), 1003-1012.
| Crossref | Google Scholar |

Li J, He F, Zhang L, Du B, Tao D (2019) Progressive reconstruction of visual structure for image inpainting. In ‘Proceedings of the IEEE/CVF international conference on computer vision’. pp. 5962–5971. (IEEE) 10.1109/iccv.2019.00606

Liu H, Jiang B, Song Y, Huang W, Yang C (2020) Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. In ‘Computer Vision–ECCV 2020: 16th European Conference’, 23–28 August 2020, Proceedings, Part II 16. pp. 725–741. (Springer International Publishing: Glasgow, UK) 10.1007/978-3-030-58536-5_47

Lopez-Paz D, Oquab M (2016) Revisiting classifier two-sample tests. arXiv Preprint arXiv:1610.06545.
| Crossref | Google Scholar |

Mameli F, Bertini M, Galteri L, Del Bimbo A (2021) A NoGAN approach for image and video restoration and compression artifact removal. In ‘2020 25th International Conference on Pattern Recognition’. pp. 9326–9332. (IEEE) 10.1109/ICPR48806.2021.9413095

Mao J, Zheng C, Yin J, Tian Y, Cui W (2021) Wildfire smoke classification based on synthetic images and pixel-and feature-level domain adaptation. Sensors 21(23), 7785.
| Crossref | Google Scholar | PubMed |

Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv Preprint arXiv:1411.1784.
| Crossref | Google Scholar |

Mohapatra A, Trinh T (2022) Early wildfire detection technologies in practice – a review. Sustainability 14(19), 12270.
| Crossref | Google Scholar |

Moritz MA, Batllori E, Bradstock RA, Gill AM, Handmer J, Hessburg PF, Leonard J, McCaffrey S, Odion DC, Schoennagel T, Syphard AD (2014) Learning to coexist with wildfire. Nature 515(7525), 58-66.
| Crossref | Google Scholar | PubMed |

Namozov A, Im Cho Y (2018) An efficient deep learning algorithm for fire and smoke detection with limited data. Advances in Electrical and Computer Engineering 18(4), 121-128.
| Crossref | Google Scholar |

Nikolenko SI (2021) ‘Synthetic data for deep learning. Vol. 174.’ (Springer Nature) 10.1007/978-3-030-75178-4

Quan W, Zhang R, Zhang Y, Li Z, Wang J, Yan DM (2022) Image inpainting with local and global refinement. IEEE Transactions on Image Processing 31, 2405-2420.
| Crossref | Google Scholar | PubMed |

Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
| Crossref | Google Scholar |

Sagong MC, Shin YG, Kim SW, Park S, Ko SJ (2019) Pepsi: fast image inpainting with parallel decoding network. In ‘Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition’. pp. 11360–11368. (IEEE) 10.1109/cvpr.2019.01162

Sheikh HR, Sabir MF, Bovik AC (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on Image Processing 15(11), 3440-3451.
| Crossref | Google Scholar | PubMed |

Shuai L, Bo W, Ranran D, Zhiqiang Z, Sun L (2016) A novel smoke detection algorithm based on fast self-tuning background subtraction. In ‘2016 Chinese control and decision conference (CCDC)’. pp. 3539–3543. (IEEE) 10.1109/CCDC.2016.7531596

Song Y, Yang C, Shen Y, Wang P, Huang Q, Kuo CCJ (2018) SPG-Net: Segmentation prediction and guidance network for image inpainting. arXiv preprint arXiv:1805.03356.
| Crossref | Google Scholar |

Tan M, Pang R, Le QV (2020) Efficientdet: scalable and efficient object detection. In ‘Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition’. pp. 10781–10790. (IEEE) 10.1109/cvpr42600.2020.01079

Varghese R, Sambath M (2024) YOLOv8: a novel object detection algorithm with enhanced performance and robustness. In ‘2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS)’. pp. 1–6. (IEEE) 10.1109/ADICS58448.2024.10533619

Vinay K, Jain C (2022) Fire and smoke detection with deep learning: a review. i-Manager’s. Journal on Digital Signal Processing 10(2), 22-32.
| Crossref | Google Scholar |

Wan Z, Zhang J, Chen D, Liao J (2021) High-fidelity pluralistic image completion with transformers. In ‘Proceedings of the IEEE/CVF International Conference on Computer Vision’. pp. 4692–4701. (IEEE) 10.1109/iccv48922.2021.00465

Wang C, Xu C, Wang C, Tao D (2018a) Perceptual adversarial networks for image-to-image transformation. IEEE Transactions on Image Processing 27(8), 4066-4079.
| Crossref | Google Scholar | PubMed |

Wang TC, Liu MY, Zhu JY, Tao A, Kautz J, Catanzaro B (2018b) High-resolution image synthesis and semantic manipulation with conditional gans. In ‘Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition’. pp. 8798–8807. (IEEE) 10.1109/cvpr.2018.00917

Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Qiao Y, Change Loy C (2018c) Esrgan: Enhanced super-resolution generative adversarial networks. In ‘Proceedings of the European Conference on Computer Vision (ECCV) Workshops’. pp. 1–23. (IEEE) 10.48550/arXiv.1809.00219

Wang Q, Wu B, Zhu P, Li P, Zuo W, Hu Q (2020) ECA-Net: Efficient channel attention for deep convolutional neural networks. In ‘Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition’. pp. 11534–11542. (IEEE) 10.1109/cvpr42600.2020.01155

Wang Y, Wang G, Chen C, Pan Z (2019) Multi-scale dilated convolution of convolutional neural network for image denoising. Multimedia Tools and Applications 78, 19945-19960.
| Crossref | Google Scholar |

Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4), 600-612.
| Crossref | Google Scholar | PubMed |

Wang Z, Wu L, Li T, Shi P (2022) A smoke detection model based on improved YOLOv5. Mathematics 10(7), 1190.
| Crossref | Google Scholar |

Xie C, Tao H (2020) Generating realistic smoke images with controllable smoke components. IEEE Access 8, 201418-201427.
| Crossref | Google Scholar |

Xu G, Zhang YM, Zhang QX, Lin GH, Wang JJ (2017) Deep domain adaptation based video smoke detection using synthetic smoke images. Fire Safety Journal 93, 53-59.
| Crossref | Google Scholar |

Yang C, Lu X, Lin Z, Shechtman E, Wang O, Li H (2017) High-resolution image inpainting using multi-scale neural patch synthesis. In ‘Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition’. pp. 6721–6729. (IEEE) 10.1109/cvpr.2017.434

Yang S, Xiao W, Zhang M, Guo S, Zhao J, Shen F (2022) Image data augmentation for deep learning: a survey. arXiv preprint arXiv:2204.08610.
| Crossref | Google Scholar |

Yu F, Koltun V (2015) Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122.
| Crossref | Google Scholar |

Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In ‘Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition’. pp. 5505–5514. (IEEE) 10.1109/cvpr.2018.00577

Yuan F, Zhang L, Xia X, Huang Q, Li X (2019a) A wave-shaped deep neural network for smoke density estimation. IEEE Transactions on Image Processing 29, 2301-2313.
| Crossref | Google Scholar | PubMed |

Yuan F, Zhang L, Xia X, Wan B, Huang Q, Li X (2019b) Deep smoke segmentation. Neurocomputing 357, 248-260.
| Crossref | Google Scholar |

Zeng Y, Lin Z, Lu H, Patel VM (2021) Cr-fill: generative image inpainting with auxiliary contextual reconstruction. In ‘Proceedings of the IEEE/CVF International Conference on Computer Vision’. pp. 14164–14173. (IEEE) 10.1109/iccv48922.2021.01390

Zhang QX, Lin GH, Zhang YM, Xu G, Wang JJ (2018a) Wildland forest fire smoke detection based on faster R-CNN using synthetic smoke images. Procedia Engineering 211, 441-446.
| Crossref | Google Scholar |

Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018b) The unreasonable effectiveness of deep features as a perceptual metric. In ‘Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition’. pp. 586–595. (IEEE) 10.48550/arXiv.1801.03924

Zhao Y, Lv W, Xu S, Wei J, Wang G, Dang Q, Liu Y, Chen J (2024) Detrs beat yolos on real-time object detection. In ‘Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition’. pp. 16965–16974. (IEEE) 10.48550/arXiv.2304.08069

Zheng CX, Song GX, Cham TJ, Cai JF, Phung D, Luo LJ (2022) High-quality pluralistic image completion via code shared VQGAN. arXiv preprint arXiv:2204.01931.
| Crossref | Google Scholar |