Early forest flame and smoke detection based on improved feature extraction module with enhanced image processing inspired by YOLOV7

Dataset

The effectiveness of forest fire detection using convolutional neural networks heavily relies on the quality and scale of the dataset. With high-quality datasets, deep learning models can learn more features and exhibit superior generalisation capabilities. We have gathered over 25,000 images from various public datasets, including BoWFire Dataset (2021), FiSmo (Cazzolato et al. 2017), Firesense (Kucuk et al. 2008), VisiFire (2021), and HPWREN Fire (2020). Our dataset encompasses diverse forest fire images, including different types and scenes, as well as smoke images and non-flame images portraying forest scenes, shadows, sunset glow, and water mist. Fig. 2 shows some representative examples from the dataset (Table 1).

Fig. 2.

The samples of the dataset. (a) Forest flame images only, (b) forest smoke images, (c) pictures with flames and smoke, and (d) negative samples.

Multi-scale guided filtering Retinex image enhancement method

In this study, we propose a guide filtering method based on Retinex. Due to the different sources of forest flame smoke data and the influence of natural environmental factors from different lighting angles, the obtained smoke images will have uneven lighting and large colour deviation. When these forest flame and smoke images are directly inputted into the detection network, the network may not be able to directly extract the smoke features and distinguish information due to excessive strong light or shadow occlusion, resulting in high false negatives in the detection methods.

In Retinex theory, an image I(x, y) is represented as the multiplication of light layer B(x, y) and reflection layer R(x, y), and principal component analysis (PCA) is used to extract the brightness channel layer of the original image as the light layer. The Single Scale Retinex (SSR) reflector is represented by Eqn 1:

where i represents the ith component of the original image and P represents the PCA operation. The Multi Scale Retinex (MSR) reflector is represent by Eqn 2:

In Eqn 2, N is the number of the scales and n_i is the ith component of the nth scale. w_n is the nth scale-dependent weight and w₁ + … + w_n = 1. In this study, N is equal to 3, the three scales are 80, 120, and 250, and w₁ = w₂ = w₃ = 1/3.

In practice, shadow occlusion or light is too strong, the illumination layer of the image will change dramatically, and the direct use of the above method may cause pseudo-halo and detail loss etc. Guided filtering has the ability to protect edge and gradient. Therefore, guided filtering is used to further improve MSR.

A two-level decomposition is performed using guided filtering, expressed as Eqn 3:

where Guilter filter(·) is Guided filtering (GF), r_j is the local filtered window radius, and ε_j is a regularisation parameter introduced to prevent the guiding filter coefficient from becoming too large. We use F₁ to represent the image after the first filtering, and F₂ to represent the image after the second filtering. The flow of GF-Retinex image enhancement algorithm is in Algorithm 1.

To prove the effectiveness of the G-Retinex algorithm, three evaluation indexes, are used to evaluate the algorithm: (1) information entropy (IE); (2) peak signal-to-noise ratio (PSNR); and (3) running time. IE is the average amount of information contained in each picture, and the higher the information entropy, the richer the information contained in the image, specifically expressed as Eqn 4:

where p(i) is the probability distribution for all histogram counts. PSNR is an unbiased image quality parameter. The higher the peak signal-to-noise ratio, the better the image enhancement effect. PSNR is expressed as Eqn 5:

where I(x, y) and I_enhanced(x, y) are the former image and enhanced image intensity, respectively. M and W are the height and width of the image, respectively. We selected 100 smoke images in the dataset and compared them with typical image enhancement algorithms. The experiment results are in Table 2.

Retinex algorithm was applied to images with smoke, and the test results are in Fig. 3. A noticeable improvement can be observed when comparing the original smoke image with the MSR-GF method. Particularly, under ample lighting conditions, the MSR-GF method effectively enhances the details of small smoke objects within the image. This enhancement algorithm addresses the issue of missing smoke texture information caused by blurred images with low pixel density. Compared with the SSR method, G-Retinex can improve the confidence of object detection.

Fig. 3.

From left to right are the original image, the image after SSR processing, and the smoke image after G-Retinex image enhancement.

Flame segmentation based on YCbCr

Owing to the remarkable learning capacities exhibited by convolutional neural networks, deep learning has become extensively embraced in the realm of object detection tasks. Object detection networks, commonly employed, can be broadly categorised into two groups: (1) one-stage networks; and (2) two-stage networks. One-stage networks, as opposed to their two-stage counterparts, eliminate the necessity of generating proposal regions and instead directly predict the category and location of objects utilising the backbone network. This methodology presents various benefits, including accelerated computation speed and reduced computational costs, rendering it more apt for real world applications.

In Fig. 4, we can intuitively see the difference between the absolute value of Cb and Cr components in the flame region and the non-flame region. According to the comparison of many flame images, the threshold conditions of each component of flame colour in YCbCr space are in Eqn 6:

In Eqn 6, Y(x, y) is luminance value, Cb(x, y) is the blue chrominance signal value, and Cr(x, y) is the red chrominance signal value.

Fig. 4.

Absolute value histogram of the difference between Cb and Cr in the flame region.

First, the image is converted into a grey value image, set the pixel value of Cb and Cr difference between 110 and 150, and the rest is set to 0. After converting into binary image, the small hole in the image is removed to realise the flame segmentation. The specific process is in Fig. 5.

Fig. 5.

The process of flame segmentation based on YCbCr.

Architecture of proposed network

Due to the remarkable learning capabilities exhibited by convolutional neural networks, deep learning has gained widespread popularity in the field of object detection tasks. Object detection networks, commonly utilised, can be broadly categorised into two groups: (1) one-stage networks; and (2) two-stage networks. In contrast to two-stage networks, one-stage networks eliminate the necessity of generating proposal regions and directly predict the category and location of objects using the backbone network. This approach presents several advantages, including faster computation speed and lower computational costs, rendering it more suitable for real-world scenarios.

Inspired by the design principles of the YOLO network, we introduce a novel and cost-efficient one-stage convolutional network framework specifically tailored for real time fire detection in this study. The goal is to capitalise on the advantages of one-stage networks, such as their speed and efficiency, to devise an effective solution for real time fire detection scenarios. The comprehensive network model is in Fig. 6.

Fig. 6.

Overall network structure.

Feature extraction backbone

In the context of geographical location and weather conditions, mitigating the impact of environmental factors on the accuracy of fire scene detection, particularly in forest fire scenarios, is imperative. The simultaneous detection of both smoke and flames plays a crucial role in ensuring the prompt and precise communication of fire-related information. Traditional backbone networks typically rely on stacked convolutional blocks for feature extraction, but this often results in a substantial amount of redundant information, leading to decreased accuracy and slower convergence speed. Addressing this challenge and preserving the detection accuracy of lightweight backbone networks necessitates innovative approaches.

In contrast to the conventional practice of stacking convolution blocks, this paper adopts the DFC-C3ghost module (Fig. 7). This module comprises the ghost module and the decoupled fully connected (DFC) attention mechanism. The ghost module incorporates a residual structure, comprising three steps: conventional convolution, ghost generation, and feature map stitching. The DFC attention mechanism facilitates the generation of attention maps with receptive fields using fully connected layers with fixed weights.

Fig. 7.

The structure of DFC-C3ghost module.

A direct implementation of FC layer to generate the attention map is formulated as Eqn 7:

(7)ahw=∑h′,w′Fhw,h′w′⊙zh′,w′

where ⊙ is element-wise multiplication, F is the learnable weights in the FC layer, and A = {a₁₁, a₁₂, …, a_HW} is the generated attention map. The expression of the DFC attention mechanism is formulated as Eqn 8:

(8){a ′ h w=∑ h ′ = 1H F h,hw′ H ⊙ z h ′ w,h=1,2,…,H,w=1,2,…,W, a h w=∑ w ′ = 1W F h,hw′ H ⊙a ′ h w ′,h=1,2,…,H,w=1,2,…,W,

where F^H and F^W are transformation weights. The information aggregation process is in Fig. 8. It involves two parallel branches, namely the ghost module and the DFC attention module, extracting information from different perspectives while utilising the same input. The output is obtained by element-wise multiplication of the features from the ghost module and the attention from the DFC attention module. This combined output contains comprehensive information from both branches, allowing for a more holistic representation.

Fig. 8.

The information aggregation process of different patches.

ELAN-S neck

In this paper, we learned from the neck network of YOLOV7 and introduced the optimised PANet network. At the same time, ELAN module has been improved, and the improved ELAN-S module is in Fig. 9.

Fig. 9.

The structure of ELAN-S module.

Attention mechanism is widely used in convolutional neural networks. Traditional attention mechanisms, like CBAM, typically combine spatial attention and channel attention in parallel or sequential manners. However, in the human brain, these two forms of attention often work together synergistically. To achieve more effective attention, it is crucial to evaluate the importance of individual neurons. In neuroscience, information-rich neurons often exhibit distinctive firing patterns compared to neighbouring neurons. Additionally, the activation of neurons generally inhibits the surrounding ones, known as spatial inhibition. As a result, neurons with spatial inhibition should be assigned higher importance. The energy function is defined in Eqn 9:

(9)et(wt,bt,y,xi)=( y t−tˆ)2+1M−1∑i=1M−1( y o − xˆ i )2

where t̂ = w_tt + b_t, x̂_I = w_tx_i + b_t. Binary labels are used here, regular terms are added, and the final energy function is defined as Eqn 10:

(10)et(wt,bt,y,xi)=1M−1∑i=1M−1(−1−( w t x i+ b t))2+(1−( w tt+ b t))2+λwt2

The analytical solution of this formula is Eqn 11:

(11)wt=−2(t−ut)(t− u t )2+2σt2+2λbt=−12(t+ut)wt

and ut=1M−1∑i=1M−1xi,σt2=1M−1∑i=1M−1( x i− u t)2. Therefore, the minimum energy formula is Eqn 12:

(12)et⁎=4( σ ˆ2+λ)(t− u ˆ)2+2 σ ˆ2+2λ

The formula means that the lower the energy, the more distinct the neuron t is from the surrounding neuron, and the more important it is. Therefore, the importance of neurons is 1et⁎. And the input features are enhanced by Eqn 13:

(13)Xˆ=sigmoid(1et⁎)⊙X

where X is input feature and X̃ is on behalf of enhanced features. Based on this formula, SimAM is proposed and good results are obtained. SimAM deduces 3D attention weights for feature graphs without additional parameters.

We also placed the traditional CBS module with a GSConv structure. The GSConv module first takes an input and performs a regular convolution with downsampling. It then utilises DWConv (depthwise convolution) to further process the data. The results of both convolutions are concatenated together. Finally, a shuffle operation is applied to rearrange the corresponding channels from the previous two convolutions, placing them next to each other. In this way, the semantic information is protected effectively and the feature extraction ability of the model is improved.

Training

To evaluate the accuracy and effectiveness of the enhanced model, we utilised the PyTorch deep learning framework for model design and training. The trained model’s weights were then employed to predict forest flame and smoke images. The experiments were conducted using a GeForce RTX 4070 graphics card on the Windows 11 operating system. The details of our experiments are in Table 3.

The experiments were trained on our mixed dataset. Considering the GPU memory size and the cost of time, we set the batch-size of the model to 8, initial learning rate to 0.01. According to the SGD optimiser, we adjusted the learning rate to set weight decay to 0.0005. Training parameters of our model were designed based on the YOLOV7 shown in Table 4. In our work, we considered using pre-trained weights to accelerate model convergence and reduce retraining costs. However, due to the extensive modifications, we made to both the backbone and neck networks, the pre-trained weights from YOLOV7 did not significantly enhance the training speed. Therefore, we chose to train our network from scratch. Additionally, while we explored other methods such as knowledge distillation, we found that the performance gains were minimal and it significantly increased inference time.

Comparison and evaluation

To analyse and demonstrate the early forest fire detection performance of our improved model, we employed the Microsoft COCO evaluation metrics. These metrics are widely used for evaluating object detection tasks and provide a comprehensive assessment of the model’s performance. Our model was trained on the training set and evaluated on the validation set. The main metrics used for evaluation are Average Precision (AP) and Average Recall (AR), which are calculated based on Precision (P) and Recall (R). The formulas for calculating AP and AR, derived from Precision and Recall, are presented in Eqns 19–22:

where TP, FP, and FN are true positive, false positive, and false negative, respectively. In Eqn (22), the o indexes the IoU between the prediction box and the ground truth box.

We compare our model with typical detectors. As seen in Table 5, we use AP, Precision and Recall for comparation. AP_0.5 means average precision at IoU = 0.5. AP_s, AP_M, AP_L indicate the AP for small objects (area < 32²), medium objects (32² < area < 96²) and large ones (area > 96²). To sum up, our model is superior in AP_0.5, AP_0.5:0.95, Precision, Recall and FPS compared with other typical detectors. Comprehensive improvements have been made for the model to acquire better performance in detecting flames and smoke against the background of a forest fire. The respective accuracy of flame and smoke in this experiment is in Table 6. It can be seen that compared with YOLOV7, the current model with the highest accuracy, our model has improved the accuracy of both flame and smoke. We also made ablation experiments to explore the reasonableness and effect of the proposed improvement in Table 5.

Detection performance and analysis

Extensive experiments have demonstrated the satisfactory performance of our model in early detection of forest flame and smoke. When compared to the well-regarded YOLOV7, one of the top detectors currently available, our model excels in addressing the issue of false detection of flame and smoke. The detection results depicted in Fig. 10 illustrate the effectiveness of our model. In Fig. 10a and b, our model effectively addresses the challenge of missing small target flame detection. Similarly, in images (c) and (d), our model overcomes issues related to the missed detection of some miniature flame targets. Thanks to the improved extracted backbone, our model exhibits a robust ability to detect flame spots in photos. Consequently, our introduced model proves valuable for the early detection of forest fire flame information.

Fig. 10.

Results detected by (a, c, e, g) YOLOV7 and (b, d, f, h) our model.

In Fig. 10e and f, conventional detectors often overlook smoke that blends with the background. In contrast, our model adeptly captures such instances. Even for small smoke targets, our model not only detects them effectively but also demonstrates a high degree of confidence. In summary, the improved model proves highly effective in detecting flame and smoke across diverse backgrounds in forest fire scenarios.

Experimental results show that the detection speed of our model reaches 28 frames per second, which meets the requirements of real time monitoring. In the actual application of forest fire monitoring, the detection system can be deployed on the forest fire prevention watchtower, and the forest scene is captured by the monitoring equipment in the watchtower to realise the automatic monitoring of forest fire. In case of fire, timely alarm. Considering that the model size reaches 38.9 M, which is still larger and not suitable for deployment on small airborne devices, the model parameters can be reduced by pruning, quantisation, distillation, and other methods to facilitate deployment to edge devices. This research is in progress. It is important to emphasise that the computer vision model is an assistive technology to provide information assistance for forest fire rescue, helping relevant departments and personnel to take rapid response measures.

Results for smoke image, flame image, smoke flame mixed image and negative sample image are in Figs 11–14.

Fig. 11.

Detection results using our model, which are smoke pictures.

Fig. 12.

Detection results of our model, which are flame pictures.

Fig. 13.

Detection results of our model, which are flame mixed with smoke picture.

Fig. 14.

Detection results of our model, negative sample image similar to flame and smoke.