Image-based recognition of parasitoid wasps using advanced neural networks

Dataset

The dataset used for automated classification includes 11 genera of parasitoid wasps, of which 10 belong to the family Diapriidae and subfamily Diapriniiae. Only one taxon, the genus Ismarus, is from the family Ismaridae. Both the Diapriinae and Ismaridae were selected for the proof of concept because the diversity, while still challenging, is significantly less incomprehensible and the identification less demanding than for the more diverse and abundant subfamily Belytinae. The specimens were mostly collected in southern Germany, mainly in Bavaria. Since 2011, Malaise traps have been set regularly to cover various (even the most specialised) habitats, ranging from private gardens to the high alpine region. A complete list of evaluated specimens and associated location data are available in Hübner and Shirali (2024). A standardised integrative taxonomic approach consisting of DNA barcoding and morphology was used to identify the specimens: specimens were preliminary identified (to genus if possible and sex) and sequenced (Padial et al. 2010; Schlick-Steiner et al. 2010; Chimeno et al. 2023). The Sanger sequencing of the preliminarily identified material was conducted at the CCBD in Guelph, Canada (see https://ccdb.ca/) using a voucher recovery approach. Genetic results were uploaded to the BOLD platform (see https://www.boldsystems.org/) for cross-referencing. After the molecular analysis, all questionable specimens were re-evaluated morphologically. Images of other hymenopteran species were pooled into another group, ‘other Hymenoptera’, comprising 121 images of other Hymenoptera such as Braconidae, Ichneumonidae, Chalcidoidea and also some Diapriidae that did not belong to the 10 previously mentioned genera because these belonged to the subfamily Belytinae. The word ‘class’ hereinafter will refer to target groups that belong together and are to be sorted. This does not refer to the taxonomic hierarchical term.

We employed two systems for image capturing: an Olympus camera E-M10 with a Novoflex Mitutoyo Plan Apo 5× microscope lens, controlled by OM Capture software (ver. 3.0, see https://www.om-digitalsolutions.com/en/) was used to take deep-focused images by stacking 70–130 individual images; and we took images with a prototype of the Entomoscope (Wührl et al. 2024). All specimens were photographed in ethanol, mimicking the light and sample conditions used for the DiversityScanner. All images were subsequently stacked using Helicon Focus (ver. 8, see https://www.heliconsoft.com/heliconsoft-products/helicon-focus/). We used 2257 colour images in our study, as summarised in Table 1. One additional test dataset, including non-Hymenoptera specimens, has been curated to evaluate our pipeline’s performance to exclude non-target species using an outlier detection model. This step is vital to avoid misclassifications in practical applications, such as mistakenly identifying a honey bee (Apis) as a target Hymenoptera species. Detailed taxonomy and the number of images in these test datasets are presented in Table 2. DNA barcoding and morphological (expert knowledge) methods were applied to identify the species. All images are available in Hübner and Shirali (2024).

Data preprocessing

In the computer vision field, the efficiency of model training and classification accuracy is significantly influenced by the quality and preparation of input images. This section delineates the preprocessing steps to prepare the insect image dataset for effective machine-learning model training.

Crop and resize using Grounding DINO

To enhance the model’s focus on the insect and to minimise background noise, images are first cropped to the Region of Interest (ROI) using the Grounding DINO model (Liu et al. 2023), as depicted in Fig. 1. This model employs a zero-shot object detection approach, leveraging image and text features to predict bounding boxes around the insect based on the text prompt ‘Insect. Wasp. Wings.’ with a box threshold of 0.29 and text threshold of 0.25. These cropped images are resized to a uniform size of 224 × 224 pixels. This standardisation step preserves critical insect features for further processing.

Fig. 1.

Object detection using Grounding DINO with subsequent cropping is visualised.

Deep learning model architectures

Three different deep learning models were selected and evaluated in this study: ConvNeXt (Li et al. 2022), BEiTv2 (Peng et al. 2022) and YOLOv8 (G. Jocher, A. Chaurasia and J. Qui, see https://github.com/ultralytics/ultralytics). These models were selected for proficiency in handling complex computer vision tasks, particularly in identification and classification. Our approach is grounded in transfer learning and fine-tuning methodologies, ensuring that the models are adapted to our specific requirements.

ConvNeXt XLarge (Li et al. 2022) is an advanced convolutional neural networks (CNNs) variant known for the exceptional feature extraction capabilities. This incorporates multiple layers designed to process and interpret intricate image details, leveraging advanced activation functions and optimisers to ensure efficient learning and high classification accuracy. The model supports multi-label classification with a sigmoid activation function and handles an image size of 224 × 224 pixels with a batch size of 32 and stochastic depth regularisation with a rate of 0.3. The class weights are similarly adjusted for genera classes. The second model is BEiTv2 (Peng et al. 2022), a Transformer-based model adapted to understanding and interpreting complex image patterns. The unique attention mechanism is instrumental in identifying subtle variations within images, making this a crucial tool for ensuring model stability and robustness under diverse imaging conditions. The model processes images of 224 × 224 pixels with a batch size of 32 and employs a dropout regularisation of 0.3 applied to Attention-MLP (multilayer perceptron) blocks. Class weights for genera classes are weighted by a factor of three. The third model is YOLOv8, the latest iteration in the YOLO (You Only Look Once) series, selected for the rapid object detection capabilities that are also suitable for classification tasks. The architecture, balanced for speed and accuracy, makes this ideal for real-time applications for which immediate, precise classification is essential. Owing to the framework’s limitation in not supporting multi-label classification, we train two separate models, one for genus classification and one for sex determination. Both models leverage ImageNet pre-training weights (Russakovsky et al. 2015) and all layers are made trainable, an approach that maximises learning from our dataset. These models are designed for multi-output classification, utilising a softmax activation function. The models are capable of processing larger images of 640 × 640 pixels, operating with a batch size of 64 and incorporating a dropout regularisation of 0.3. The class weights for both models are set to default. This configuration ensures optimal performance and accuracy in our classification tasks. In conclusion, the architecture of each model has been tailored to meet the specific requirements of this project. ConvNeXt’s advanced convolutional approach, BEiT’s attention-based mechanism, and YOLO’s speed and precision collectively contribute to the successful implementation of the classification tasks in this study.

Training setup and process

A standard personal computer with a powerful NVIDIA RTX 4080 GPU was used with Python (ver. 3.10), TensorFlow (ver. 2.10.1), PyTorch (ver. 2.0.1), Keras, CUDA (ver. 11.7) and Anaconda software was used for classification. This integrated environment provides the efficiency and flexibility to train deep learning models. During the training process, all three models were trained for a maximum of 150 machine-learning epochs using the AdamW optimiser with a consistent learning rate of 0.001. We employed a four-fold cross-validation approach to optimise model performance. This allowed us to assess the models’ performance on different subsets of the data, mitigating the risk of overfitting and providing a more robust evaluation of the generalisation capabilities. In addition to cross-validation, we also applied early stopping, model check pointing, and learning rate reduction techniques, with training progress monitored. Notably, model weights were saved whenever improvements were observed during validation. BEiTv2 and YOLOv8 utilised categorical cross entropy for loss functions, whereas ConvNeXt employed binary cross entropy.

Outlier detection

An algorithm for automatic classification is expected to reliably differentiate between insects that belong to the predefined classes for classification and specimens that do not belong to these classes. To enhance this capability, we implemented a preliminary filtering stage using an outlier detection model prior to our main image classifier. This allowed the automatic filtering of collections that had not previously been presorted for the predefined classes. This outlier detection model classified a specimen into one of the two groups, ‘Hymenoptera for classification’ and ‘Non-Hymenoptera’. ‘Hymenoptera for classification’ includes specimens within the Hymenoptera genera we targeted for detailed analysis. The second group, ‘Non-Hymenoptera,’ consisted of all other insect specimens that do not belong to the order Hymenoptera. This broad category includes a variety of insects, some examples of which are provided in Table 2 as other insects. This prefiltering is carried out by a one-class support vector machine (OCSVM) based on the BEiTv2 – a pretrained deep learning model with ImageNet weights. Da Silva Puls et al. (2023) have demonstrated that ViTs perform best for this task. During this process, the classification layer is removed, leaving the model to serve as an effective feature extractor. This model transforms the input images into a lower-dimensional feature space, capturing low-level and high-level image features. Subsequently, a OCSVM on these feature representations extracted from the training dataset is trained. Any new testing data that falls within the boundary of the OCSVM is assigned to the trained class and data points outside the boundary are declared as outliers or Non-Hymenoptera.

Principal Component Analysis (PCA) is subsequently employed to reduce the dimensionality of the data from 1024 to 128 features per image to maintain data quality while reducing computational complexity. In this next step, the data are normalised using the mean and variance of the training dataset. Subsequently, the OCSVM is trained on the reduced, normalised feature representations. This approach does not involve training a neural network, therefore there is no need for a separate validation dataset. Instead, the validation dataset is combined with the training dataset for training the OCSVM on the positive class, making this suitable for detecting outliers that, in this context, are the other insects.

The entire approach is implemented using the open-source machine learning library Scikit-learn (Pedregosa et al. 2011). Parameter tuning is performed through a grid search to optimise the OCSVM’s performance.

Classification performance metrics

The performance metrics for genus and sex classification of the three different deep learning (DL) models are provided in Table 3. The performance metrics include the test classification accuracy and the F1-score for the best model selected across four training runs using fourfold cross-validation.

The performance metrics show that BEiTv2 consistently outperforms the other models in genus and sex classification tasks. ConvNeXt XLarge also exhibits strong performance, while YOLOv8 performs competitively, albeit with lower accuracy and F1-score than in the two other models. For this reason, only the classification results of the best-performing model, BEiTv2 are presented below.

The classification results for the 11 predefined classes of Hymenoptera and one ‘Other_Hymenoptera’ class are depicted in a confusion matrix in Fig. 2 and 3 for the tasks of genus and sex classification.

Fig. 2.

Confusion matrix with genus classification results of the BEiTv2 model.

Fig. 3.

Confusion matrix with sex classification results of the BEiTv2 model.

In addition, the graphs of the classification results for training and validation accuracy, and loss for the BEiTv2 model are given in Fig. 4, 5 and 6. These figures represent the best fold of the cross-validation training process. These provide a comprehensive view of the model’s learning progress throughout training, illuminating the overall performance and convergence behaviour.

Fig. 4.

Smoothed training (orange) and validation (blue) genus accuracy, BEiTv2, with the original graph transparent. Note: ‘Epoch’ refers to a machine-learning epoch.

Fig. 5.

Smoothed training (orange) and validation (blue) sex accuracy, BEiTv2, with the original graph transparent. Note: ‘Epoch’ refers to a machine-learning epoch.

Fig. 6.

Smoothed training (orange) and validation (blue) combined genus and sex loss, BEiTv2, with the original graph transparent. Note: ‘Epoch’ refers to a machine-learning epoch.

The figures show a steady increase in accuracy and corresponding decrease in loss, suggesting the model is learning effectively. Notably, the close alignment of the training and validation curves indicates that the model is not overfitting, performing similarly on both seen and unseen data. Moreover, the absence of a plateau in improvement or a significant gap between training and validation performance suggests that underfitting is not occurring. Hence, the model exhibits a balanced learning trajectory, suggesting robustness and reliability when applied to similar unseen data.

Class activation maps

Class Activation Mapping (CAM) (Zhou et al. 2016) is a technique used for generating heat maps to highlight class-specific regions of images that impact the classification result. In Fig. 7, heat maps for two different insect specimens are provided as examples: the genus Paramesius (top) and Spilomicrus (bottom). The left side represents heat maps associated with the predicted genus. The antennae, head and thorax are consistently significant in predicting the genus. On the right side, the heat maps related to sex prediction are displayed, in which the antennae are crucial for sex prediction. These results show that the classification algorithm considers features similarly to how a taxonomic expert would.

Fig. 7.

Class activation heatmaps for genus classification (left) and sex classification (right). Red areas indicate regions with higher weighting in the classification.

Identification of non-target Hymenoptera

The outlier detection method was assessed using two different test datasets: one is described in Table 2 and the other is a split of our main dataset. The results are visualised in Fig. 8. The method misclassified 23 of the total of 652 resulting images.

Fig. 8.

Confusion matrix for outlier detection.

In the context of our study, ‘inliers’ are images that the outlier model correctly identifies as belonging to the category of Hymenoptera but not necessarily to the specific target genera of Diapriidae that are our classification focus. Conversely, ‘outliers’ are images that do not belong to the category of Hymenoptera and are therefore beyond the focus of our model’s training criteria.

Notably, this approach achieved 100% accuracy on the test split of our dataset as expected because our outlier model was trained specifically on this dataset. Regarding the ‘Other Insects’ images, this model demonstrated prowess by identifying 90.6% of the images as outliers. This indicates the model’s ability to distinguish these insects from Hymenoptera effectively. The Diapriidae and Belytinae images, as part of ‘Other Hymenoptera,’ presented a unique challenge. There were variations in image quality, background and differences in camera sources. Despite these challenges, our model detected 82.7% of the images as inliers, underscoring the potential for accurate classification even under adverse conditions. Overall, these results demonstrate the model’s robustness and accuracy in classifying closely related but non-target Hymenoptera species, even under non-ideal conditions.