Monitoring tropical freshwater fish with underwater videography and deep learning

Study area and fish monitoring method at RUM

The study area included catchments and associated billabongs located in the ARR in the northern Australian wet–dry tropics (Fig. 1). Fish monitoring was conducted annually in two deep-channel billabongs, as well as several lowland backflow billabongs (herein referred to as backflow billabongs), of Magela and Nourlangie creek catchments. The billabongs lie within, or are surrounded by, the World Heritage listed KNP. Magela Creek originates in the Arnhem Land sandstone escarpment and plateau formation, flows through a defined sandy stream channel across lowland terrain, then connects and flows through diffuse pathways along a broad floodplain that discharges into the estuarine reaches of the East Alligator River. Nourlangie Creek similarly originates in the same sandstone formations, flows through sand channels and floodplain, before discharging into estuarine reaches of the South Alligator River. RUM (Fig. 1) is located adjacent to the lowland, mid reaches of Magela Creek and the mine operator manages a portion of its on-site pond-water inventory by releasing treated waters into Magela Creek during the wet season. Annual fish sampling was conducted from May to June, when billabongs approach hydrological isolation from the feeding stream system. The majority of data used for the present study were collected from the channel billabongs, exposed site Mudginberri (Magela Creek catchment) and control site Sandy (Nourlangie Creek catchment). Supplementary data for fish species not sufficiently represented in the channel billabongs were also collected from two backflow billabongs, Gulungul and Georgetown, both located in Magela Creek catchment (Fig. 1).

Fig. 1.

The locations of channel billabongs (Mudginberri and Sandy) and backflow billabongs (Gulungul and Georgetown) in relation to Magela and Nourlangie creeks.

In each of the two channel billabongs, 10 replicate unbaited video cameras (GoPro, models 5–9), 5 surface and 5 benthic, were deployed along each of five transects concurrently ~5 m apart, recording for a minimum of 65 min each. In each of the backflow billabongs, six replicate unbaited video cameras were deployed along each of five transects concurrently ~10 m apart, in the benthic position, for a minimum recording of 65 min each. All cameras in channel and backflow billabongs were oriented towards the bank or habitat structure. Each 65 min of footage was analysed sequentially by personnel trained in local fish identifications, with the maximum number of individuals of each species present in any one frame (MaxN) recorded as a semi-quantitative measure of abundance.

Ethics approval

This research was conducted with approval from the Animal Ethics Committee at Charles Darwin University, Northern Territory, Australia, under project number A20005.

Conceptual flow chart of methods

A conceptualisation of the steps used to derive a deep learning method to characterise fish assemblages from underwater videography, and to compare results to manual characterisation for monitoring, is provided in Fig. 2. Images used for fish identification and to train a CNN were annotated and labelled by trained observers from historical monitoring videos; these labelled images formed the training dataset (see ‘Training dataset’ section). Fish features used for identification, present in the training dataset, were used to train deep learning models in Azure Custom Vision (ver. 3.4, Microsoft, see https://azure.microsoft.com/en-us/products/cognitive-services/custom-vision-service/), with the models evaluated using precision, recall and mean average precision (see ‘Deep learning model training: Azure Custom Vision service’ section). An independent test dataset was derived for fish assemblage characterisation (MaxN), using both trained observers (see ‘Test dataset’ section) and deep learning (see ‘Model scoring and batch processing’ section). Comparisons between fish assemblages derived by trained observers and the general deep learning model were evaluated using MaxN-precision, MaxN-recall and MaxN-F1 (see ‘Evaluation metrics’ section), optimised with custom thresholds for accepting a model’s prediction (see ‘Optimising predictive thresholds’ section) and evaluated for monitoring potential impacts at RUM (see ‘Comparisons between fish assemblages’ section).

Fig. 2.

Conceptualisation of the steps used to derive a deep learning method for fish identification and its verification.

Training dataset

The imagery used for annotation and labelling was extracted from 392 underwater camera deployments collected from Sandy and Mudginberri billabongs between 2016 and 2019 (Supervising Scientist 2020). This imagery was supplemented with 180 camera deployments collected in July 2019 from two backflow billabongs, Gulungul and Georgetown. Videos were pre-screened for high levels of fish activity, then images were extracted from video files at a rate of 1 frame per second using customised open-source code in Python (see ‘Data availability’ section; Van Rossum and Drake 2010). This resulted in over 2 million frames for image annotation. Frames were manually examined and those without fish were discarded and those with fish present were annotated using a bounding box, then labelled by one of two experienced freshwater fish ecologists. All identifications were taken to species level where possible. It was not possible to consistently identify and separate eel-tail catfish (family Plotosidae) from within the genus Neosilurus (i.e. N. hyrtlii or N. ater), flat-head goby (family Gobiidae) from within the genus Glossogobius (i.e. G. aureus or G. giuris), gudgeons (family Eleotridae) from within the genus Oxyeleotris (i.e. O. selheimi or O. lineolate), nor distinguish fork-tailed catfish (family Ariidae) from within the genus Neoarius (i.e. N. graeffei or N. midgleyi) to species level from underwater imagery, so genus-level identification was used instead. Fig. 3 shows the 23 taxa of tropical freshwater fishes that were annotated and labelled.

Fig. 3.

Frames extracted from video footage demonstrating lowest taxonomic identification used, inter-species variability in fish morphology, differences in background and light conditions, and typical view of fish from underwater videography.

To standardise the annotation process and define ‘what makes a good label’ for a model’s intended purpose, the following criteria were developed:

Deep learning model training: Azure Custom Vision service

The Microsoft Azure Custom Vision service was used for image annotation, labelling and model training. Azure Custom Vision is an artificial intelligence (AI) service and end-to-end platform with a GUI that enables operators without coding experience to label and train deep learning models. As such, the operator is not required to select model architectures and tuning parameters. Based on the data, the platform optimises model training and hyperparameter tuning to maximise prediction accuracy. For this study and in the Azure Custom Vision platform, an object detection project that leverages multi-taxa tagging per image was created. Two ‘domain’ models were trialled, a larger general model that is optimised for a broad range of object detection tasks, and a smaller compact model for the constraints and advantage of real-time object detection on edge devices. The compact model can be exported and so can be run locally outside of Azure Custom Vision, whereas the general model is hosted on the program’s cloud platform and is presently not publicly available. The smaller compact model results in reduced accuracy and was trained for sharing purposes only. Results from compact model training are made available in see Supplementary Table S1.

When all images that met the annotation and labelling criteria were labelled, a model was trained using the advanced training feature in Azure Custom Vision for the maximum training time allowed at the time of this study, namely 96 h of computing time. A testing dataset was parsed from the training dataset, at an 80:20 train to test split, by the Azure Custom Vision platform and used during model training to calculate evaluation metrics.

Test dataset

An independent test dataset was derived from two-hundred eighty-six 1-h video files collected from channel billabong fish monitoring (Mudginberri and Sandy) in May 2020, 2021 and 2022 (Supervising Scientist 2021). All video files contributing to the test dataset were standardised to a 1-h analysis period, after discarding the first 5 min of imagery (representing settling time); the subsequent 1 h of imagery was processed manually by trained personnel and deep learning (see ‘Model scoring and batch processing’ section for the latter) to derive MaxN for the fish assemblage.

Model scoring and batch processing

The trained general model (from ‘Deep learning model training: Azure Custom Vision service’ section) was used to predict fish taxa from all videos in the test dataset and to calculate MaxN for each taxon in the model. For each camera location, the multiple video files were processed by sequentially extracting from each file a frame from each second of video. Each frame was assessed against the general model for object detection and taxa classification. For every frame, the model recorded the number of fish taxa detected, percent confidence in its prediction, and frame and video number. For every frame with a positive fish detection, an image visualising the prediction was saved. Azure Batch, a cloud-based distributed computing platform, was used to expedite model scoring of the ~1 million frames. For this study, video frames and code were deployed and processed in parallel across 30 nodes (Microsoft Azure Data Science Virtual Machine D4s powered by a 3rd generation Intel Xeon Platinum 8370C (Ice Lake)).

Statistical analysis

Deep learning model training results

A total of 44,112 images representing 23 fish taxa were annotated with 82,904 bounding boxes and labelled (Table 1). mAP, precision and recall for overall model performance were 83.6, 81.3 and 89.1% respectively (Table 1). Of the 23 taxa used for model training, 11 taxa recorded average precision (AP) >90%, half of which had >8000 annotations per taxa. The remaining 12 taxa recorded AP between 29 and 87% with total number of annotations ranging between 20 and 3033. Banded grunter (Amniataba percoides) had the highest AP (96.3%) and Black bream (Hephaestus fuliginosus) had the lowest AP (29.0%).

Optimising decision thresholds

Thresholds ranged between 70 and 99.9% for recording a MaxN (Fig. 4), whereas these varied across fish taxa, some trends were evident. For taxa low in abundance in test videos, MaxN estimates had greater uncertainty because of poor model detectability. This is evident in the five taxa (Glossogobius spp., Leiopotherapon unicolor, Melanotaenia nigrans, Mogurnda mogurnda and Oxyeleotris spp.) that did not record a MaxN. These taxa were constrained at 99.9% threshold and individually contributed <1% to fish assemblage composition (see Supplementary Table S2). Although lower thresholds would have recorded a MaxN, as evident in the higher MaxN-recall at decreasing thresholds (Fig. 4), the MaxN-precision remains low (<30%). This would result in more FP than TP detections, thus increasing the chance of type 1 error. This logic was applied to other low abundant taxa such as Scleropagaes jardinii and Nemetalosa erebi where the highest MaxN-F1 resulted in similar counts of FP and TP detection. A 70% threshold was allocated to one taxon, H. fulifinosus, the only fish that recorded 100% MaxN-Precision at a threshold below 99.9%. For all taxa, the greatest MaxN-recall and lowest MaxN-precision was calculated at 20% threshold. The highest MaxN-F1 score of 96.1% was calculated for A. percoides, with a threshold of 95%, where the model and trained observer MaxN matched 100% of the time. Taxa with a high number of annotations (>2000) were allocated a high threshold (≥95%). The most abundant fish in videos and with the greatest number of annotations were Ambassis macleayi, A. percoides and Craterocephalus stercusmuscarum, which had respectively 16,105, 12,199 and 3033 annotations and thresholds between 95 and 98%. From the assemblage derived with optimal thresholds, the mean MaxN-F1, MaxN-precision and MaxN-recall scores across 286 videos were 54.4, 76.5 and 56.6% respectively (see Table S2).

Fig. 4.

Taxa-specific measures of MaxN-F1, MaxN-precision and MaxN-recall (%) at increasing confidence thresholds for accepting the model’s output probability. Measures were derived from mean per video MaxN for deep learning and trained observers. Asterisks (*) denote the optimal threshold.

Trained observer v. deep learning for monitoring at RUM

PERMANOVA results showed significant differences between methods (deep learning and trained observer) (Pseudo-F = 14.977, P_(MC) = 0.0002), billabongs (Sandy and Mudginberri) (Pseudo-F = 3.6648, P_(MC) = 0.0244) and among the years (Year) (Pseudo-F = 5.6761, P_(MC) = 0.0001) being compared. PCO of axes 1 and 2, cumulatively representing 55% of the variation in Bray–Curtis similarity (Fig. 5), showed clear separation of samples by method. This is evident across the x-axis and is primarily due to a larger, trained-observer-derived MaxN for abundant and, often schooling, taxa (e.g. C. stercusmuscarum, A. macleayi, Neosilurus spp. and Melanotaenia splendida inornata) (Fig. 6). The interactions between method and year (Pseudo-F = 2.7674, P_(MC) = 0.0011) and billabong and year (Pseudo-F = 3.6875, P_(MC) = 0.0002) were also significant, indicating that difference between methods and the difference between billabongs varied across years.

Fig. 5.

Principal coordinate ordination (PCO) of fish assemblages (mean MaxN per transect) at Mudginberri and Sandy billabongs derived using deep learning and trained observers in 2020, 2021 and 2022 (n = 10). Transect numbers are denoted above each symbol.

Fig. 6.

Taxa-wise mean MaxN using deep learning and trained observers (n = 286).

There was no significant difference for the remaining factors and their respective interactions (method × billabong and method × billabong × year, see Supplementary Table S3). Although differences in MaxN were observed between methods, the non-significant interaction between method × billabong × year indicates the distribution of taxa and their relative abundances in samples determined by each method reflect the same relative changes between billabongs over 3 years. This is shown in the PCO where the assemblages for each billabong and year are otherwise arranged in a similar pattern on their respective sides of the ordination, namely strong concordance between methods on the y-axis of the PCO plot for separation of billabong and year (Fig. 6).

Deep learning for fish monitoring

Deep learning is a valuable tool for automating the measurement of tropical freshwater fish assemblages, and when used in a real-world monitoring scenario such as the present study, produced results that approached those of trained observers. To the best of our knowledge, the training dataset we developed is the largest and most diverse generated for tropical freshwater fish worldwide, and was produced using a low-code, user-friendly tool. Deep learning was able to effectively detect spatio-temporal patterns in fish assemblages, expanding enormously future applications in the assessment and measurement of fish communities. In particular, with automated fish counting systems, environmental managers can increase the number of replicates used both temporally and spatially to sample an area and generate more detailed ecological datasets. Additionally, new measures of abundance can be explored such as MeanCount – the mean number of fish detected in a series of frames over an analysis period (Schobernd et al. 2014) – due to the ease of collecting fish abundance data for any time interval of video.

If deep learning and automated systems for processing underwater fish videography are to supplement or replace conventional traditional manual methods that rely on high levels of human interaction, deep learning must provide comparable ecological information. Tropical freshwater fish assemblages are dependent upon a range of ecological processes, from the flood pulse concept (Junk 1999) to changes in habitat structure (Buckle and Humphrey 2013). This study aimed to determine if assemblages measured by deep learning capture the same trends and variation through space and time as data derived from trained observers. Our results indicate significant differences between deep learning and trained observers for deriving MaxN of multiple fish taxa in underwater video. A significant difference between methods is not unexpected because of the complexity of fish detection and classification in underwater videography and was evident in the separation of samples in multivariate space, in particular, across the x-axis in the PCO (Fig. 5). By contrast, there was no significant interaction in method, billabong and year from PERMANOVA testing. That is, the distribution of taxa and their relative abundances in samples determined by deep learning and trained observers reflect the same relative changes between Mudginberri and Sandy billabongs over a 3-year period. The limitations presented by deep learning were primarily lower average species-wise MaxN measures for some taxa and inability to detect rarer taxa because of lack of specific types of training data, as described below. Regardless, we have demonstrated the use of deep learning in a real-world fish monitoring program, and present results that show where efforts to improve the deep learning model can be targeted to close the gap between trained observers and deep learning.

Our study’s overall mAP of 83.6% was comparable to other multi-species fish object detection F1 scores from test data of 79.8% for 14 taxa (Jalal et al. 2020) and 83.3% for 4 taxa (Knausgård et al. 2022). However, these studies used a two-staged detector approach where object detection models were trained on one category, ‘fish’, then detections of fish were assessed against an image classification model. Our results suggest that similar performance can be achieved on novel test data using a single-stage multi-taxa object detection model. We present the first mean per video MaxN-F1 score between trained observers and deep learning of 54.4% for tropical freshwater fish. On average, deep learning’s estimate of fish abundance was approximately half that of a trained observer. This difference in accuracy between the model’s training metrics (e.g. 83.6% mAP) and its performance in real-world situations (e.g. 54.4% MaxN-F1) emphasises the need to validate a model against the specific analytical routines it will be used for in practice. It also highlights the importance of understanding any limitations of the model, such as underestimation of abundance, that may not have been apparent in the standard training, validation and testing process.

Thresholds

We present an approach to define the optimal threshold for deriving MaxN per taxon. This was highly effective for taxa with lower volumes of training data and was necessary to reduce the number of FPs. Thresholds for accepting the model’s output probability were highly variable among taxa. Given the differences in training data size and the subsequent precision, recall and mAP results for each taxon from model training, the range in thresholds between 70 and 99.9% were expected and consistent with multi-taxa fish classification thresholds derived elsewhere (Villon et al. 2020). Constraining the prediction thresholds for each taxon improved the model’s overall performance. Our results indicate the importance of deriving taxon-specific thresholds and highlight the implications and limitations of using one threshold for accepting multi-taxa deep learning model predictions. For example, if 95% acceptance criteria were applied to all model predictions taxa such as A. percoides would record the optimum balance of precision and recall. By contrast, taxa such as Ambassis agrammus would sacrifice precision for recall and record 143 FP predictions at 95% threshold compared to 3 FP predictions at 99.9% threshold, which was calculated as the optimum balance between precision and recall.

Deep learning and MaxN

The fish monitoring program designed for impact detection at RUM is based on an indicator approach that measures a subset of ‘common’ species in the knowledge that no single method can capture all species, and all monitoring methods have their inherent biases. Analyses of fish assemblage data using Bray–Curtis similarity places greater emphasis on the numerically dominant taxa and less weight on uncommon taxa. Overall, when the deep learning model was applied to test data to derive MaxN for real-world monitoring of fish, the evaluation metrics were lower compared to those derived during model training with Azure Custom Vision. As the model results were optimised for precision when selecting thresholds, and consequently the MaxN-F1, this result was expected and is commonly encountered when applying deep learning to real world scenarios. For the most abundant fish, A. macleayi, A. percoides and C. stercusmuscarum, the MaxN-F1 score was similar to or greater than the model’s mAP. This may be explained by the way MaxN is used as a measure of abundance. MaxN has become the common metric used for quantifying fish abundance from underwater video (Cappo et al. 2001; Whitmarsh et al. 2017). Although it is a conservative estimate of abundance (Campbell et al. 2015), MaxN has proved its capability in detecting spatio-temporal variability in fish assemblages and ecological nuances in fish data (Whitmarsh et al. 2017).

For deep learning to be successfully used for MaxN estimates, the model must be able to confidently identify one frame out of the 3600 possible frames in a 60-min analysis period for which the maximum number of fish of a particular taxon are present. The MaxN may occur multiple times in a 60-min video and the model may perform differently based on how the fish are displayed in those frames. If the MaxN was recorded in a frame where most fish were deep in the field of view or were obscured by vegetation, then the model may not detect all individuals present in the frame and will underestimate MaxN. Conversely, if all fish are clearly in view, and therefore representative of the training data, it is more likely the model will confidently return the correct MaxN. In this study, we observed consistently lower measures of MaxN for abundant schooling taxa presenting deep in the field of view or obscured by vegetation. There is the potential that lower measures of an already conservative abundance estimate may reduce the ability to detect changes in fish assemblages for monitoring purposes. For monitoring at RUM, a number of numerically dominant fish species have greater abundances in the exposed (Mudginberri) site than the reference (Nourlangie) site (e.g. M. splendida inornata, C. stercusmuscarum and A. macleayi). Underestimates of these numerically dominant taxa increase the possibility of a type 1 error because of the method’s artefact in the down-weighting of their abundances in the exposed site. Although the distribution of taxa and their relative abundances in samples determined by deep learning and trained observers reflect the same relative changes between Mudginberri and Sandy (non-significant three-way PERMANOVA interaction), significant differences between methods were observed using PERMANOVA. The implications of such artefacts are yet to be assessed. Assessment would entail statistical comparisons of multi-year data derived from deep learning and trained observers, with possible data manipulation to mimic impact upon numerically dominant taxa. In parallel, improved deep learning detection of schooling numerical dominants is also required.

Currently, one of the greatest challenges with deep learning for fisheries estimates is access to high quality, annotated and labelled training data that cover all taxa expected in a test environment (Villon et al. 2022). While such datasets are desirable, considerable efforts may be required to curate and achieve high performance across all taxa for monitoring purposes, particularly all uncommon species. As our aim was to develop a model that could accurately detect the MaxN for each taxon in a 60-min analysis period, several considerations were made around annotation and labelling to optimise performance. Bounding box annotations were chosen over polygons because of the speed and efficiency for generation of a bounding box dataset. Polygons are preferred when the objects of interest are often overlapping or close to each other and where use of bounding boxes would result in high degrees of overlap. However, polygons require significantly more resources and can cost twice as much as bounding boxes to annotate (Mullen et al. 2019). Fisheries monitoring in marine ecosystems using videography often use baits to attract fish to the camera, resulting in imagery with overlapping fish (Ditria et al. 2021; Harvey et al. 2021). For videography in freshwater ecosystems that do not use baited bags, the density of fish observed per frame can be lower because of the lack of an attractant, resulting in more frames with individual fish spread across the field of view rather than overlapping. In these scenarios the less resource-intensive bounding boxes are adequate.

Training data

Generating a training dataset from video with fish moving in 3-D space can be a significant challenge for deep learning in terms of distinguishing all taxa and their identifying features (Jalal et al. 2020). Precision, recall and overall accuracy increase logarithmically with increasing number of images when training deep learning models (Shahinfar et al. 2020). Therefore, to maintain high taxa detection accuracy across multiple species classes, a neural network must be trained with enough labelled images across a range of poses, orientations, lighting conditions and body size when moving through the field of view, to return a uniform and consistently accurate score when making inferences on unseen footage. Although the dataset generated in this study has reasonably few images of low-abundant taxa, we were still able to build an understanding and present results on taxa-specific, deep learning performance. Our results indicate that taxa that are comparatively unique, such as A. percoides with distinct black and white bands across its body (Fig. 3) and low morphological variation between sexes, reproductive cycle and life stage (Bishop et al. 2001), could achieve a high MaxN-F1 score (96%) on a test set when trained with >12,000 annotations. Conversely, for a similar number of annotations for taxa with high intra-species variation, such as M. splendida inornata, there was reduced accuracy on a test set (81%) and a higher number of FP detections. Factors such as changes in colour during reproductive cycle (Crowley and Ivantsoff 1982) and a preference for habitat near the surface of the water column with associated influences of poor light and water clarity introduce additional noise into the imagery that must be learnt.

Conservation and biodiversity assessment

Missing taxa from a model prediction has implications when monitoring metrics rely upon taxa richness in impact or conservation and biodiversity assessments. This includes consideration of taxon-specific migration responses to the magnitude of wet season rainfall, which may see the effect of missing taxa exacerbated in years where movement of typically low-abundant fish in channel billabongs is enhanced by antecedent wet season productivity (Crook et al. 2020, 2023). In other instances, usually uncommon taxa may dominate in stochastic fashion at decadal intervals. Until training datasets are improved, deep learning as a monitoring tool should be used with these considerations in mind. To apply this learning to conservation and biodiversity assessments, it is important to prioritise the generation of training data for uncommon or rare species. Until such data is available potential procedures during QA/QC of biodiversity data may mitigate the effect of low abundant taxa, such as visual interrogation of model predictions to confirm the correct taxa are assigned for MaxN measures, which is more time effective than traditional video processing methods. Furthermore, where there is an overlap in taxa similarity, for example L. unicolor and juvenile Lates calcarifer can be mistaken because of their similar body shape, the model may incorrectly assign taxa into the similar category (Vuttipittayamongkol et al. 2021). This will appear as an overestimate in abundance and may be obvious in assemblage composition data, warranting further investigation and access to additional training data to improve the model. The low-abundant taxa were also noted by trained observers as having morphological overlap to each other. For example, L. ordensis, M. cyprinoides and N. erebi are all taxa that often appear elongated when swimming through the water and have similar silver-coloured bodies (Fig. 3). As the impact of training data imbalance on model performance is influenced by the presence of category overlap (Vuttipittayamongkol et al. 2021), other approaches may be required to confidently detect such taxa until training data can be improved. Although not tested here, some promising methods to improve training data imbalance include data augmentation with generative adversarial networks used to generate ‘fake’ data and balance the dataset (Ali-Gombe et al. 2019), and ensemble modelling where multiple models are trained for subsets of the taxa and test imagery is scored against all models. With the latter, significant increases in computational requirements are expected.

Although this study did not focus on optimising model performance through different data augmentations, downsampling or upsampling, by making the full annotated and labelled dataset publicly available, further research on the implications of imbalance for fish taxa detection is possible.

Taxa-specific response to deep learning

This study identified that MaxN abundance was lower for the deep learning method than for trained observers for most fish species. This was most evident in schooling taxa such as A. macleayi, C. stercusmuscarum, M. splendida inornata and Neosilurus spp., which had MaxN abundance two or more times greater for trained observers (Fig. 6). One possible explanation for this observation is that schools are often present at a range of distances relative to the camera, and individuals in the background of the image often returned confidence probabilities that did not meet the threshold set for that fish. In comparison, individuals closer to the camera lens, in clear focus and with key identifying features in view, were confidently predicted and recorded a MaxN value. Additionally, when multiple fish in view are orientated differently towards the camera, predictive performance changes. This is observed for A. percoides in Fig. 7d where, in the first (left) frame, one fish moves away from a lateral view and the bandings that distinguish this species become obscured. When facing directly toward the camera, confidence drops to 86.6% indicating that for this frame it would be below the prediction threshold of 95% and not be included in a MaxN estimate. However, because the sequence of frames will eventually show all three fish in a lateral, readily identifiable view (central and right frames of Fig. 7d), the prediction will exceed the threshold of 95% for accepting model confidence probability, and the MaxN of 3 will be scored. The same holds for A. macleayi; among other features, ichthyologists use the dark patch, often appearing as a black dot, positioned at the base of the pectoral fin for species identification. This feature separates this species from other ambassid species (Pusey et al. 2004). As the fish moves to show a lateral view with the black patch in combination with a clearer view of the body structure, the confidence probability increases from 72.2% (central frame, Fig. 7e) to 99.2% or greater (left and right frames, Fig. 7e).

Fig. 7.

Spatio-temporal variation in deep learning predictions of fish. Bounding boxes represent predictions made above the threshold for acceptance (green), below the threshold but equal to or greater than 80% (white), less than 80% (yellow) and FP predictions (red). Variation in predictions of schools deep in the field of view are depicted for C. stercusmuscarum (a, b), A. macleayi (c) and A. percoides (d). Changes in prediction confidence as fish move in 3-D space are depicted in a sequence of frames for A. percoides and A. macleayi (e).

Our results suggest that the model is learning the morphological features ichthyologists use for taxonomic identification of fish, and that this form of interpretation of deep learning predictions, where ecology and data science intersect, is critical to inform how deep learning is best applied to automating fish monitoring and where effort is optimised on improving performance. For example, to improve the accuracy of M. splendida inornata described previously, the collection of training data can focus on the morphological variants that are unequal within the distribution of samples. One approach is that taxa can be separated into multiple categories if the intraspecies variation is divergent enough to warrant such a split. We have identified taxa with the same number of annotations that were learnt by deep learning at different precisions. This may be because some taxa are more difficult to learn because of their complex morphology, inter-species similarity, intra-species variation and behaviour that can influence the ability to identify taxa across test scenarios. Notwithstanding and despite the underestimation of MaxN using deep learning, we provide evidence that the approach preserved the relative fish assemblage differences between both billabongs and among years. In these respects, results are similar to those derived by a trained observer when making comparisons between assemblages.