Monitoring tropical freshwater fish with underwater videography and deep learning
Andrew Jansen
A * , Steve van Bodegraven A , Andrew Esparon A , Varma Gadhiraju B , Samantha Walker A , Constanza Buccella A , Kris Bock B , David Loewensteiner A , Thomas J. Mooney A , Andrew J. Harford A , Renee E. Bartolo A and Chris L. Humphrey A
A
B
Abstract
The application of deep learning to monitor tropical freshwater fish assemblages and detect potential anthropogenic impacts is poorly understood.
This study aimed to compare the results between trained human observers and deep learning, using the fish monitoring program for impact detection at Ranger Uranium Mine as a case study.
Fish abundance (MaxN) was measured by trained observers and deep learning. Microsoft’s Azure Custom Vision was used to annotate, label and train deep learning models with fish imagery. PERMANOVA was used to compare method, year and billabong.
Deep learning model training on 23 fish taxa resulted in mean average precision, precision and recall of 83.6, 81.3 and 89.1%, respectively. PERMANOVA revealed significant differences between the two methods, but no significant interaction was observed in method, billabong and year.
These results suggest that the distribution of fish taxa and their relative abundances determined by deep learning and trained observers reflect similar changes between control and exposed billabongs over a 3-year period.
The implications of these method-related differences should be carefully considered in the context of impact detection, and further research is required to more accurately characterise small-growing schooling fish species, which were found to contribute significantly to the observed differences.
Keywords: artificial intelligence, biodiversity, channel billabong, computer vision, convolutional neural network, Kakadu National Park, object detection, Ramsar wetlands.
References
Ali-Gombe A, Elyan E, Jayne C (2019) Multiple fake classes GAN for data augmentation in face image dataset. In ‘2019 International Joint Conference on Neural Networks (IJCNN)’, 14–19 July 2019, Budapest, Hungary. Paper N-20152. (IEEE) doi:10.1109/IJCNN.2019.8851953
Almeida JS (2002) Predictive non-linear modeling of complex data by artificial neural networks. Current Opinion in Biotechnology 13(1), 72-76.
| Crossref | Google Scholar | PubMed |
Alshdaifat NFF, Talib AZ, Osman MA (2020) Improved deep learning framework for fish segmentation in underwater videos. Ecological Informatics 59, 101121.
| Crossref | Google Scholar |
Australian Institute of Marine Science, University of Western Australia, Curtin University (2020) OzFish dataset – machine learning dataset for baited remote underwater video stations. Available at https://apps.aims.gov.au/metadata/view/38c829d4-6b6d-44a1-9476-f9b0955ce0b8 [Dataset, verified 1 February 2020]
Bishop KA, Pidgeon RWJ, Walden DJ (1995) Studies on fish movement dynamics in a tropical floodplain river: prerequisites for a procedure to monitor the impacts of mining. Australian Journal of Ecology 20(1), 81-107.
| Crossref | Google Scholar |
Buckle D, Humphrey C (2013) Monitoring using fish community structure. In ‘eriss research summary 2011–2012’. (Eds RA van Dam, A Webb, SM Parker) Supervising Scientist report 204, pp. 97–101. (Supervising Scientist, Australian Government Department of Sustainability, Environment, Water, Population and Communities: Darwin, NT, Australia) Available at https://www.dcceew.gov.au/sites/default/files/documents/ssr204.pdf
Buda M, Maki A, Mazurowski MA (2018) A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks 106, 249-259.
| Crossref | Google Scholar | PubMed |
Bush A, Compson ZG, Monk WA, Porter TM, Steeves R, Emilson E, Gagne N, Hajibabaei M, Roy M, Baird D (2019) Studying ecosystems with DNA metabarcoding: lessons from biomonitoring of aquatic macroinvertebrates. Frontiers in Ecology and Evolution 7, 434.
| Crossref | Google Scholar |
Campbell MD, Pollack AG, Gledhill CT, Switzer TS, DeVries DA (2015) Comparison of relative abundance indices calculated from two methods of generating video count data. Fisheries Research 170, 125-133.
| Crossref | Google Scholar |
Cappo M, Speare P, Wassenberg T, Harvey E, Rees M, Heyward A, Pitcher R (2001) The use of baited remote underwater video stations (BRUVS) to survey demersal fish stocks – how deep and meaningful. In ‘Video sensing of the size and abundance of target and non-target fauna in Australian fisheries: a national workshop. FRDC Proiect 2000/187’, 4–7 September 2000, Rottnest Island, WA, Australia. (Eds E Harvey, M Cappo) pp. 63–71. (Fisheries Research Development Corporation: Canberra, ACT, Australia) Available at https://www.frdc.com.au/sites/default/files/products/2000-187-DLD.pdf
Chinchor N (1992) Muc-4 evaluation metrics. In ‘Proceedings of the Fourth Message Understanding Conference’, 16–18 June 1992, McLean, VA, USA. pp. 22–29’. (Morgan Kaufmann) Available at https://aclanthology.org/M92-1002.pdf
Connolly RM, Fairclough DV, Jinks EL, Ditria EM, Jackson G, Lopez-Marcano S, Olds AD, Jinks KI (2021) Improved accuracy for automated counting of a fish in baited underwater videos for stock assessment. Frontiers in Marine Science 8, 658135.
| Crossref | Google Scholar |
Crook DA, Buckle DJ, Morrongiello JR, Allsop QA, Baldwin W, Saunders TM, Douglas MM (2020) Tracking the resource pulse: movement responses of fish to dynamic floodplain habitat in a tropical river. Journal of Animal Ecology 89(3), 795-807.
| Crossref | Google Scholar | PubMed |
Crook DA, Wedd D, Adair BJ, Mooney TJ, Harford AJ, Humphrey CL, Morrongiello JR, King AJ (2023) Evaluation and refinement of a fish movement model for a tropical australian stream subject to mine contaminant egress. Environmental Biology of Fishes 106, 469-490.
| Crossref | Google Scholar |
Crowley LELM, Ivantsoff W (1982) Reproduction and early stages of development in two species of Australian rainbowfishes, Melanotaenia nigrans (richardson) and Melanotaenia splendida inornata (Castelnau). Australian Zoologist 21(1), 85-95.
| Crossref | Google Scholar |
Dafforn KA, Johnston EL, Ferguson A, Humphrey CL, Monk W, Nichols SJ, Simpson SL, Tulbure MG, Baird DJ (2016) Big data opportunities and challenges for assessing multiple stressors across scales in aquatic ecosystems. Marine and Freshwater Research 67(4), 393-413.
| Crossref | Google Scholar |
Del Sole A (2018) Introducing Microsoft cognitive services. In ‘Microsoft Computer Vision APIs distilled: getting started with cognitive services’. pp. 1–4. (Apress: Berkeley, CA, USA) doi:10.1007/978-1-4842-3342-9_1
Ditria EM, Lopez-Marcano S, Sievers M, Jinks EL, Brown CJ, Connolly RM (2020) Automating the analysis of fish abundance using object detection: optimizing animal ecology with deep learning. Frontiers in Marine Science 7, 429.
| Crossref | Google Scholar |
Ditria EM, Connolly RM, Jinks EL, Lopez-Marcano S (2021) Annotated video footage for automated identification and counting of fish in unconstrained seagrass habitats. Frontiers in Marine Science 8, 629485.
| Crossref | Google Scholar |
Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2010) The pascal visual object classes (VOC) challenge. International Journal of Computer Vision 88(2), 303-338.
| Crossref | Google Scholar |
Follana-Berná G, Palmer M, Lekanda-Guarrotxena A, Grau A, Arechavala-Lopez P (2020) Fish density estimation using unbaited cameras: accounting for environmental-dependent detectability. Journal of Experimental Marine Biology and Ecology 527, 151376.
| Crossref | Google Scholar |
Garcia R, Prados R, Quintana J, Tempelaar A, Gracias N, Rosen S, Vågstøl H, Løvall K (2020) Automatic segmentation of fish using deep learning with application to fish size measurement. ICES Journal of Marine Science 77(4), 1354-1366.
| Crossref | Google Scholar |
Hannweg B, Marr SM, Bloy LE, Weyl OLF (2020) Using action cameras to estimate the abundance and habitat use of threatened fish in clear headwater streams. African Journal of Aquatic Science 45, 372-377.
| Crossref | Google Scholar |
Harvey ES, McLean DL, Goetze JS, Saunders BJ, Langlois TJ, Monk J, Barrett N, Wilson SK, Holmes TH, Ierodiaconou D, Jordan AR, Meekan MG, Malcolm HA, Heupel MR, Harasti D, Huveneers C, Knott NA, Fairclough DV, Currey-Randall LM, Travers MJ, Radford BT, Rees MJ, Speed CW, Wakefield CB, Cappo M, Newman SJ (2021) The BRUVS workshop – an Australia-wide synthesis of baited remote underwater video data to answer broad-scale ecological questions about fish, sharks and rays. Marine Policy 127, 104430.
| Crossref | Google Scholar |
Hebert PDN, Cywinska A, Ball SL, deWaard JR (2003) Biological identifications through DNA barcodes. Proceedings of the Royal Society of London – B. Biological Sciences 270(1512), 313-321.
| Crossref | Google Scholar |
Humphrey CL, Dostine PL (1994) Development of biological monitoring programs to detect mining-waste impacts upon aquatic ecosystems of the alligator rivers region, northern territory, Australia. Internationale Vereinigung für Theoretische und Angewandte Limnologie: Mitteilungen 24(1), 293-314.
| Google Scholar |
Humphrey CL, Bishop KA, Brown VM (1990) Use of biological monitoring in the assessment of effects of mining wastes on aquatic ecosystems of the alligator rivers region, tropical northern Australia. Environmental Monitoring and Assessment 14(2-3), 139-181.
| Crossref | Google Scholar | PubMed |
Humphrey CL, Bishop KA, Dostine PL (2016) Vulnerability of fish and macroinvertebrates to key threats in streams of the Kakadu region, northern Australia: assemblage dynamics, existing assessments and knowledge needs. Marine and Freshwater Research 69(7), 1092-1109.
| Crossref | Google Scholar |
Jalal A, Salman A, Mian A, Shortis M, Shafait F (2020) Fish detection and species classification in underwater environments using deep learning with temporal information. Ecological Informatics 57, 101088.
| Crossref | Google Scholar |
Junk WJ (1999) The flood pulse concept of large rivers: learning from the tropics. River Systems 11, 261-280.
| Crossref | Google Scholar |
Khan S, Rahmani H, Shah SAA, Bennamoun M (2018) A guide to convolutional neural networks for computer vision. Synthesis Lectures on Computer Vision 8(1), 1-207.
| Crossref | Google Scholar |
King AJ, George A, Buckle DJ, Novak PA, Fulton CJ (2018) Efficacy of remote underwater video cameras for monitoring tropical wetland fishes. Hydrobiologia 807(1), 145-164.
| Crossref | Google Scholar |
Knausgård KM, Wiklund A, Sørdalen TK, Halvorsen KT, Kleiven AR, Jiao L, Goodwin M (2022) Temperate fish detection and classification: a deep learning based approach. Applied Intelligence 52, 6988-7001.
| Crossref | Google Scholar |
Liberty E, Karnin Z, Xiang B, Rouesnel L, Coskun B, Nallapati R, Delgado J, Sadoughi A, Astashonok Y, Das P, et al. (2020) Elastic machine learning algorithms in Amazon SageMaker. In ‘SIGMOD’20: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data’, 14–19 June 2020, Portland, OR, USA. (Eds D Maier, R Pottinger, AH Doan, W-C Tan, A Alawini, HQ Ngo) pp. 731–737. (Association for Computing Machinery: New York, NY, USA) Available at https://ssc.io/pdf/modin711s.pdf
Mullen JF Jr, Tanner FR, Sallee PA (2019) Comparing the effects of annotation type on machine learning detection performance. In ‘2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)’, 16–17 June 2019, Long Beach, CA, USA. pp. 855–861. (IEEE) doi:10.1109/CVPRW.2019.00114
Nichols JD, Williams BK (2006) Monitoring for conservation. Trends in Ecology & Evolution 21(12), 668-673.
| Crossref | Google Scholar | PubMed |
Norouzzadeh MS, Nguyen A, Kosmala M, Swanson A, Palmer MS, Packer C, Clune J (2018) Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning. Proceedings of the National Academy of Sciences 115(25), E5716-E5725.
| Google Scholar |
Pudaruth S, Nazurally N, Appadoo C, Kishnah S, Vinayaganidhi M, Mohammoodally I, Ally YA, Chady F (2020) Superfish: a mobile application for fish species recognition using image processing techniques and deep learning. International Journal of Computing and Digital Systems 10, 1-14.
| Google Scholar |
Pusey BJ, Burrows DW, Kennard MJ, Perna CN, Unmack PJ, Allsop Q, Hammer MP (2017) Freshwater fishes of northern Australia. Zootaxa 4253(1), 1-104.
| Crossref | Google Scholar | PubMed |
Salman A, Siddiqui SA, Shafait F, Mian A, Shortis MR, Khurshid K, Ulges A, Schwanecke U (2020) Automatic fish detection in underwater videos by a deep neural network-based hybrid motion learning system. ICES Journal of Marine Science 77(4), 1295-1307.
| Crossref | Google Scholar |
Schobernd ZH, Bacheler NM, Conn PB (2014) Examining the utility of alternative video monitoring metrics for indexing reef fish abundance. Canadian Journal of Fisheries and Aquatic Sciences 71(3), 464-471.
| Crossref | Google Scholar |
Shahinfar S, Meek P, Falzon G (2020) “How many images do I need?” understanding how sample size per class affects deep learning model performance metrics for balanced designs in autonomous wildlife monitoring. Ecological Informatics 57, 101085.
| Crossref | Google Scholar |
Sheaves M, Bradley M, Herrera C, Mattone C, Lennard C, Sheaves J, Konovalov DA (2020) Optimizing video sampling for juvenile fish surveys: using deep learning and evaluation of assumptions to produce critical fisheries parameters. Fish and Fisheries 21(6), 1259-1276.
| Crossref | Google Scholar |
Shen C (2018) A transdisciplinary review of deep learning research and its relevance for water resources scientists. Water Resources Research 54(11), 8558-8593.
| Crossref | Google Scholar |
Supervising Scientist Division (2011a) Environmental monitoring protocols to assess potential impacts from ranger minesite on aquatic ecosystems: fish community structure in channel billabongs. Internal report 590. (Supervising Scientist: Darwin, NT, Australia) Available at https://www.dcceew.gov.au/science-research/supervising-scientist/publications/internal-reports/monitoring-protocols-fish-community-structure
Supervising Scientist Division (2011b) Environmental monitoring protocols to assess potential impacts from ranger minesite on aquatic ecosystems: fish community structure in shallow lowland billabongs. Internal report 589. (Supervising Scientist: Darwin, NT, Australia) Available at https://www.dcceew.gov.au/science-research/supervising-scientist/publications/internal-reports/monitoring-protocols-macroinvertebrate-community-structure-billabong
Supervising Scientist (2020) Annual technical report 2019–20. (Commonwealth of Australia) Available at https://www.dcceew.gov.au/science-research/supervising-scientist/publications/ss-annual-technical-report-2019-20
Supervising Scientist (2021) Annual technical report 2020–21. (Commonwealth of Australia) Available at https://www.dcceew.gov.au/science-research/supervising-scientist/publications/ss-annual-technical-report-2020-21
Villon S, Mouillot D, Chaumont M, Subsol G, Claverie T, Villéger S (2020) A new method to control error rates in automated species identification with deep learning algorithms. Scientific Reports 10(1), 10972.
| Crossref | Google Scholar |
Villon S, Iovan C, Mangeas M, Vigliola L (2022) Confronting deep-learning and biodiversity challenges for automatic video-monitoring of marine ecosystems. Sensors 22(2), 497.
| Crossref | Google Scholar | PubMed |
Vuttipittayamongkol P, Elyan E, Petrovski A (2021) On the class overlap problem in imbalanced data classification. Knowledge-Based Systems 212, 106631.
| Crossref | Google Scholar |
Wearn OR, Freeman R, Jacoby DMP (2019) Responsible AI for conservation. Nature Machine Intelligence 1(2), 72-73.
| Crossref | Google Scholar |
Whiteside TG, Bartolo RE (2015) Mapping aquatic vegetation in a tropical wetland using high spatial resolution multispectral satellite imagery. Remote Sensing 7(9), 11664-11694.
| Crossref | Google Scholar |
Whitmarsh SK, Fairweather PG, Huveneers C (2017) What is Big BRUVver up to? Methods and uses of baited underwater video. Reviews in Fish Biology and Fisheries 27(1), 53-73.
| Crossref | Google Scholar |
Xu W, Zhu Z, Ge F, Han Z, Li J (2020) Analysis of behavior trajectory based on deep learning in ammonia environment for fish. Sensors 20(16), 4425.
| Crossref | Google Scholar | PubMed |
Xu X, Li W, Duan Q (2021) Transfer learning and SE-ResNet152 networks-based for small-scale unbalanced fish species identification. Computers and Electronics in Agriculture 180, 105878.
| Crossref | Google Scholar |
Zhang B, Xie F, Han F (2019) Fish population status detection based on deep learning system. In ‘2019 IEEE International Conference on Mechatronics and Automation (ICMA)’, 4–7 August 2019, Tianjin, PR China. pp. 81–85. (IEEE) doi:10.1109/ICMA.2019.881626
