Biodiversity Data Journal :
Software Description
|
Corresponding author: Victor Anton (victor@wildlife.ai), Matthias Obst (matthias.obst@marine.gu.se)
Academic editor: Danwei Huang
Received: 09 Nov 2020 | Accepted: 11 Feb 2021 | Published: 24 Feb 2021
© 2021 Victor Anton, Jannes Germishuys, Per Bergström, Mats Lindegarth, Matthias Obst
This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Citation:
Anton V, Germishuys J, Bergström P, Lindegarth M, Obst M (2021) An open-source, citizen science and machine learning approach to analyse subsea movies. Biodiversity Data Journal 9: e60548. https://doi.org/10.3897/BDJ.9.e60548
|
|
The increasing access to autonomously-operated technologies offer vast opportunities to sample large volumes of biological data. However, these technologies also impose novel demands on ecologists who need to apply tools for data management and processing that are efficient, publicly available and easy to use. Such tools are starting to be developed for a wider community and here we present an approach to combine essential analytical functions for analysing large volumes of image data in marine ecological research.
This paper describes the Koster Seafloor Observatory, an open-source approach to analysing large amounts of subsea movie data for marine ecological research. The approach incorporates three distinct modules to: manage and archive the subsea movies, involve citizen scientists to accurately classify the footage and, finally, train and test machine learning algorithms for detection of biological objects. This modular approach is based on open-source code and allows researchers to customise and further develop the presented functionalities to various types of data and questions related to analysis of marine imagery. We tested our approach for monitoring cold water corals in a Marine Protected Area in Sweden using videos from remotely-operated vehicles (ROVs). Our study resulted in a machine learning model with an adequate performance, which was entirely trained with classifications provided by citizen scientists. We illustrate the application of machine learning models for automated inventories and monitoring of cold water corals. Our approach shows how citizen science can be used to effectively extract occurrence and abundance data for key ecological species and habitats from underwater footage. We conclude that the combination of open-source tools, citizen science systems, machine learning and high performance computational resources are key to successfully analyse large amounts of underwater imagery in the future.
marine biodiversity, autonomous underwater vehicles, remotely-operated vehicles, artificial intelligence, big data, image analysis, participatory science, Essential Biodiversity Variables, research infrastructure, biodiversity monitoring
Biological observation techniques in the marine environment need to improve radically to serve our understanding of marine ecosystems under the influence of multiple stressors including long-term global change (
In-situ monitoring systems need to be coupled to data services that allow for swift exploration, processing and long-term storage (
Here, we present the Koster Seafloor Observatory, an open-source modular approach for managing, processing, and analysing large amounts of subsea movie data for marine ecological research. The Koster Seafloor Observatory allows scientists to upload underwater footage to a customised citizen science website and then train machine learning algorithms with those classifications provided by citizen scientists. These algorithms can be accessed through an Application Programming Interface (API) allowing researchers to test the performance of the algorithms under different confidence and overlapping thresholds, share their models with a wider audience and extract species observations from new footage.
Mapping cold water corals in Sweden's first marine national park
We piloted the Koster Seafloor Observatory to extract data on spatiotemporal distribution and relative abundance of habitat-building species from deep-water recordings in a Marine Protected Area, the Kosterhavets National Park in Sweden. The Park, established in 2009, contains a highly diverse and unique marine ecosystem. The seafloor in the deeper waters of the Park has oceanic connections and hence contains much of the bottom-dwelling fauna, which is otherwise only found in deep oceanic waters (
The Koster Seafloor Observatory is divided into three main modules: data management, citizen science and machine learning with high performance computing (Fig.
Module 1: Data management (
In the data management module, researchers store and process the data in a way that maximises efficiency, convenience and opportunities for sharing and collaboration. To store and access the raw data, we use long-term and short-term storage servers. The long-term storage server, or cold storage, archives large amounts of files that need not be accessed frequently. In our case, these include recordings from Remotely-Operated Vehicles (ROVs) managed by the University of Gothenburg, Sweden. The movies (mp4 and mov formats) are on average 1-2 hours long and have been systematically collected from all expeditions since the late 1990s (Fig.
The short-term storage server, or hot storage, stores a small proportion of files that are frequently used for analysis. Here, we transferred 60 movies from the cold storage to a project-specific short-term storage server (Suppl. material
We created a SQLite database to link all information related to the movies and the classifications provided by both citizen scientists and machine learning algorithms (Fig.
Module 2: Citizen science (
In the citizen science module, researchers and citizen scientists work together to efficiently and accurately annotate raw data. To identify the species recorded in our footage, we created a citizen science website. The site is hosted in Zooniverse, the largest citizen science platform in the world. The website contains rich supporting material (e.g. background, tutorials, field guides) and features two workflows that help citizen scientists to classify biological objects in video (workflow 1) and locate these objects in still images (workflow 2).
Workflow 1 (species identification):
Citizen scientists are presented with 10-second clips of underwater footage and need to select at least one of the 27 available choices (Fig.
We compared the classifications provided by an expert to those provided by citizen scientists to estimate the accuracy of citizen scientists to identify cold water corals (Table
Confusion matrices derived from applying different citizen scientists agreement thresholds (Cit.Sci. Agr.) when comparing expert classifications to citizen scientist classifications of 2,594 underwater videos. Each video was classified by an expert and eight different citizen scientists. Classifications of cold water coral were retained and all other classifications were grouped as "Other". Expert classifications were compared to citizen scientist classifications with at least 80%, 60% and 40% of agreement amongst their responses (i.e. an agreement threshold of 80% corresponds to an agreement on the classifications of at least seven of the eight citizen scientists who annotated the clip).
Cit. Sci. Agr. ≥ 80% | Cit. Sci. Agr. ≥ 60% | Cit. Sci. Agr. ≥ 40% | |||||
Coral | Other | Coral | Other | Coral | Other | ||
Expert | Coral | 111 | 467 | 315 | 263 | 475 | 103 |
Other | 2 | 2014 | 22 | 1994 | 84 | 1932 |
Workflow 2 (object location):
Citizen scientists are presented with a still image of the species of interest. To annotate the image, citizen scientists need to draw rectangles around the individuals of the species (Fig.
We used a four-stage video processing framework to upload clips and still images to the Koster Seafloor Observatory and download the annotations provided by citizen scientists (Fig.
Module 3: Machine learning and High Performance Computing (
In the machine learning and High Performance Computing module, researchers train, test and expose state-of-the-art machine learning models. The aggregated citizen scientist annotations are used to train object-detection models that track and identify the species of interest. In our case study, we used 409 user-annotated ground-truth frames obtained from workflow 2 (Suppl. material
We made the trained model available through an application programming interface (API), where it can be used by researchers to run predictions of the species of interest in new recordings (Fig.
We compared manual observations of cold water corals provided by an expert to those provided by our machine learning model to estimate the accuracy of the model under different confident thresholds (Table
Confusion matrices derived from applying different confidence thresholds (ML confidence) when overlaying manual with machine-based observations in movies corresponding to 132 squares of a spatial grid within the Kosterhavets National Park, Sweden. Detailed metadata for these recordings are provided in Suppl. material
ML confidence = 0.5 | ML confidence = 0.7 | ML confidence = 0.9 | |||||
Coral | No coral | Coral | No coral | Coral | No coral | ||
Expert | Coral | 54 | 15 | 52 | 17 | 28 | 41 |
No coral | 13 | 50 | 5 | 58 | 1 | 62 |
The last component of this module is a data visualisation toolkit that enables researchers to explore and visualise the ecological data extracted from the outputs of the machine learning model. In our case, we mapped the cold water coral annotations provided by the expert and the machine learning model with a 0.7 confidence threshold (Fig.
Comparison of manual and machine learning model-based spatial distribution of cold water coral in the reef area Säcken in Kosterhavets National Park, Sweden. Spatial distribution is based on coral observations in ROV movies corresponding to 132 squares of the spatial grid. Confidence threshold (Conf) for the model is set to 0.7. Grid size 5 m.
Discussion
The functionalities of the Koster Seafloor Observatory have been tested in the present case study, which illustrates the scientific potential of this open-source and modular approach. Our approach can be used to extract ecological data on abundance and distribution for many benthic species from underwater recordings. Underwater footage is today routinely collected by many research institutes, which may allow for a concerted analysis of such data over broad spatial and temporal scales in the future. Such analyses may calculate data products for biological state variables on regional or even global level, so-called Essential Biodiversity Variables or EBVs (
In order to scale up analysis of underwater imagery in the future to extract ecological data for larger regions, longer time periods and more species, several technical bottlenecks have to be addressed. Data archiving functions can fall under organisational or governmental responsibilities and may not be fulfilled by a single global system. Consequently, most underwater recordings are currently locally archived and cannot be discovered. Here, further work is needed to promote the use of open interoperable archives and data portals (e.g. European Marine Data Archive, EMODnet portal) that enable researchers to adequately publish metadata associated with underwater recordings. Another important technical bottleneck is the disconnection between many essential data services that need to interact to successfully analyse image data. We suggest that seamless links should be developed especially between citizen science platforms (for training of machine learning models) and high-performance computation services (for extracting ecological data from large amounts of imagery). Regional, national and global research infrastructures should take a leading role in this development to overcome current technical challenges.
The project was funded by Ocean Data Factory, an expert network supported by grants from Sweden’s Innovation Agency (grant agreement no. 2019-02256), the Swedish Agency for Marine and Water Management (grant agreement no. 956-19) and the Swedish Research Council (through Swedish LifeWatch grant agreement no. 829-2009-6278). The presented work was furthermore supported by the NeIC programme DeepDive and the Horizon 2020 project ENVRIplus (grant agreement no. 654182).
Our approach is open for use in research, as well as public and academic education for analysis of community composition in marine ecosystems.
We thank the data providers who allowed us to use movie material. These include ROV-pilots (especially Tomas Lundälv, Lisbeth Jonsson and Roger A. Johansson), as well as the data curator at the University of Gothenburg (Lars Ove Loo) and Chalmers Technical University (Ola Benderius). We acknowledge the tremendous help from taxonomic experts Thomas Dahlgren, Kennet Lundin and Björn Källström who actively curated the citizen science platform, as well as the ROV pilots who offered their material for use (Tomas Lundälv, Lisbeth Jonsson and Roger A. Johansson), while Emil Burman helped with the translation of the site. We also thank the Zooniverse team and the 2,451 citizen scientists who helped us classify the footage. We also thank the two reviewers of this manuscript for their comments and suggestions. Finally we are grateful for support by the Center for Sea and Society and the Gothenburg Global Biodiversity Center.
VA, MO, and JG conceived and designed the study. VA and MO set up and continue to maintain the Zooniverse site. VA and JG wrote the code for both Github projects (data processing workflow and model). MO worked with the public contributions to the Zooniverse site. PB and ML contributed with the data management and archiving of the original movies, the manual annotations of movies, as well as the analysis of the model results. VA, MO and JG contributed equally to the writing and revision of the manuscript.
Instances of Desmophyllum pertusum used to train Koster YOLO machine learning model.
This file contains metadata from the movies used to test the model and illustrate its application. To access the movie data files, contact the authors or search the filenames in the Swedish National Data Service: https://snd.gu.se/en/catalogue/study/snd1069.
Model output from analysis of the selected movies in Supplementary material 2. Explanation of variables: FilenameInThisStudy (movieID), frame_no_start (frame number when the object was detected for the first time), frame_no_end (frame number when the object was detected for the last time), max_conf (highest confidence value achieved by the object throughout the consecutive frames), x (x-position of the upper-left corner of the bounding box with the highest confidence value), y (y-position of the upper-left corner of the bounding box with the highest confidence value), w (width of the bounding box with the highest confidence value), h (height of the bounding box with the highest confidence value).