The Biodiversity Information Standards (TDWG) group and the Genomic Standards Consortium (GSC) have emerged as de facto (meta)data standards authorities in the biodiversity domain. The former’s scope spans biodiversity data at large, while the latter focuses on genomic, and then multi-omic, data and metadata such as lab protocols or chemical/physical measurements. Their activities, technologies, and management structures have been largely parallel, with some notable exceptions catalyzed through joint interest groups such as the Genomic Biodiversity Working Group (GBWG).

The overlap of TDWG and the GSC in multi-omic biodiversity data is an opportunity to begin sustainable convergence of the (meta)data standards these organizations maintain. Most notably among these, are the Darwin Core (DwC) and the Minimal Information about any (x) Sequence (MIxS) specifications. This memorandum builds on the output of a GBWG task group to propose a solution for sustained mapping and scalable interoperation of both DwC and MIxS. Its goal is to ensure that TDWG and the GSC create a lasting and continuous model to synchronize their standards, eventually promoting full bi-lateral integration.

Recognizing that both the Biodiversity Information Standards (TDWG) group and the Genomic Standards Consortium (GSC) have established well-adopted and community-driven (meta)data specifications for sequence-based biodiversity data;

Further recognizing that users of one standard specification should not have to invest additional effort in independently translating their (meta)data into another;

It is resolved that:

• The GSC and TDWG will maintain and endorse an authoritative and machine-readable mapping*16 of the fields in their MIxS and DwC (meta)data standard specifications;

• These authoritative mappings (in SSSOM-compliant tab-separated value files) and other digital references will be maintained in the GBWG GitHub repository within the TDWG organization and with TDWG-issued IRIs for the mapping files;

• Further, both organizations will provide bilaterally endorsed reference implementations of how to use their counterpart’s specification in their data structures (e.g., a DwC Archive incorporating fields mapped to MIxS in a DwC extension);

• Any necessary modification of identifiers (URNs, URLs, URIs, IRIs, etc) or other component of a standard issued by one organization which impacts the other should be declared and the particulars agreed upon in documented appendices to this MoU;

• When one specification is updated, the TDWG DwC Maintenance Group and the GSC Compliance and Interoperability Group (CIG) will hold joint sessions to update and validate any mappings and reference implementations to ensure clarity in the multi-omic biodiversity data community.

• The communication channels to communicate updates of either specification will be the MIxS issue tracker*22 and the DwC issue tracker*23. If any of the terms in the mapping are subject to review, the parties will notify each other through those communication channels.

Additionally recognizing that unilateral innovation and research actions will propose and implement alternative mappings and extensions to sequence-based metadata specifications.

It is further resolved that:

• Only those modifications which have been reviewed and endorsed by mechanisms bi-laterally convened by TDWG and the GSC will be considered standardized;

• Innovation is still welcome, and both organizations will welcome input and inspiration from application-driven modifications of the base standard.

Signatories:

12. October 2022 Representative of TDWG Executive (Deborah L Paul)

13. October 2022 Representative of the GSC Board (Lynn Schriml)

We created a DwC extension*24 including the MIxS core keys that do not have a counterpart in DwC, and thus were not included in the mapping (see above). Used in combination with the SSSOM record generated by our TG, the MIxS-DwC extension allows a complete encapsulation of MIxS core in a DwC Archive (modulo some semantic and syntactic mismatches, see Recommendations for Semantic and Syntactic Mapping)

Of the 96 keys contained in MIxS core, we included the 82 terms that were not mapped in the extension.

The TG’s GitHub repository hosts both the list of keys*25 and a list of excluded (mapped) keys*26. For the keys included in the extension, we have developed a Darwin Core Archive (DwC-A) extension definition in XML*27, which provides the standard set of terms that are available, onto which one can map one’s own CSV*28.

Following the terms of our MoU draft, this extension will be bilaterally endorsed by the GSC and TDWG to assure users that they are implementing an officially recognized recommendation. The manner in which this is declared (e.g., as a header in the DwC-A reference implementation) will be decided upon by the relevant bodies in the GSC and TDWG.

While the bilaterally endorsed GSC-TDWG extension provides stability, we recognize that the needs of the biodiversity community are more diverse and require more nimble forms of data exchange. In the creation of these more ad hoc extensions, the risk of creating siloed / bespoke data products (and thus reducing global interoperability) is often countered by the practicality of advancing with fewer overheads and at a more rapid pace than standards bodies can be expected to match. Here, without taking a position on the “better” route, we recognize the reality of this scenario.

To demonstrate how metadata fields relevant to sequence-based biodiversity data can relate to the core outputs of this TG, we include a variation of the MIxS-DwC extension - the DNA-derived data extension - developed by GBIF*29 as an example of the use (and customization) of the MIxS-DwC extension introduced above. Note again, that this DNA-derived data extension is not built on standards-body synchronization.

The DNA-derived data extension includes all keys of the MIxS-DwC extension, but brings in additional keys necessary to satisfy the exchange needs of the GBIF/OBIS/Atlas of Living Australia (ALA) networks. The additional keys originate primarily from the Minimum Information for Publication of Quantitative Real-Time PCR Experiments (MIQE) recommendation and Global Genome Biodiversity Network (GGBN).

Additionally, the DNA-derived data extension also takes measures to optimize the formatting and machine-readability of keys from MIxS. This stems from the fact that some MIxS key-value pairs are not atomic, i.e., they include multiple values in the same field (e.g., the MIxS key “pcr_primers” requires the user to enter a value that is comprised of a string that represents both the forward and reverse primer sequence, separated by a semicolon). This value-level formatting creates a bespoke data structure which then requires custom software or code to parse, limiting interoperability with external systems. Thus, in the case of pcr_primers, the DNA-derived data extension uses alternative keys, based on the MIxS key, which are associated with atomic values: pcr_primer_forward and pcr_primer_reverse. This allows for more efficient and unambiguous data ingestion into search indices, relational databases, or similar solutions, with minimal processing.

We acknowledge that it is a balance for application profiles to both comply with community standard specifications, while also satisfying the needs of the systems using them. To include and represent the evolving needs of the community and applications in existing community standards, we encourage that requests for changes or new keys are directed directly to the GSC*30 or TDWG*31.

The Simple Standard for Sharing Ontology Mappings (SSSOM) offers a framework to represent ontology mappings in a precise way, with a structured way to include rich provenance. For the work of this TG, we have implemented an SSSOM mapping between the DwC standard and the MIxS checklist.

SSSOM provides a minimal set of standard elements for the dissemination of mappings between terms. This helps to ensure a reliable interpretation of mappings and enables sharing and data integration between human and machine agents.

As described in the Recommendations for semantic and syntactic alignment, even closely related MIxS and DwC terms, may have semantic variance, and expect values with different syntax. To manage that variance, we propose extending the list of SSSOM metadata elements to include elements to capture the syntactic mapping (syntax_predicate_id, syntax_comment; see Approach: Mapping) in addition to the existing semantic mapping metadata elements.

During the process of mapping, it is very useful to include additional attributes/columns in the SSSOM matrix in which information, upon which the mapping is based, can be stored.

We thus propose adding such columns during the process (e.g., definitions [subject_definition, object_definition], syntax requirements [subject_valueSyntax, object_valueSyntax]; see Approach: Mapping). Once the process is over, a leaner SSSOM product can be released omitting these supporting attributes.

For mapping keys from metadata standards to one another, this TG recommends:

1. Follow the SSSOM guidance*32.

2. Until official guidance is offered from the SSSOM team, apply the extension proposed above (see Approach: Mapping) to additionally capture the mapping of syntax requirements:

   1. using the SSSOM predicate_id and corresponding comment to capture the semantics, and the syntax_predicate_id and corresponding syntax_comment to capture the syntactic mapping of terms.

3. Communicate any needed extensions to the SSSOM team via their issue tracker*33.

Due, in part, to the different approaches to atomization described above and below, many of the proposed relationships between MIxS and DwC keys required one-to-many or many-to-one mappings. This usually occurred when one specification offered multiple similar alternative keys for a phenomenon (e.g., DwC offers five keys relevant to “depth” measurements, while MIxS only offers one).

Recognizing that many keys in DwC or MIxS have community- and development-specific legacies, we recommend:

1. A mapping between metadata standards should be all-encompassing, and may thus include many-to-one, many-to-many, or one-to-many mappings.

2. Implementers, who represent a community of practice, can add notes on what keys they think are the most sensible.

3. In the long term, the standards agencies should aim to reduce the complexity of keys, moving towards atomization, to support more one-to-one relationships, eventually supporting full convergence.

DwC and MIxS specifications both offer guidance on the syntax expected for each value in a given key-value pair, alongside general notes on the expected semantics. In DwC, a value’s expected semantics*34 are captured in the Definition and Notes attributes of the List of Darwin Core Terms*35, while the Examples attribute shows expected value syntax. MIxS offers similar semantic guidance in the Definition attribute, with syntax and similar conventions specified in the Expected value, Value syntax, Preferred unit, and Examples attributes of the MIxS checklist*36. Since DwC and MIxS have been developed independently from one another, and complex/bespoke syntax is common to both specifications, there is considerable divergence in their conventions. These include:

• For measured values, MIxS expects the unit to be included as part of the value, while in DwC the unit is not allowed as part of the value - it is either inherent in the term definition or requires a separate term to specific the unit (optional for verbatim fields*37, which are expected to represent the measurement as originally recorded).

• For measured values, MIxS offers a “preferred” unit option, which - as the label implies - is not mandatory, while DwC either clearly defines the expected unit for a value or allows any unit to be used in a unit field related to the value field (except for verbatim fields).

• Some MIxS keys, such as lat_lon, expect values that capture two or more measured/derived values. DwC typically separates these measured/derived values across two or more keys (e.g., decimalLatitude and decimalLongitude).

• Also, several MIxS fields allow for a numeric value or a range, followed by a measurement unit (size_frac, samp_size, temp, depth, etc.). Darwin Core generally opts for atomic values associated with its keys.

Incompatibilities, such as those above, create (meta)data silos between communities using one or the other specification. Mappings built upon these can (in general) only be semantically and syntactically loose, and implementers must create and maintain converters or automated translators between the two, severely limiting and likely causing error propagation in machine-to-machine exchanges.

The SSSOM community is actively looking at ways to address these kinds of data structure mappings, and whether to address them as in scope for SSSOM, or to address these using the LinkML-transformer framework.

To secure improved semantic and syntactic alignment, this TG recommends the following:

1. The use of more explicit labels (terms), associated with less ambiguous definitions (many of which are more descriptive than definitional).

   1. Additionally, further cross-organization efforts to align the semantics of their fields in successive releases, using their obsolescence/change processes as appropriate.

2. Examples or descriptions of what is within and outside of the semantic scope/range of each field.

3. For any non-verbatim fields, clear guidance on what syntax is expected in each field (e.g., how many terms, separated how, with or without which unit?).

4. Re-use of existing and established terms from more general standards organizations within each specification (e.g., using dcterms:license to capture licensing information within MIxS and DwC).

5. Alignment to official external standards (e.g., using ISO 8601 to capture the time and date of an event)*38.

6. Synchronization between standards bodies ahead of new releases for closer syntactic alignment.

7. Semantic stability and standard syntax so stable converters can be written.

8. Atomic key-value structures, such that no complex or bespoke data structure exists in each value. For example, splitting ranges into dedicated start and stop fields.

   1. With advancement towards RDF- or JSON-based representations, allowing lists to be rendered as repeated key-value pairs.

9. Removing units from values by, for example, requiring a standard unit in the definition of each key.

In addition to MIxS core, MIxS contains numerous “environmental packages” which bundle keys which improve the contextualization of sequences in a given sampling environment. These are especially relevant for associating specific chemical and physical environmental measurements with specimens collected from these environments. Examples include marine, soil, food, and host-associated packages. These packages were created as a means to keep the core set relatively small, while rapidly accounting for the needs of sub-domains. These keys, and specifications of expected values, however, have not been harmonized or otherwise made interoperable with information standards published and used in Earth and environmental sciences.

Thus, this TG created SSSOM mappings and harmonization notes only for MIxS keys that directly pertained to sequences (MIxS core), rather than the specific environment they were obtained from.

Recognizing that the standardization domain/mandate of the GSC does not extend to standards of environmental parameters, this TG recommends that:

1. Any sustained reference implementation of a MIxS extension of DwC - endorsed by the GSC and TDWG - is limited to those MIxS keys which closely pertain to sequences (MIxS core), rather than the environments they originate from (MIxS environmental packages).

2. The GSC, as it begins to transition MIxS into RDF, should make efforts to map and eventually replace their environmental keys with equivalent, well-described keys from an information standards body working in the Earth and environment domain. We strongly advise that this is done as a joint activity with TDWG, to prevent decoupling and the need for downstream re-alignment.

3. Users wishing to use the MIxS environmental package keys in DwC Archives should use the MeasurementOrFact (MOF)*39 collection of keys (cast as an MoF class and associated properties, see Suppl. material 1 Section 2 for technical clarification) in the DwC specification. In our analysis, we found it valid to include a qualified mapping to a MIxS key URI as a value associated with the DwC “measurementRemark” key. This - alongside the other MOF key-value pairs - would allow any key in the MIxS environmental packages (either directly measured [measurement] or asserted to be true [fact]) to be represented in DwC. *40

   1. While we demonstrate how to link MIxS environmental package keys to DwC’s MoF, we draw attention to the fact that the GSC’s mandate is not within the standardization of Earth and environment metadata. Thus, where possible, users should attempt to use values from more Earth and environmental vocabularies, thesauri, ontologies, etc.

   2. Please see Suppl. material 1 Section 2 for an example of the above, and note the term measurementType.

4. TDWG and the GSC, in partnership with one or more standards bodies in the Earth and environmental sciences (e.g., the Earth Science Information Partners), convene a task group (or extend and expand this TG with a new mandate) to provide recommendations on how to sustainably and FAIRly incorporate well-adopted and more formally standardized environmental parameters into both MIxS and DwC.

To our knowledge, there is no sustained attempt to secure interoperability between the competing standards (most of which are informal, ad hoc, or de facto, as are MIxS and DwC) in this space. Some organizations and efforts of interest are listed below.

• Parameter vocabularies

  • The British Oceanographic Data Centre (BODC)*41 Natural Environment Research Council (NERC) Vocabularies*42, e.g., BODC Parameter Usage Vocabulary*43

• The Open Geospatial Consortium (OGC)*44

• Climate and Forecasting Variables*45

We note that, while this vacuum exists, implementers will create their own internal standards for expediency*46. This does provide some basis for later alignment but also creates overhead as more unaligned information standards are released, compete for users, and decouple information systems and communities. We, therefore, re-emphasize the need for both TDWG and the GSC to engage with information standards communities in the Earth and environment domain to integrate their specifications.

Information on licensing is critical for data reusability (as declared in the FAIR Principles Wilkinson et al. 2016). Such information is captured in DwC through the import of the Dublin Core key dcterms:license*47; however, there is no equivalent key provided in the MIxS specification.

Recognizing that the GSC does not currently intend to extend their core checklist to include a key for licensing information*48, this TG recommends that implementers extend MIxS records with the Dublin Core key http://purl.org/dc/terms/license to capture data reuse restrictions.

While saying this, we also recognize the need for further discussions around the subject of license and reuse in conjunction with access and benefit sharing discussions around the Nagoya protocol and digital sequence information, as well as in conjunction with the implementation of the CARE principles Carroll et al. 2020 (e.g., through Traditional Knowledge and Biocultural Labels and Notices). Thus, additional and more nuanced fields expanding on reuse restrictions should be considered as they are being developed.