Creation of a text dataset of taxonomic descriptions

Overview of the Dataset and Workflow

As shown in Fig. 1, the document processing pipeline consists of two main workflows: one for PDF documents (top and bottom) and one for web data (right side).

In the PDF workflow (agents are described in Turenne et al. (2026)), each PDF is processed page by page using OCR (Optical Character Recognition) to extract the text content. The individual page texts are then concatenated into a single OCR file per PDF. The resulting OCR text is segmented into family-, genus- and species-level files. botanical descriptions are hierarchical, species descriptions are often incomplete in isolation, with characters shared by all species within a genus described at the genus level and characters common to all genera within a family described at the family level.

In the web scraping workflow, we first detect web pages associated with individual species and extract the textual content. If the content is not in English, we retrieve the corresponding Wikipedia entry in the original language for the species. The result is a unified text file per species, combining both web and Wikipedia data (if it exists).

Finally, in the integration step, all processed text files — whether derived from PDFs or web sources — are consolidated into a single structured repository, organised by species.

Data Acquisition

OCR-based extraction

Over 550 monographs spanning 36 regions were analysed (Table 1). Most PDFs, published between 1845 and 2000, scanned images from botanical volumes requiring OCR. We used Tesseract OCR v.5.5.0, an open-source engine combining classical algorithms and long short-term memory neural networks, capable of handling historical and low-quality scans (Smith 2007, Tesseract-OCR-Project 2026). The model incorporates image features, optional dictionaries and frequency tables. OCR converted scanned pages into machine-readable text, enabling downstream natural language processing tasks, such as tokenisation, text extraction and analysis of botanical content.

A detailed list of sources and references is provided in the Suppl. material 1. A complete list of the PDF sources used for text analysis is provided in the Supplementary Material.

Web scraping

Certain websites provide high-quality information about plant species and Wikipedia is amongst the most comprehensive. As a globally recognised knowledge repository, Wikipedia allowed us to retrieve 122,000 pages, each corresponding to a distinct plant species (i.e. one page per species). This constitutes a substantial body of natural language text that can be exploited for large-scale botanical data mining. Unlike scanned PDF documents, Wikipedia pages do not require OCR processing, significantly simplifying the extraction pipeline.

Table 2 shows the copyright requirements for web sites. While Wikipedia provides a valuable, broad overview of plant species from all over the world, we deliberately differentiate our data sources by prioritiing web scraping from specialised websites that offer deeper expertise and localised precision in specific botanical areas.

Species-related information was collected only for pages with sufficient textual content. Since each website has its own HTML structure and CSS styling, a dedicated scraper was developed for each source. Due to the ethical and respectful use of web resources (to avoid overload of infrastructure), data collection was rate-limited, making the overall process time-consuming. Scraping time for all websites is 1,370 hours in total. A token is supposed to be a string with alphabetical letters (not a number or a punctuation). We tokenise using regular expressions. Tokens were created by splitting the text using a predefined list of character delimiters.

Wikipedia

Data obtained from Wikipedia enables investigation about the popularity of plant genera by analying species-level data extracted from Wikipedia, assessing how frequently and prominently different genera are represented across the corpus (Fig. 2). It reflects contributor engagement and a potential audience, which may be further characterised using Google Trends data. We can also extract a set of genera that are over-represented in Wikipedia (Table 3). Table 3 shows the number of genera found in Wikipedia: 14,822, representing 31.9% of the total 46,427 genera.

Species Name Recognition and Normalisation

Dictionaries

To structure the dataset, each text segment was reduced to a file-species unit, with files named for specific plant species. Accurate extraction of scientific names is crucial, linking botanical knowledge sources (Lakshmi et al. 2012). Unlike Nainia et al. 2024, who used fine-tuned language models, we applied a computational linguistics approach using lexical transformations and curated dictionaries. Our algorithm, WordGen, merges three resources — World Checklist of Vascular Plants (WCVP-KEW 2024), World Flora Online (Miller and Ulate 2017 and TRY (Kattge et al. 2020) — providing standardised names and metadata on accepted taxa and synonyms. We adopted a reference dictionary approach, using the World Flora Online (WFO) as the primary taxonomic backbone. Entries from the World Checklist of Vascular Plants (WCVP) were compared against this reference and all names were retained in the merged dictionary together with their associated status (accepted name or synonym). This strategy allows us to preserve the richness of both sources, while maintaining traceability of taxonomic assignments. We note that discrepancies may occur between databases, including cases where a name is treated as accepted in one source and as a synonym in another. In such situations, WFO was systematically used as the primary reference, complemented by TRY and WCVP, although the biodiversity community has increasingly transitioned to utiliing the CoL/GBIF ChecklistBank, which aggregates various other resources (Berendsohn et al. 2026). We also observed that some name combinations may occur multiple times across sources and that disambiguation may require additional contextual information, such as family-level assignment. However, resolving these ambiguities was beyond the scope of the present data paper and is left for future work.

A genus groups multiple species and the standard format for scientific names follows the binomial nomenclature: Genus epithet (e.g. Quercus pubescens). When identifying subspecies (e.g. Quercus pubescens subsp. anatolica), they are treated as synonyms of the parent species to maintain consistency, effectively making subsp. anatolica synonymous with Quercus pubescens. In many botanical resources, these infraspecific names act as de facto synonyms of the accepted parent species. This means the specific distinguishing traits originally used to describe the subspecies are now incorporated into the broader concept of the accepted parent taxon. Quercus pubescens serves as the accepted scientific name for the downy or pubescent oak, with Quercus humilis recognied as a taxonomic synonym for this species.

Our dictionary contains over 1.6 million entries — including species, subspecies and official synonyms — serving as a dedicated reference for verifying taxa names compiled exclusively from trusted resources (TRY, WCVP, WFO), excluding all websites and PDFs. After normalising subspecies and synonyms to their corresponding accepted species names, we retain approximately 1.2 million unique species-level names (see Table 4). The number of recorded names is much larger than the accepted nomenclature due to the vast diversity of historical naming conventions found in historical documents, necessitating the retention of these records.

The total number of accepted plant species names remains debated across taxonomic databases, with 388,342 listed in World Flora Online (WFO) and 407,961 in the World Checklist of Vascular Plants (WCVP) (Schellenberger Costa et al. 2023).

Fig. 3 illustrates the distribution of botanical descriptions across textual datasets. Identifying the beginning of each species section is critical for extracting meaningful, species-specific information. However, this task is challenging due to textual inconsistencies and OCR-induced errors in digitied documents. Therefore, a robust species name recognition tool is essential.

Construction of the Plant Description Text

To construct the plant description text collection, a diverse set of printed and web-based resources was compiled and processed within the defined timeframe. The types and numbers of data sources included in the dataset are summarisd in Table 5.

The resulting text collection comprises multiple complementary dataset representations, including file-species and unique file-accepted species formats, providing both species-level and taxonomically consolidated records. A summary of the dataset composition is presented in Table 6.

Sections related to species in a taxonomic volume (e.g. shown here for Pteris tripartita and Pteris novae-caledoniae) correspond to distinct blocks of descriptive content.

WordGen tool

Accurate recognition of species names is crucial for both the OCR and indexing steps. To address this, we developed WordGen (for more details, seeTurenne et al. (2026)), a name-matching algorithm, based on transformation rules including insertion, deletion and substitution. The algorithm is designed to tolerate minor typographical or OCR errors, especially in species names. WordGen takes either a genus or a genus + epithet as input and performs approximate string matching, based on a curated botanical dictionary.

As our reference sources are dictionaries, such as World Flora Online (WFO), it is important to note that mapping authorship between the names recorded in PDFs and those listed in WFO is not straightforward and can be a significant source of error. Two illustrative examples are Achyranthes talbotii and Berberis empetrifolia:

Data from WFO:

wfo-0000516490 | urn:lsid:ipni.org:names:58932-1 | 750143 | "Achyranthes talbotii" | species | wfo-4000000251 | "Hutch. & Dalziel"

Data in a PDF document (Flora of Cameroon, p. 33):

2. Achyranthes talbotii Hutchinson & Dalziel

Mapping “Hutch. & Dalziel” to “Hutchinson & Dalziel” or “Lam.” to “Lam.-Gay” is not trivial. While the use of full binomial with author citation denominations (including author citation through normalisation using IPNI) would improve taxonomic precision, it would also significantly reduce recall, as such complete information is not consistently available across the corpus. Therefore, our approach prioritises species name matching while acknowledging this limitation.

This highlights the need not only to perform automated extraction of names, but also to evaluate the quality of these extractions. Failure to correctly map authorship may result in the omission of numerous instances.

Our corpus is geographic area-orientated, which greatly facilitates the disambiguation of species names.

As Fig. 4 shows, the algorithm performs fuzzy matching of species names by comparing a given input ("leptostylis gzrandidolia") against a known dictionary of epithets. This figure shows only one rule amongst several tested and applied on the input. It extracts valid species epithets associated with the genus Leptostylis and generates candidate corrections using insertion rules (e.g. inserting one letter at each position). The algorithm then validates candidates by checking if any match the input through a defined pattern, ultimately returning the partially corrected name "Leptostylis grandifolia".

1. Genus Input

If only a genus is provided, WordGen assumes that the entire input string corresponds to a genus name. For example, assuming the correct genus is Leptostylis:

Insertion Error: WordGen("leptostylisx") → leptostylis

2. Genus + Epithet Input

In cases where both genus and epithet are provided (e.g. leptostylis grandifolia), WordGen assumes the genus is correctly spelled and performs fuzzy matching on the epithet only:

Insertion Error: WordGen("leptostylis grandifolias") → grandifolia

(Tolerance: fixed 5-character prefix and up to 7-character mismatch in suffix).

By anchoring on genus names and correcting epithet variations, WordGen plays a key role in parsing large-scale, semi-structured botanical documents.

To evaluate the quality of our technique, based on WordGen and dictionary lookup, we built an evaluation dataset using indexed species names from the “Flora of New Caledonia series”. This collection has 25 volumes. We extracted all cited species names, including those in section headers. Evaluation is explained in section "Quality control: Technical Validation".

Normalisation

Normalisation occurs at two levels.

First, a taxonomic normalisation links scientific names found in the text to a standard reference, such as a taxonomic dictionary. A name may refer either to an accepted taxon or to a synonym of that taxon. For example, Leptostylis grandifolia is a synonym of the accepted name Pycandra grandifolia. Thus, two documents referring to these two names actually describe the same biological entity, even though they use different terms.

Second, a string-level normalisation addresses character-level variations and typographical inconsistencies found in original texts. These errors do not involve taxonomic synonymy, but they prevent correct linking to dictionary entries. For instance, in the New Caledonia subdataset used for evaluation, the taxon Hymenophyllum leratii appears as Hymenophyllum le ratii>, where an unintended space splits the epithet. Although visually and semantically identical, the dictionary lookup would fail without correction. Another example is a pseudo-Latin declension error, where Xylosma caledonica is written by the author as Xylosma caledonicum. This variant clearly refers to the same species.

Such character-level discrepancies were manually corrected during evaluation (in the New Caledonia subdataset only) to ensure accurate normalisation. These modifications involved very small changes and were applied conservatively to preserve the integrity of the evaluation. Without such normalisation, overly strict string matching could penalise high-quality recognitions due to insignificant orthographic variations.

Quality control: Technical Validation

Dictionary quality assessment

Tables 7, 8, 9 present the evaluation results, based on the New Caledonia collection, which comprises 25 volumes. In this dataset, 1,236 genera and 4,818 species names were cited (Table 7 ). When these are normalised to accepted names, we identify 2,749 unique scientific names, of which 99% are present in our dictionaries (Table 7). Focusing specifically on scientific names extracted from passages describing taxonomic descriptions, we find 1,718 accepted names, with 99.82% included in the dictionaries (Table 9) (see Table 8 with synonyms).

Automatic indexing assessment

To evaluate the indexing performance, we compared the species names identified as cited passages in the taxonomic descriptions against the groundtruth from the New Caledonia dataset, which contains 1,718 species names. Recall was used as the evaluation metric. Our method achieved a recall of 98.89% for species names and 99.52% for genera.

Nineteen taxa (scientific names) were not identified, mainly due to name variations caused by transcription errors or transformations. Examples include:

"11. Parsonsia catalprcarpa Baillon", which should be "Parsonsia catalpaecarpa";

The breakdown of missed detections (out of 19 cases) is as follows:

14 cases (74%) due to OCR errors;

Table 10 presents the results of the training and test evaluation conducted on the New Caledonia dataset.

Table 11 presents the application of the algorithm for taxa and passage detection on unseen data, specifically the Cameroon dataset. This dataset includes an annotated sample of 132 genus sections and 457 taxa sections (groundtruth). The recall achieved is 93.22%, representing a realistic and reliable measure for automatic taxon section identification.

Quality of data assessment

We aim to assign a confidence label to each species file using linguistic and document features to assess the quality of the extracted information. Let T be a text segment extracted from an OCR and information extraction pipeline. Let W = {w₁, w₂, …, w_n} be the set of alphabetic tokens extracted from T, where n = |W|. We define three linguistic features:

1. Lexical length ratio: R_SL(T) = ||w| ≤ 3| / (1 + ||w| ≥ 5|);

2. Adverb frequency: Let A(T) be the number of adverbs detected in T using a part-of-speech heuristic;

3. Punctuation density: The semi-colons ratio is defined as: R_C(T) = C(T) / n.

Each feature contributes to a partial score defined as in Table 12.

Then the global confidence score φ(T) is defined as :

φ(T) = f1(R_SL(T)) + f2(A(T)) + f3(R_C(T)) with φ(T) in the interval [0,3].

We assign the confidence label as stipulated in Table 13.

This function φ(T) defines a rule-based classifier that maps a text segment T to a confidence label {Low, Medium, High}. Applying this method to our dataset produced 19216 files with Low value (5.6% of the data text collection) (Suppl. material 2).

Files are organised into regional directories, each containing subdirectories for raw data (such as PDFs or scraped files), OCR-processed content (typically from monographs) and species-level files. Species files are named according to their corresponding scientific taxon names. Table 5 provides a summary of the entire dataset, while Table 6 presents the number of files it contains. Fig. 5 illustrates the geographical coverage of the dataset, excluding data from Wikipedia.

1847-1-01 - 2025-7-01

Creative Commons Public Domain Waiver (CC-Zero)

Creative Commons Public Domain Waiver (CC-Zero)

large text collection about world-wide Flora taxonomic and morphological descriptions

https://zenodo.org/records/20027402

1