TG2-TO DO List for standard submission

[Tasilee]

A list of things for @ArthurChapman, @Tasilee, @tucotuco, @chicoreus and @ymgan to complete the BDQ Core standard submission.

Introduction Document and Landing Pages

• (Done) Replicate target directory structure for the standard within _review @tucotuco
• (Done) Files under _review directory that require some manual content @chicoreus (content parsed out of supplemental into users guides and other documents).
• (Done) Diagrams showing a generic approach to using Validations, Measures, Amendments, Validations, Measures @ymgan
• (Done) Place diagram svg files within _review directory. in _review/docs/intro @chicoreus
• (Done) Modfications to target directory structure to include derivative files for review @chicoreus See: _review/README for current state of files and directories
• (Done) Identify files that are currently maintained outside of _review and copied into it. @chicoreus see: https://github.com/tdwg/bdq/blob/master/tg2/_make_review/copy_files.sh
• (Done) Pare down Introduction document to minimum normative and non-normative needed @tucotuco (done through section 1)
• (Done) Add landing pad pages for bdqffdq (required), and bdqcore (desirable). @chicoreus
  • (Done) Create bdqcore landing page. @chicoreus
    • (Done) Refine which text goes on bdqcore landing page and which on bdqcore term-list @Tasilee @chicoreus
  • (Done) Create bdqffdq landing page. @chicoreus
  • (Done) Add normative guidance about use of bdqffdq terms to bdqffdq landing page. @chicoreus
• (Done) Add list of documents with target audience for each to end of introduction @chicoreus
• (Done) Add in the Acknowledgements and Contributions @ArthurChapman
• (Done) Update Glossary @ArthurChapman
• (Done) Check and edit Section 6 Glossary as new section added @Tasilee
• (Done) Add Oracle mark text for Java to glossary @ArthurChapman

Supplementary Document

• (Done) Edit Supplementary Document @Tasilee
• (Done) Put supplementary information into supplementary document @Tasilee
• (Done) Move normative information from supplementary document to appropriate other documents, limit content of supplementary to just supplementary. @chicoreus
• (Done) Format supplementary document @ArthurChapman
• (Done) Copy the bdqTag vocabulary into a markdown table in the supplementary document and remove from vocabulary/glossary_terms.csv @chicoreus
• (Done) Copy the bdqTestField vocabulary into a markdown table in the supplementary document and remove from vocabulary/glossary_terms.csv. @chicoreus
• (Done) Add history of the development of the tests to Section 3.
• (Done) Merge the three duplicative sets of Test Field descriptions in the supplementary documentation in to one.
• (Done) Add Competency Questions section for bdqffdq and bdqcore to the supplementary documentation. @chicoreus
• (Done) Write additional sparql queries on bdqcore RDF and use these as competency questions in supplement @chicoreus
• (Done) Review Supplement Document @ArthurChapman
• (Done) Review document @Tasilee
• (Done) Add discussion of bdq_issue_to_csv, bdqtestrunner, and kurator-ffdq to supplement. @chicoreus

Vocabularies and termlist documents

• (Done) Flesh out Vocabulary landing page @ArthurChapman, @tucotuco
• (Done) Review which terms go where @ArthurChapman @chicoreus
• (Done) Review bdq and bdqdim term definitions @ArthurChapman
• (Done) Update bdqffdq: Ontology @chicoreus @ArthurChapman
  • (Done) Review bdq/bdqffdq vocabulary definitions @ArthurChapman @chicoreus
  • (Done) Move ontology into tdw/bdq respository @chicoreus
  • (Done) Update bdqffdq ontology skos:definitions and skos:comments @chicoreus
  • (Done) Give each bdqffdq term a skos:prefLabel, skos:definition, rdfs:label, and rdfs:comment @chicoreus
  • (Done) Create csv export of ontology @chicoreus (note, needs manual edit with each export from owl).
  • (Done) Integrate First draft Python code to produce human readable display of ontology into _review/build/ , done as draft_build_bdqffdq.py @chicoreus
  • (Done) Add Argument and related terms for Parameters and Arguments on Specifications to bdqffdq @chicoreus
• (Done) Provide rdf distributions of Vocabulary Term List documents following SDS section 2.2.4. (owl document for bdqffdq)
  • (Done) rdf artifacts for Vocabulary Terms lists of bdq, bdqdim, bdqenh, bdqcrit @chicoreus
  • (Done) Update rdf artifact for bdqcore generated with kurator-ffdq @chicoreus
    • (Done) Multiple tests in each Policy @chicoreus,
    • (Done) Class names matched to bdqffdq. @chicoreus
    • (Done) Add Criterion, Enhancement, DataQualityDimension instances from bdqcrit, bdqenh, bdqdim vocabularies @chicoreus
    • (Done) Stable identifiers for methods, specifications, information elements in RDF generated from CSV @chicoreus (tg2/core/information_element_guids.csv and tg2/core/TG2_tests_additional_guids.csv used by kurator-ffdq with -guidFile and -ieGuidFile options)
      • (Done) Provide for reuse of uuids between builds of RDF artifact from configuration files @chicoreus
      • (Done) stable identifier file for core test methods/specifications/profiles @chicoreus
      • (Done) stable identifier file for information elements @chicoreus
      • (Done) add stable identifiers for methods/specifications/profiles for multirecord measures @chicoreus
      • (Done) Add Arguments and Parameters to bdqcore RDF serialization @chicoreus
      • (Done) Add and use stable identifier file for arguments @chicoreus guids stable in reexports of tests from markdown tables and rebuilds of rdf as of 4c56c3c see that commit message for commits in bdq_issue_to_csv and kurator-ffdq.
• (Done) Add a vocabulary extension file for bdqffdq @chicoreus
• (Done) Build Vocabulary term list files in _review/docs/list/{term}/index.md files @chicoreus
  • (Done) Build script for term list files modified to run with local configuration files for draft standard @chicoreus
  • (Done) Configure and run for bdq, bdqdim, bdqenh, bdqcrit vocabularies. @chicoreus
  • (Done) Update Vocabulary Landing page, point links to correct targets (generated landing pages and term-list documents)
• (Done) Review human readable term-list files in
  • (Done) https://github.com/tdwg/bdq/blob/master/tg2/_review/docs/list/bdq/index.md @ArthurChapman
  • (Done) https://github.com/tdwg/bdq/blob/master/tg2/_review/docs/list/bdqdim/index.md @ArthurChapman
  • (Done) https://github.com/tdwg/bdq/blob/master/tg2/_review/docs/list/bdqcrit/index.md @ArthurChapman
  • (Done) https://github.com/tdwg/bdq/blob/master/tg2/_review/docs/list/bdqenh/index.md @ArthurChapman
  • (Done) Checked for consistency of headings between all four documents. Header files updated @ArthurChapman @chicoreus
• (Done) Document and close Issue BDQ Core - VOCABULARY of terms #152 as this vocabulary has been split into separate files and added to _review @ArthurChapman
• (Done) Add remaining glossary items from BDQ Core - VOCABULARY of terms #152 that aren't in vocabularies into glossary section of introduction @chicoreus
• (Done) Add vocabularies and pages for terms to apply to Criterion and Enancement fundamental concepts @chicoreus
• (Done) Hold discussion with Cecilie Svenningsen (GBIF) on Vocabularies @ArthurChapman
• (Done) Provide GBIF with list of future Vocabularies needed for Data Quality @ArthurChapman
• (Done) Review bdq term list header document @Tasilee
• (Done) Build term key for each vocabulary, or pull in a term table from relevant vocabs. @chicoreus, @ArthurChapman, @Tasilee, @tucotuco
• Confirm that names of vocabularies, their terms, associated documents, and versioned terms conform to the expectations of the standards metadata document https://tdwg.github.io/rs.tdwg.org/ and the SDS @chicoreus @tucotuco

Quick Reference Guide

• (Done) Create Preface for the Quick Reference Guide @Tasilee
• (Done) Decide what descriptors that will be in the Quick Reference Guide @Tasilee
• (Done) Build bdqcore Quick Reference Guide @Tasilee
• (Done) Create draft QRG Footer (Citation based on Darwin Core QRG) @Tasilee
• (Done) Python script to produce [quick reference guide] @chicoreus (https://github.com/tdwg/bdq/blob/master/tg2/_review/docs/terms/bdqcore/index.md) for combined singlerecord and multirecord bdqcore tests @chicoreus .
• (Done) Incorporate this script with header/footer assembly @tucotuco @chicoreus

Users Guide

• (Done) Complete template in _review/build/template/guide/bdqffdq/ @chicoreus
• (Done) Generate TOC (sections for populating) @chicoreus
• (Done) Script to build and assemble materials into _review/docs/guides directory @chicoreus
• (Done) Review/edit document @Tasilee
• (Done) Review/edit document @ArthurChapman
• (Done) Request review of sections by team members @chicoreus
• Review content to confirm no missing assumptions in understanding and clarity of text for consumers.

Implementer's Guide

• (Done) Complete template in _review/build/template/guide/bdqffdq/ @chicoreus
• (Done) Generate TOC (sections for populating) @chicoreus
• (Done) Script to build and assemble materials into _review/docs/guides directory @chicoreus
• (Done) Review document @Tasilee
• (Done) Review/edit document @ArthurChapman
• (Done) Fix Sections 8.2 and 8.3 @Tasilee @chicoreus
• (Done) Add in Namespace Abbreviations (section 1.4) @chicoreus
• (Done) Allocate sections to team members (https://github.com/tdwg/bdq/blob/master/tg2/_review/docs/implementers/index.md)
• (Done) Add in section 6.1 Notes on how to change a Parameter @chicoreus
• (Done) Add note on stub code generation by kurator-ffdq @chicoreus

bdqffdq Guide

• (Done) Complete template in _review/build/template/guide/bdqffdq/ @chicoreus
• (Done) Create visualizations to explain ontology, add to bdqffdq guide page. @chicoreus (files are in _review/build/template/guide/bdqffdq/)
• (Done) Edit the above ontology visualizations @ArthurChapman
• (Done) Script to build and assemble materials into _review/docs/guides directory @chicoreus

Standards Maintenance Document

• (Done) Stub for Standard Maintenance document
• (Done) Add a section on Test Data @Tasilee
• (Done) Add a section on supporting code (bdq_issue_to_csv, bdqtestrunner, kurator-ffdq) @chicoreus

Tests

• (Done) Review/edit (English) Description in the light of current Expected Responses (CORE) @ArthurChapman
• (Done) Add skos:prefLabel for multi-record measures in bdq_issue_to_csv @chicoreus
• (Done) Build script that copies latest validation data files (test data and non-printing characters) into _review directory. @chicoreus See _make_review/copy_files.sh
• (Done) Add implementation for TG2-ISSUE_COORDINATES_CENTEROFCOUNTRY #287 run against validation data. @chicoreus
• (Done) Check generated Examples in Test Data match those in the Tests and align accordingly @Tasilee
• (Done) Check and align inconsistencies in phrasing in the Specification of "in bdq:sourceAuthority" vs "in the bdq:sourceAuthority" @Tasilee
• (Done) Check consistency of use of bdq:sourceAuthority and ISO references in all tests and align @Tasilee
• (Done) Complete review of FAIL cases in validation data, update test specifications, validation data, or implementation as needed (also under Test Data below) @chicoreus @Tasilee @ArthurChapman ** Filteredpush implementations are passing all validation rows as of v93
• (Done) Replace all instances of EMPTY and NOT EMPTY, etc. in the test specifications with bdq:Empty @Tasilee @ArthurChapman
• (Done) Complete conversion of CSV list of tests (produced by bdq_issue_to_csv) into a term-version file.**
  • (Done)Add status column, with 'recommended' as value. @chicoreus
  • (Done) Rename bdqcore_terms.csv to bdqcore_term_versions.csv. @chicoreus
  • (Done) Dowstream changes from changes to file. @chicoreus
• (Done) Update and add links to source code for implementations in test descriptors for filtered push libraries @chicoreus
• (Done) Freeze editing of markdown table representations of tests (decision by all)
  • (Done) Final export from markdown tables with bdq_issue_to_csv @chicoreus (was done, no changes subsequent to edits to tg2/core/TG2_tests.csv by @tucotuco, now master with tg2/core/TG2_multirecord_measure_tests.csv)
  • (Done) Close all open CORE issues @chicoreus
  • Then: Update rdf artifact for bdqcore generated with kurator-ffdq and other generated artifacts @chicoreus

Bibliography (References)

• (Done) Prepare intro for Bibliography @Tasilee
• (Done) Format RFC 2119 reference and include in reference list @Tasilee
• (Done) @Tasilee to send @ArthurChapman a list of References that need correction
• (Done) Correct References in the Test Issues @ArthurChapman, @Tasilee
• (Done) Prepare a combined list of References @Tasilee
• (Done) Update with additional new references @Tasilee
• (Done) Move references to a template, add header/footer/document configuration files, setup for either self standing references page or references included in pages when built @chicoreus current state is references embedded at build time into introduction and references as a separate document.
• New releases of related code, update references with links to zenodo, make sure each piece of software has text in the appropriate document with a citation. @chicoreus
  • rdfbeans
  • ffdq-api (v2.0.2) Lowery & Morris 2024 https://doi.org/10.5281/zenodo.14031588
  • kurator-ffdq (v3.0.0) Lowery et al. 2024 https://doi.org/10.5281/zenodo.14026643
  • bdq_issue_to_csv (v1.0.0) Morris 2024 https://doi.org/10.5281/zenodo.14057551
  • bdqtestrunner (v1.0.0) Morris 2024 https://doi.org/10.5281/zenodo.13932178
  • event_date_qc (v3.0.5) Morris & Lowery 2024 https://doi.org/10.5281/zenodo.14056663
  • sci_name_qc (v1.1.2) Morris & Dou 2024 https://doi.org/10.5281/zenodo.14052951
  • geo_ref_qc (v2.0.1). Morris, Lowery & Wieczorek 2025 https://doi.org/10.5281/zenodo.14064324
  • rec_occur_qc (v1.0.1) Morris 2025 https://doi.org/10.5281/zenodo.14968501

Authors and Contributors

• (Done) Prepare file on Authors and Contributions @chicoreus, @Tasilee, @ArthurChapman
• (Done) Add code library functions to insert authors/contributors sections in consistent form into templates @chicoreus .
• (Done) Chase up missing authors and ORCID_IDs @ArthurChapman @chicoreus
• (Done) Decide on breakdown of Authors, Contributors and Acknowledgements @tucotuco, @ArthurChapman, @Tasilee, @chicoreus**
• (Done) Assemble acknowlegements in introduction document. (Add configuration file for acknowledgements? No Seems a good idea that works with authors configuration file, but only needs listing in the intro/index.md document). @chicoreus @ArthurChapman @Tasilee
• (Done) Review/edit/re-order and standardize details on Authors and Contributors

Test Data

• (Done) Add new column for 'Date Last Updated' and populate with current date @Tasilee
• (Done) Rationalize (align for consistency) of Response.comments and align with Test Examples (e.g., tense, phrasing) @Tasilee
• (Done) Complete review of FAIL cases in validation data, update test specifications, validation data, or implementation as needed @chicoreus @Tasilee @ArthurChapman
• (Done) Resolve 3 apparent duplicate rows @Tasilee (in v91, fixed in v93) Note: All filtered push qc implementations passing all validation rows except the 3 that appear as duplicates, and the annotation test for which there is no implementation.

Consistency Checks

• (Done) Align (2) full and (2) summary sections about Test Types across 4 documents @Tasilee
• (Done) Review and make consistent descriptive text about test types on bdqcore landing page, bdqcore term-list page, supplement, users guide, implementers guide. @Tasilee @chicoreus
• (Done) Refine descriptive text about test types to match needs of local context and add normative statements where appropriate to bdqcore landing page, bdqcore term-list page, users guide, and implementers guide. @Tasilee @chicoreus
• Consistent Headings
  • (Done) Purpose and Audience statements for each document (@chicoreus added), needs refinement (@Tasilee).
  • (Done) Consistent parts and ordering for section 1 of documents, following expectations of SDS (first pass by @chicoreus done, second pass by @Tasilee done, third pass by @chicoreus done).
• Check links
  • linkchecker https://github.com/tdwg/bdq/blob/master/tg2/_review/index.md
  • Check links to anchors in documents (multiple anchor references are broken).

General

• (Done) Identify potential Review Managers
• (Done) Prepare and send off Report to Executive (All)
• (Done) Make verbal report and summary to TDWG Business Meeting at TDWG2024 @ArthurChapman
• (Done) Prepare diagrams for each of the test types (NAME, SPACE, TIME, OTHER) showing linkages between the tests @ArthurChapman
• (Done) Templates and build scripts for all /docs/ pages except landing page for standard, maintinance document, and references page. @chicoreus
• (Done) Add definitions for IRI, term_localName, termName, etc. to build_term_key() in function_lib.py @chicoreus
• (Done) Add Section in all Footer Documents for Acronyms and Glossary linking to Tables in Introduction and for References linking to the Reference document @ArthurChapman
• (Done) Link to glossary from all footer pages.
• (Done) Check all documents for Acronyms and add any missing ones to the Table in 5 Acronyms @Tasilee
• (Done) Run a spell/typo check on all major text documents @Tasilee
• (Done) Simplify and standardize usage of "framework", "Framework", "bdqffdq: Framework" and "Framework ontology" etc. All to agree, and then @Tasilee to carefully align text
• Produce RDF triples from the diagrams using showing linkages between the tests using improvedBy, targetedIssue, targetedMeasure, targetedValidation terms @ArthurChapman (??I'm still not sure what needs to be done here @chicoreus?? @chicoreus has resent email describing this.)
• Identify potential Maintenance Managers
• Final (editorial) review of all documents :) @tucotuco
• Final check of links in documents, in case of moved targets etc @chicoreus

External

• File issue against TDWG SDS for need to have a key for terms
• Confirm that Issues related to Darwin Core have been filed or commented on in dwc repository

[chicoreus]

Fixed the nested list formating in the list of actions, see: https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#nested-lists

[chicoreus]

Have we got agreement that we are ready to "Freeze editing of markdown table representations of tests (decision by all)"? @Tasilee @ArthurChapman @tucotuco @ymgan ?

[chicoreus]

Per decision above, All CORE issues in BDQ Core are now closed. See note in https://github.com/tdwg/bdq/blob/master/tg2/_review/README.md for location of current master copies of the Test descriptors.