Metadata that we'd like to store

[JackKelly]

Metadata about the entire dataset

• Mapping from dimension name to a list of coordinate labels for that dimension
• Array shape (although this can be inferred from the length of the coord label arrays)
• Associate time spans of the init datetimes with NWP model versions.
• Map NWP model version to:
  • Horizontal resolution & coord labels
  • Vertical resolution & coord labels
  • Number of ensemble members
  • Info about how to interpret the GRIB data sections? So we can just load the data section?
• Dataset name, author, etc.
• Data from the GRIB files:
  • From Section 1
    • Identification of originating/generating centre
    • Identification of originating/generating subcentre
    • GRIB master table version number (this may change over the course of the dataset)
    • Version number of GRIB Local tables used to augment Master tables (this may change)
    • Significance of reference time
  • From Section 3:
    • Physical Meaning of Vertical Coordinate (unless this can be unambiguously determined from the idx file?
  • A map from the parameter abbreviation string (e.g. "VTMP") to detailed information about that param (unit, description, etc):
  • A map from the step string to detailed information about that step (e.g. the important parts of the product template). (Perhaps we'll have to parse some actual GRIB files to get all these details?)

Metadata per GRIB message

(To start with, let's see how far we can get without defining an "extended idx" format! But it may be necessary to parse every GRIB file and extract an "extended idx". Except there are a HUGE number of GRIB files in NODD datasets, so this may be impractical!)

Additional to the information in existing .idx files:

• Length of each message
• "Sequences of GRIB sections 2 to 7, sections 3 to 7 or sections 4 to 7 may be repeated within a single GRIB message" (quoted from the WMO Manual on Codes). Describe the byte offset of each data section? Because we want to allow users to load only the data sections that they need.
• Maybe store the byte offset of the data sections? If our dataset-wide metadata accurately captures all we need to know about each data section then we only need to load exactly the data sections?

Related

• NWPs change shape over time. How to detect, store, and handle these changes? #15