Review Handling of Special Constants, Field Formats, and High Priority Bug Fixes

[jordanpadams]

💡 Description

It appears we have some improvements needed for handling of special constants and field formats according to the standards and what errors/warnings/info messages should be output to to the user.

👶 Child Tasks

• Validate stalls when validating collection inventory file with duplicates records #849
• validate 3.5.0-SNAPSHOT attempts to look at all files, not just *.xml #854
• Validate 3.5.0-snapshot Giving Unexpected Errors for Matching Filenames in Separate Directories #873
• SXXP0003 Error when running on PDS3 data sets for v3.3.3 or v3.4.1 #874
• Validate failing products that contain "collection" not in the beginning of filename #880
• validate incorrectly handles special constant high_instrument_saturation #831
• As a user, I want an ERROR to be thrown when a character table value does not match the expected validation_format #817
• As a user, I want a WARNING to be thrown when a delimited or character table value does not match the expected field_format #816
• In text tables, validate attempts to match pattern associated with data_type before checking Special_Constants #837
• Validate not retrying on sch load failure #903

[msbentley]

Hi @jordanpadams @jmafi Special_Constants is on my list of "things to raise with DDWG" so before we make too many changes to validate, I think a discussion is needed here, in particular wrt how we handle IEEE754 data + special constants...

[al-niessner]

@jordanpadams

If I understand #816 and #817 correctly, then check_format() needs to change its error if field or validation format and all errors in this function need to have error/warning counterparts. Leave the message the same otherwise I hope. Correct?

[jordanpadams]

@al-niessner correct. right now we are treating all precision the same. We need to know not just that there is a precision value, but whether or not it is exact precision (validation_format) vs. max precision (field_format). So I think the current check here should be a warning for field_format, but we need an additional check for not equal for validation_format

[jordanpadams]

All sub-tasks completed. Closing out