🐛 Dictionary migration doesn't check program or entity level exceptions

[hknahal]

Describe the bug

If program or entity level exceptions are deleted, and then migrate to new dictionary, it does not trigger system to check program or entity level exceptions.

Steps To Reproduce

1. Enabled program level exception for specimen schema to allow tumour_grading_system and tumour_grade fields to be Unknown.
2. Submitted specimen clinical file with tumour_grading_system = Unknown and tumour_grade = Unknown. Validated and signed-off on clinical data.
3. Deleted program level exception for specimen schema (no longer allows tumour_grading_system and tumour_grade to be Unknown).
4. Migrated to new dictionary. But no errors reported in Submitted Data view for previously submitted tumour_grading_system and tumour_grade fields if they are submitted as Unknown.

It appears dictionary migration doesn't trigger the system to check the program/entity level exceptions when re-validating.

Expected behaviour

After dictionary migration, expected error to be reported in Submitted Data view for fields that are submitted as Unknown but have no exceptions applied anymore.

[joneubank]

@hknahal Question: what is the expected behaviour when an exception is removed? It's possible we need the system to re-evaluate all the data at that point instead of waiting for new submission or migration.

[hknahal]

@joneubank If an exception is deleted, the submitted data should be re-validated. For example, after Step #3 above (when exceptions for tumour_grading_system and tumour_grade are deleted), any fields that still have tumour_grading_system = Unknown or tumour_grade = Unknown should become invalid.

[joneubank]

I agree. To my knowledge, this is not done today?

I think we need to track two desired features:

1. re-validate after exceptions are modified
2. include exceptions in validations during migration (all validations should account for migrations)

[hknahal]

No, it's not done yet. I agree, the two features you mention would resolve this issue.

[Buwujiu]

@joneubank @hknahal Do we want to release exceptions now or wait until we implement these two features?

[joneubank]

I would release and give us the ability to accept data with exceptions. Removing exceptions is a use case we don't have immediate expectation to need.

[hknahal]

I think we should go ahead and release exceptions instead of waiting. This won't block any of the programs.

[Buwujiu]

Same thoughts here! I will make a plan to track these two issues, and we shall be able to prepare for the release!