Errors reported incorrectly when handling too many files #2081

mateuszpawlik · 2024-08-15T10:07:41Z

I'm executing BIDS Validator v1.14.8 on a large dataset (~800GB, ~4500 files). The validator reports incorrectly the following errors:

[ERR] You have to define 'RepetitionTime' for this file. (code: 10 - REPETITION_TIME_MUST_DEFINE)
[ERR] You have to define 'EchoTime1' and 'EchoTime2' for this file. (code: 15 - ECHO_TIME1-2_NOT_DEFINED)
[ERR] You have to define 'TaskName' for this file. (code: 50 - TASK_NAME_MUST_DEFINE)

These values are present in the JSON files.

It seems that the validator doesn't consider the JSON files. Is it possible that I reach some limit. It's not memory because the validator executes and finishes.

When I bidsignore half of the subjects, the validation passes.

I'm happy to do more investigation but I'd need to know what.

mateuszpawlik · 2024-08-16T15:09:28Z

I understand that legacy validator may be of lower priority, but is there anything I can do to fix this?

effigies · 2024-08-16T15:25:09Z

I personally don't have any notion how to start investigating this. Are you also seeing the issue with the schema validator?

mateuszpawlik · 2024-08-16T15:27:59Z

If you mean the Deno based validator, I'm planning to execute it. I'm fetching the data now. Once I do it, I will report, but possible next week. Thanks.

mateuszpawlik · 2024-08-19T09:45:06Z

I've tried the Deno validator. It also reports weird issues for most of the files, which should not be reported. I checked some of the reported files manually and there should be no issue.

I used this command:

docker run -ti --rm -v $PWD:/data:ro denoland/deno deno run --allow-read --allow-env https://deno.land/x/[email protected]/bids-validator.ts /data

[WARNING] NIfTI file's header field for dimension information is blank or too short. (NIFTI_DIMENSION)
[WARNING] NIfTI file's header field for pixel dimension information is empty or too short. (NIFTI_PIXDIM)
[WARNING] A data file's JSON sidecar is missing a key listed as recommended. (SIDECAR_KEY_RECOMMENDED)
[ERROR] Repetition time did not match between the scan's header and the associated JSON metadata file. (REPETITION_TIME_MISMATCH)
[WARNING] You must define 'EchoTime' for this file. 'EchoTime' is the echo time (TE)
for the acquisition, specified in seconds. (...)

I also get "[WARNING] The onset column in events.tsv files should be sorted. (EVENT_ONSET_ORDER)". My guess is, that this is because we have negative onsets at the beginning.

The number of files seems to be reported correctly in the summary of both validators: 4428 Files.

Update: These issues show even if I bidsignore all subjects but one.

mateuszpawlik · 2024-08-19T12:55:59Z

We had some invalid json files in the dataset. We new about them. Fixing them caused the legacy validator to work more as expected, not reporting the errors, which I listed originally. It did not help the issues reported by the Deno validator.

It seems that something is happening internally, which causes throwing other errors when some json files are invalid. I remember reading about similar issues. Are there maybe any errors set by default and reported when something else breaks?

effigies · 2024-08-19T13:38:54Z

I've tried the Deno validator. It also reports weird issues for most of the files, which should not be reported. I checked some of the reported files manually and there should be no issue.

I used this command:
docker run -ti --rm -v $PWD:/data:ro denoland/deno deno run --allow-read --allow-env https://deno.land/x/[email protected]/bids-validator.ts /data

We've been making a bunch of fixes in the last week or so since that release, so can you re-test with:

deno run --reload -A https://github.com/bids-standard/bids-validator/raw/master/bids-validator/src/bids-validator.ts?

* [WARNING] NIfTI file's header field for dimension information is blank or too short. (NIFTI_DIMENSION)

This means that either dim[0] == 0 or min(dim[1:dim[0]]) <= 0. If that's not the case, that's a bug. If you could share the NIfTI header, I could test this. Simplest way to share:

python -c "import sys, nibabel; print(nibabel.load(sys.argv[1]).to_bytes()[:348])" <PATH>

* [WARNING] NIfTI file's header field for pixel dimension information is empty or too short. (NIFTI_PIXDIM)

min(pixdim[1:dim[0]]) <= 0

* [WARNING] A data file's JSON sidecar is missing a key listed as recommended. (SIDECAR_KEY_RECOMMENDED)

This is almost impossible not to have, and a result of the schema validator systematically reporting RECOMMENDED sidecar fields. It's going to be noisy because BIDS has a lot of these, but they've only been selectively applied in the legacy validator. (Preview: I'm going to be advocating for reducing many fields to OPTIONAL.)

* [ERROR] Repetition time did not match between the scan's header and the associated JSON metadata file. (REPETITION_TIME_MISMATCH)

This could be a rounding problem: https://github.com/bids-standard/bids-validator/issues/2091

* [WARNING] You must define 'EchoTime' for this file. 'EchoTime' is the echo time (TE)
  for the acquisition, specified in seconds. (...)

This is a problem in the schema, I'm going to submit a patch today.

I also get "[WARNING] The onset column in events.tsv files should be sorted. (EVENT_ONSET_ORDER)". My guess is, that this is because we have negative onsets at the beginning.

This is surprising. I've just checked the sorting function and it should handle negative numbers fine. Would you open an issue with a failing file?

mateuszpawlik · 2024-08-19T13:40:56Z

Now I'm getting the following error with the legacy validator, which doesn't make sense to me, because why the json files in phenotype directory should be validated against a schema and have the listed properties.

[ERR] Invalid JSON file. The file is not formatted according the schema. (code: 55 - JSON_SCHEMA_VALIDATION_ERROR)
                ./phenotype/ASR.json
                        Invalid JSON file. The file is not formatted according the schema.
                        Evidence:  should have required property 'PixelSize'
                ./phenotype/ASR.json
                        Invalid JSON file. The file is not formatted according the schema.
                        Evidence:  should have required property 'PixelSizeUnits'

Please visit https://neurostars.org/search?q=JSON_SCHEMA_VALIDATION_ERROR for existing conversations about this issue.

effigies · 2024-08-19T13:42:38Z

That looks like it's being picked up by a microscopy rule.

effigies · 2024-08-19T13:42:50Z

It seems that something is happening internally, which causes throwing other errors when some json files are invalid. I remember reading about similar issues. Are there maybe any errors set by default and reported when something else breaks?

I don't know about that, but I've only been tangentially involved in the legacy validator. @rwblair might remember something here?

mateuszpawlik · 2024-08-19T14:16:29Z

That looks like it's being picked up by a microscopy rule.

That is weird but thank you. Renaming the file resolves this issue but it's not really a solution.

mateuszpawlik · 2024-08-19T14:20:29Z

We've been making a bunch of fixes in the last week or so since that release, so can you re-test with:

deno run --reload -A https://github.com/bids-standard/bids-validator/raw/master/bids-validator/src/bids-validator.ts?

docker run -ti --rm -v $PWD:/data:ro denoland/deno deno run --allow-read --allow-env --reload -A https://github.com/bids-standard/bids-validator/raw/master/bids-validator/src/bids-validator.ts /data
error: Relative import path "@std/path" not prefixed with / or ./ or ../
    at https://raw.githubusercontent.com/bids-standard/bids-validator/master/bids-validator/src/main.ts:5:25

This issue grew too much 🙈 We can split it.

effigies · 2024-08-19T14:30:13Z

Damn. Okay, apparently that method was disabled by #2077. Use https://github.com/bids-standard/bids-validator/raw/deno-build/bids-validator.js.

mateuszpawlik · 2024-08-19T15:07:52Z

I also get "[WARNING] The onset column in events.tsv files should be sorted. (EVENT_ONSET_ORDER)". My guess is, that this is because we have negative onsets at the beginning.

This is surprising. I've just checked the sorting function and it should handle negative numbers fine. Would you open an issue with a failing file?

This problem we actually found in our data 🙈

mateuszpawlik · 2024-08-19T15:17:30Z

Damn. Okay, apparently that method was disabled by #2077. Use https://github.com/bids-standard/bids-validator/raw/deno-build/bids-validator.js.

The command I used:

docker run -ti --rm -v $PWD:/data:ro denoland/deno deno run --allow-read --allow-env --reload -A https://github.com/bids-standard/bids-validator/raw/deno-build/bids-validator.js /data

The entire output:

TypeError: Attempted to load JSON module without specifying "type": "json" attribute in the import statement.
    at async loadSchema (https://raw.githubusercontent.com/bids-standard/bids-validator/deno-build/bids-validator.js:22:26)
    at async validate (https://raw.githubusercontent.com/bids-standard/bids-validator/deno-build/bids-validator.js:45:18)
    at async main (https://raw.githubusercontent.com/bids-standard/bids-validator/deno-build/bids-validator.js:19:24)
    at async https://raw.githubusercontent.com/bids-standard/bids-validator/deno-build/bids-validator.js:3:16 {
  code: "ERR_MODULE_NOT_FOUND"
}
Warning, could not load schema from https://bids-specification.readthedocs.io/en/latest/schema.json, falling back to internal version
        [WARNING] TOO_FEW_AUTHORS The 'Authors' field of 'dataset_description.json' should contain an array of values -
with one author per value.
This was triggered based on the presence of only one author field.
Please ignore if all contributors are already properly listed.

                /dataset_description.json

        Please visit https://neurostars.org/search?q=TOO_FEW_AUTHORS for existing conversations about this issue.

        [WARNING] JSON_KEY_RECOMMENDED A JSON file is missing a key listed as recommended.
                License
                /dataset_description.json

                SourceDatasets
                /dataset_description.json

        Please visit https://neurostars.org/search?q=JSON_KEY_RECOMMENDED for existing conversations about this issue.

        [WARNING] NO_AUTHORS The Authors field of dataset_description.json should contain an array of fields -
with one author per field. This was triggered because there are no authors, which
will make DOI registration from dataset metadata impossible.

                Authors
                /dataset_description.json

        Please visit https://neurostars.org/search?q=NO_AUTHORS for existing conversations about this issue.

        [WARNING] SIDECAR_KEY_RECOMMENDED A data file's JSON sidecar is missing a key listed as recommended.
                GradientSetType
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval

                2378 more files with the same issue

                MRTransmitCoilSequence
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval

                2378 more files with the same issue

                PulseSequenceType
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval

                2378 more files with the same issue

                MTState
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval

                2378 more files with the same issue

                SpoilingType
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-mov/anat/sub-s001_ses-mov_acq-hcpli_T1w.nii

                87 more files with the same issue

                NumberShots
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval

                2378 more files with the same issue

                ParallelReductionFactorOutOfPlane
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval

                2378 more files with the same issue

                ParallelAcquisitionTechnique
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval

                2378 more files with the same issue

                PartialFourierDirection
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval

                2378 more files with the same issue

                EffectiveEchoSpacing
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/fmap/sub-s001_ses-dwi_acq-gefmsl56_magnitude1.nii

                625 more files with the same issue

                MixingTime
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval

                2378 more files with the same issue

                PhaseEncodingDirection
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv

                385 more files with the same issue

                TotalReadoutTime
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/fmap/sub-s001_ses-dwi_acq-gefmsl56_magnitude1.nii

                625 more files with the same issue

                MultibandAccelerationFactor
                /sub-s001/ses-dwi/anat/sub-s001_ses-dwi_acq-hcpli_T1w.nii
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_sbref.nii

                575 more files with the same issue

                SpoilingState
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bvec

                2289 more files with the same issue

                ParallelReductionFactorInPlane
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bvec

                749 more files with the same issue

                InversionTime
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bvec

                2307 more files with the same issue

                SliceEncodingDirection
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bval
                /sub-s001/ses-dwi/dwi/sub-s001_ses-dwi_acq-256dir_dir-AP_dwi.bvec

                1973 more files with the same issue

                ScanOptions
                /sub-s001/ses-dwi/fmap/sub-s001_ses-dwi_acq-gefmsl56_magnitude1.nii
                /sub-s001/ses-dwi/fmap/sub-s001_ses-dwi_acq-gefmsl56_magnitude2.nii

                536 more files with the same issue

                NumberOfVolumesDiscardedByScanner
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-1_bold.nii
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-2_bold.nii

                1538 more files with the same issue

                DelayTime
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-1_bold.nii
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-2_bold.nii

                1538 more files with the same issue

                AcquisitionDuration
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-1_bold.nii
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-2_bold.nii

                1538 more files with the same issue

                DelayAfterTrigger
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-1_bold.nii
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-2_bold.nii

                1538 more files with the same issue

                Instructions
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-1_bold.nii
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-2_bold.nii

                1854 more files with the same issue

                TaskDescription
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-1_bold.nii
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-2_bold.nii

                1854 more files with the same issue

                CogAtlasID
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-1_bold.nii
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-2_bold.nii

                1854 more files with the same issue

                CogPOID
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-1_bold.nii
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-2_bold.nii

                1854 more files with the same issue

                TaskName
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv

                630 more files with the same issue

                InstitutionName
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv

                630 more files with the same issue

                InstitutionAddress
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv

                630 more files with the same issue

                InstitutionalDepartmentName
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv

                630 more files with the same issue

                StimulusPresentation
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                Manufacturer
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                ManufacturersModelName
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                DeviceSerialNumber
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                StationName
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                SoftwareVersions
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                MagneticFieldStrength
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                ReceiveCoilName
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                ReceiveCoilActiveElements
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                MatrixCoilMode
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                CoilCombinationMethod
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                ScanningSequence
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                SequenceVariant
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                SequenceName
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                PulseSequenceDetails
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                PartialFourier
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                DwellTime
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

                FlipAngle
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

        Please visit https://neurostars.org/search?q=SIDECAR_KEY_RECOMMENDED for existing conversations about this issue.

        [WARNING] ECHO_TIME_NOT_DEFINED You must define 'EchoTime' for this file. 'EchoTime' is the echo time (TE)
for the acquisition, specified in seconds. Corresponds to DICOM Tag
0018, 0081 Echo Time (please note that the DICOM term is in milliseconds
not seconds). The data type number may apply to files from any MRI modality
concerned with a single value for this field, or to the files in a file
collection where the value of this field is iterated using the echo entity.
The data type array provides a value for each volume in a 4D dataset and
should only be used when the volume timing is critical for interpretation
of the data, such as in ASL or variable echo time fMRI sequences.

                EchoTime
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_events.tsv
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-2_events.tsv

                314 more files with the same issue

        Please visit https://neurostars.org/search?q=ECHO_TIME_NOT_DEFINED for existing conversations about this issue.

        [WARNING] SUSPICIOUSLY_LONG_EVENT_DESIGN The onset of the last event is after the total duration of the corresponding scan.
This design is suspiciously long.

                /sub-s001/ses-mov/func/sub-s001_ses-mov_task-moviepc_acq-mb4mesl56_echo-1_bold.nii
                /sub-s001/ses-mov/func/sub-s001_ses-mov_task-moviepc_acq-mb4mesl56_echo-2_bold.nii

                74 more files with the same issue

        Please visit https://neurostars.org/search?q=SUSPICIOUSLY_LONG_EVENT_DESIGN for existing conversations about this issue.

        [WARNING] EVENT_ONSET_ORDER The onset column in events.tsv files should be sorted.

                /sub-s008/ses-dwi/func/sub-s008_ses-dwi_task-vw_acq-mb4mesl56_run-1_events.tsv
                /sub-s008/ses-dwi/func/sub-s008_ses-dwi_task-vw_acq-mb4mesl56_run-2_events.tsv

                122 more files with the same issue

        Please visit https://neurostars.org/search?q=EVENT_ONSET_ORDER for existing conversations about this issue.

        [ERROR] PARTICIPANT_ID_MISMATCH Participant labels found in this dataset did not match the values in participant_id column
found in the participants.tsv file.

                /participants.tsv

        Please visit https://neurostars.org/search?q=PARTICIPANT_ID_MISMATCH for existing conversations about this issue.

        [ERROR] REPETITION_TIME_MISMATCH Repetition time did not match between the scan's header and the associated JSON metadata file.

                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-1_bold.nii
                /sub-s001/ses-dwi/func/sub-s001_ses-dwi_task-aw_acq-mb4mesl56_run-1_echo-2_bold.nii

                1538 more files with the same issue

        Please visit https://neurostars.org/search?q=REPETITION_TIME_MISMATCH for existing conversations about this issue.


          Summary:                          Available Tasks:        Available Modalities:
          4428 Files, 669 GB                aw                      MRI                  
          23 - Subjects 4 - Sessions        rest                                         
                                            vw                                           
                                            moviehcp1                                    
                                            moviepc                                      

        If you have any questions, please post on https://neurostars.org/tags/bids.

The NIFTI header warnings are not there anymore.

We are aware of some of the issues but some seem not right:

[WARNING] SIDECAR_KEY_RECOMMENDED - especially these reported on *events.tsv files
[ERROR] REPETITION_TIME_MISMATCH - I will check the values

effigies · 2024-08-19T15:21:38Z

Agreed, it looks like the schema needs tightening up to avoid trying to apply sidecar rules for data files to events.tsv.

mateuszpawlik · 2024-08-19T16:15:35Z

@effigies, thank you for all your help. Many things happened in this issue. I'm wandering how to proceed. Feel free to rename this issue to reflect more its contents.

As a summary, from my side I see the following remaining problems:

Legacy validator reports "random" errors when JSON files are invalid. For me it was three JSON files in ./phenotype directory.
Legacy validator treats ./phenotype/ASR.json file as microscopy file and reports missing properties. This JSON file is valid. Renaming this file solves the issue.
Deno validator applies sidecar rules to *events.tsv files.
Deno validator reports REPETITION_TIME_MISMATCH although the values are equal.

Please let me know if you'd like me to report anything else or open separate issues for any of these problems.

effigies added the legacy label Aug 15, 2024

effigies mentioned this issue Aug 19, 2024

[Firefox] Loading module from "...schema.json" was blocked because of a disallowed MIME type ("application/json"). #2087

Closed

mateuszpawlik mentioned this issue Aug 22, 2024

Differences in legacy and schema RepetitionTime check. #2091

Closed

effigies mentioned this issue Aug 22, 2024

xml2js is breaking the deno build #2105

Closed

This was referenced Sep 9, 2024

deno: SIDECAR_KEY_RECOMMENDED complains about FlipAngle on _event.tsv file! #2128

Closed

Notes for running new deno validator on HPC using singularity #2129

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Errors reported incorrectly when handling too many files #2081

Errors reported incorrectly when handling too many files #2081

mateuszpawlik commented Aug 15, 2024 •

edited

Loading

mateuszpawlik commented Aug 16, 2024

effigies commented Aug 16, 2024

mateuszpawlik commented Aug 16, 2024

mateuszpawlik commented Aug 19, 2024 •

edited

Loading

mateuszpawlik commented Aug 19, 2024 •

edited

Loading

effigies commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

effigies commented Aug 19, 2024

effigies commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

effigies commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

effigies commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

Errors reported incorrectly when handling too many files #2081

Errors reported incorrectly when handling too many files #2081

Comments

mateuszpawlik commented Aug 15, 2024 • edited Loading

mateuszpawlik commented Aug 16, 2024

effigies commented Aug 16, 2024

mateuszpawlik commented Aug 16, 2024

mateuszpawlik commented Aug 19, 2024 • edited Loading

mateuszpawlik commented Aug 19, 2024 • edited Loading

effigies commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

effigies commented Aug 19, 2024

effigies commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

effigies commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

effigies commented Aug 19, 2024

mateuszpawlik commented Aug 19, 2024

mateuszpawlik commented Aug 15, 2024 •

edited

Loading

mateuszpawlik commented Aug 19, 2024 •

edited

Loading

mateuszpawlik commented Aug 19, 2024 •

edited

Loading