Clarify how to behave when a File object without format field is given for the File parameter with format field

[tom-tan]

• The sections of CommandInputParameter and CommandInputRecordField describe the format field as follows:

  This must be one or more IRIs of concept nodes that represents file formats which are allowed as input to this parameter, preferably defined within an ontology. If no ontology is available, file formats may be tested by exact match.

• The section of File includes more precise description of the format field as follows:

  Reasoning about format compatibility must be done by checking that an input file format is the same, owl:equivalentClass or rdfs:subClassOf the format required by the input parameter. owl:equivalentClass is transitive with rdfs:subClassOf, e.g. if <B> owl:equivalentClass <C> and <B> owl:subclassOf <A> then infer <C> owl:subclassOf <A>.

Although they describe how to validate the values of format field, they do not specify how the platforms behave in the following cases:

1. the case when the format field in the input File object and type declaration is compatible,
2. the case when the format field in the input File object and type declaration is incompatible,
3. the case when a type declaration gives the format field but an input File object does not, or
4. the case when a type declaration does not specify the format field but an input File object gives.

Although the case 1 and 2 are already covered by the conformance tests, the case 3 and 4 are not covered by the spec nor the conformance tests.
It would be nice if we can add descriptions for them.

The following is a current status of the conformance tests for CWL v1.3 and current behavior of cwltool.

Conformance tests for CWL v1.3

• The case 1 is covered by format_checking and is intended to be passed.
• The case 2 is covered by input_records_file_entry_with_format_and_bad_regular_input_file_format and is intended to be failed.
• The case 3 and 4 are not covered yet.

Current behavior of cwltool

I checked the case 3 and 4.

• The case 3:
  • tool: tests/formattest.cwl
  • job (created): tests/formattest3-job.json as shown below

    {
        "input": {
            "class": "File",
            "location": "whale.txt"
        }
    }

  • output of cwltool (it fails):

    $ cwltool --enable-dev tests/formattest.cwl tests/formattest3-job.json 
    INFO /home/vscode/.local/bin/cwltool 3.1.20240708091338.dev12+g6d8c2a4
    INFO Resolved 'tests/formattest.cwl' to 'file:///workspaces/cwl-v1.3/tests/formattest.cwl'
    ERROR Workflow error, try again with --debug for more information:
    Expected value of 'input' to have format 'http://edamontology.org/format_2330' but
     File has no 'format' defined: {
        "class": "File",
        "location": "file:///workspaces/cwl-v1.3/tests/whale.txt",
        "size": 1111,
        "basename": "whale.txt",
        "nameroot": "whale",
        "nameext": ".txt"
    }

• The case 4:
  • tool: tests/io-file-or-files.cwl
  • job: tests/formattest-job.json
  • output of cwltool (it works):

    $ cwltool --enable-dev tests/io-file-or-files.cwl tests/formattest-job.json 
    INFO /home/vscode/.local/bin/cwltool 3.1.20240708091338.dev12+g6d8c2a4
    INFO Resolved 'tests/io-file-or-files.cwl' to 'file:///workspaces/cwl-v1.3/tests/io-file-or-files.cwl'
    INFO [job io-file-or-files.cwl] /tmp/f5eknfcv$ cat \
        /tmp/7v_zk20w/stge6ff24a5-7737-44fe-8834-82be1682f57b/whale.txt > /tmp/f5eknfcv/output.txt
    INFO [job io-file-or-files.cwl] completed success
    {
        "output_file": {
            "location": "file:///workspaces/cwl-v1.3/output.txt",
            "basename": "output.txt",
            "class": "File",
            "checksum": "sha1$327fc7aedf4f6b69a42a7c8b808dc5a7aff61376",
            "size": 1111,
            "path": "/workspaces/cwl-v1.3/output.txt"
        }
    }INFO Final process status is success

Summary

The case 3 and 4 are not covered by the spec and conformance tests but it would be nice if they are covered.

For me, current behavior of cwltool for the case 4 is reasonable but behavior of case 3 is too restrictive.
It would be nice if the platform only reports a warning for case 3 rather than causes permanentFailure.