Make tests/test_scripts.py that use pytest-console-scripts script_runner fixture OS independent

[matthewfeickert]

Anyway, the problem is the string:

command = f'pyhf xml2json validation/xmlimport_input/config/example.xml --basedir validation/xmlimport_input/ --output-file {temp.strpath:s} --hide-progress'
ret = script_runner.run(shlex.split(command))

Paths have backslashes on windows, which worries me a bit there, and https://docs.python.org/3/library/shlex.html#shlex.split says that posix=True is the default, but you aren't running on a posix shell on Windows. Why are you putting a string in then splitting it up? And if you use shlex.quote in the runner to put it back together, that's 100% UNIX only.

Looking at an example:

pyhf/tests/test_scripts.py

Lines 60 to 70 in 6203a8f

	def test_import_prepHistFactory(tmpdir, script_runner):
	temp = tmpdir.join("parsed_output.json")
	command = f'pyhf xml2json validation/xmlimport_input/config/example.xml --basedir validation/xmlimport_input/ --output-file {temp.strpath:s} --hide-progress'
	ret = script_runner.run(shlex.split(command))
	assert ret.success
	assert ret.stdout == ''
	assert ret.stderr == ''
	parsed_xml = json.loads(temp.read())
	spec = {'channels': parsed_xml['channels']}
	pyhf.schema.validate(spec, 'model.json')

command is a single string for convenience of readability for us as pytest-console-scripts expects a list of command fragrments that would be space seperated, similar to nox. This is not nice to look at once the commands get more than a few segements long.

Internally pytest-console-scripts only uses shlex.split, and we could do something similar to what they do with

import os
shlex.split(command, posix=os.name == 'posix')

Originally posted by @matthewfeickert in #2363 (comment) given feedback from @henryiii

(This is required to get the full test suite running on Windows)

[matthewfeickert]

Adding

import os
...
ret = script_runner.run(shlex.split(command, posix=(os.name == "posix")))

works, but then there's a new error for

pyhf/tests/test_scripts.py

Lines 170 to 183 in 188b7a0

	def test_import_usingMounts(datadir, tmpdir, script_runner):
	data = datadir.joinpath("xmlimport_absolutePaths")
	temp = tmpdir.join("parsed_output.json")
	command = f'pyhf xml2json --hide-progress -v {data}:/absolute/path/to -v {data}:/another/absolute/path/to --output-file {temp.strpath:s} {data.joinpath("config/example.xml")}'
	ret = script_runner.run(shlex.split(command))
	assert ret.success
	assert ret.stdout == ''
	assert ret.stderr == ''
	parsed_xml = json.loads(temp.read())
	spec = {'channels': parsed_xml['channels']}
	pyhf.schema.validate(spec, 'model.json')

of

_____________________ test_import_usingMounts[inprocess] ______________________

datadir = WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_import_usingMounts_inproc0')
tmpdir = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_usingMounts_inproc0')
script_runner = <ScriptRunner inprocess>

    def test_import_usingMounts(datadir, tmpdir, script_runner):
        data = datadir.joinpath("xmlimport_absolutePaths")
    
        temp = tmpdir.join("parsed_output.json")
        command = f'pyhf xml2json --hide-progress -v {data}:/absolute/path/to -v {data}:/another/absolute/path/to --output-file {temp.strpath:s} {data.joinpath("config/example.xml")}'
    
        ret = script_runner.run(shlex.split(command, posix=(os.name == "posix")))
>       assert ret.success
E       assert False
E        +  where False = <pytest_console_scripts.RunResult object at 0x000002794399A920>.success

command    = 'pyhf xml2json --hide-progress -v C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_...\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_usingMounts_inproc0\\xmlimport_absolutePaths\\config\\example.xml'
data       = WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_import_usingMounts_inproc0/xmlimport_absolutePaths')
datadir    = WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_import_usingMounts_inproc0')
ret        = <pytest_console_scripts.RunResult object at 0x000002794399A920>
script_runner = <ScriptRunner inprocess>
temp       = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_usingMounts_inproc0\\parsed_output.json')
tmpdir     = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_usingMounts_inproc0')

tests\test_scripts.py:178: AssertionError
----------------------------- Captured log setup ------------------------------
INFO     root:file_util.py:137 copying D:\a\pyhf\pyhf\tests\test_scripts\example_bkgonly.json -> C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_import_usingMounts_inproc0
INFO     root:file_util.py:137 copying D:\a\pyhf\pyhf\tests\test_scripts\example_patchset.json -> C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_import_usingMounts_inproc0
INFO     root:dir_util.py:71 creating C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_import_usingMounts_inproc0\xmlimport_absolutePaths
INFO     root:dir_util.py:71 creating C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_import_usingMounts_inproc0\xmlimport_absolutePaths\config
INFO     root:file_util.py:137 copying D:\a\pyhf\pyhf\tests\test_scripts\xmlimport_absolutePaths\config\example.xml -> C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_import_usingMounts_inproc0\xmlimport_absolutePaths\config
INFO     root:file_util.py:137 copying D:\a\pyhf\pyhf\tests\test_scripts\xmlimport_absolutePaths\config\example_channel.xml -> C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_import_usingMounts_inproc0\xmlimport_absolutePaths\config
INFO     root:file_util.py:137 copying D:\a\pyhf\pyhf\tests\test_scripts\xmlimport_absolutePaths\config\HistFactorySchema.dtd -> C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_import_usingMounts_inproc0\xmlimport_absolutePaths\config
INFO     root:dir_util.py:71 creating C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_import_usingMounts_inproc0\xmlimport_absolutePaths\data
INFO     root:file_util.py:137 copying D:\a\pyhf\pyhf\tests\test_scripts\xmlimport_absolutePaths\data\example.root -> C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_import_usingMounts_inproc0\xmlimport_absolutePaths\data
---------------------------- Captured stdout call -----------------------------
# Running console script: ['pyhf', 'xml2json', '--hide-progress', '-v', 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_usingMounts_inproc0\\xmlimport_absolutePaths:/absolute/path/to', '-v', 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_usingMounts_inproc0\\xmlimport_absolutePaths:/another/absolute/path/to', '--output-file', 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_usingMounts_inproc0\\parsed_output.json', 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_usingMounts_inproc0\\xmlimport_absolutePaths\\config\\example.xml']
# Script return code: 2
# Script stdout:

# Script stderr:
Usage: python -m pytest.pyhf xml2json [OPTIONS] ENTRYPOINT_XML
Try 'python -m pytest.pyhf xml2json -h' for help.

Error: Invalid value for '-v' / '--mount': 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_usingMounts_inproc0\\xmlimport_absolutePaths:/absolute/path/to' is not a valid colon-separated option

and for

pyhf/tests/test_scripts.py

Lines 527 to 543 in 188b7a0

	def test_combine_merge_channels(tmpdir, script_runner):
	temp_1 = tmpdir.join("parsed_output.json")
	temp_2 = tmpdir.join("renamed_output.json")
	command = f'pyhf xml2json validation/xmlimport_input/config/example.xml --basedir validation/xmlimport_input/ --output-file {temp_1.strpath} --hide-progress'
	ret = script_runner.run(shlex.split(command))
	assert ret.success
	command = (
	f'pyhf prune {temp_1.strpath} --sample signal --output-file {temp_2.strpath}'
	)
	ret = script_runner.run(shlex.split(command))
	assert ret.success
	command = f'pyhf combine --merge-channels --join "left outer" {temp_1.strpath} {temp_2.strpath}'
	ret = script_runner.run(shlex.split(command))
	assert ret.success

of

___________________ test_combine_merge_channels[inprocess] ____________________

tmpdir = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_combine_merge_channels_in0')
script_runner = <ScriptRunner inprocess>

    def test_combine_merge_channels(tmpdir, script_runner):
        temp_1 = tmpdir.join("parsed_output.json")
        temp_2 = tmpdir.join("renamed_output.json")
        command = f'pyhf xml2json validation/xmlimport_input/config/example.xml --basedir validation/xmlimport_input/ --output-file {temp_1.strpath} --hide-progress'
        ret = script_runner.run(shlex.split(command, posix=(os.name == "posix")))
        assert ret.success
    
        command = (
            f'pyhf prune {temp_1.strpath} --sample signal --output-file {temp_2.strpath}'
        )
    
        ret = script_runner.run(shlex.split(command, posix=(os.name == "posix")))
        assert ret.success
    
        command = f'pyhf combine --merge-channels --join "left outer" {temp_1.strpath} {temp_2.strpath}'
        ret = script_runner.run(shlex.split(command, posix=(os.name == "posix")))
>       assert ret.success
E       assert False
E        +  where False = <pytest_console_scripts.RunResult object at 0x000002795EED5F60>.success

command    = 'pyhf combine --merge-channels --join "left outer" C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin...nneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_combine_merge_channels_in0\\renamed_output.json'
ret        = <pytest_console_scripts.RunResult object at 0x000002795EED5F60>
script_runner = <ScriptRunner inprocess>
temp_1     = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_combine_merge_channels_in0\\parsed_output.json')
temp_2     = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_combine_merge_channels_in0\\renamed_output.json')
tmpdir     = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_combine_merge_channels_in0')

tests\test_scripts.py:548: AssertionError
---------------------------- Captured stdout call -----------------------------
# Running console script: ['pyhf', 'xml2json', 'validation/xmlimport_input/config/example.xml', '--basedir', 'validation/xmlimport_input/', '--output-file', 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_combine_merge_channels_in0\\parsed_output.json', '--hide-progress']
# Script return code: 0
# Script stdout:

# Script stderr:

# Running console script: ['pyhf', 'prune', 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_combine_merge_channels_in0\\parsed_output.json', '--sample', 'signal', '--output-file', 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_combine_merge_channels_in0\\renamed_output.json']
# Script return code: 0
# Script stdout:

# Script stderr:

# Running console script: ['pyhf', 'combine', '--merge-channels', '--join', '"left outer"', 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_combine_merge_channels_in0\\parsed_output.json', 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_combine_merge_channels_in0\\renamed_output.json']
# Script return code: 2
# Script stdout:

# Script stderr:
Usage: python -m pytest.pyhf combine [OPTIONS] [WORKSPACE_ONE] [WORKSPACE_TWO]
Try 'python -m pytest.pyhf combine -h' for help.

Error: Invalid value for '-j' / '--join': '"left outer"' is not one of 'none', 'outer', 'left outer', 'right outer'.

cc @kratsg given the mounts, but I think this is jsut the fact that / are being used here instead of \, which Windows wants.