Running tests on remote machine? (Python)

[HiPhish]

Hi there,

at work I need to run tests on a remote machine because certain resources are only available on that machine. I write my code locally, push my branch to the machine, then SSH into it and run pytest manually from the shell. This works fine, but I want to be able to run and debug tests straight from within Neovim. I already know how to run and debug tests locally, and debugpy can debug programs over SSH through port-forwarding.

Is there a way to hook up Neotest to SSH? Maybe it would be possible to write an adapter which acts as a wrapper around the existing Python adapter? That way we could reuse most of the code we already have.

[HiPhish]

I have looked more into the way the Python adapter works. We can keep most of it, but the difficult part will be executing the test. To run the test we need to execute Python on the remote machine, but we need to run the script bundled with the adapter plugin. So we need to somehow transfer the script over to the remote machine, but I can only move files from the remote to the local machine, not the other way around.

The only solution I can think of is to feed the script over standard input to SSH: ssh user@remote 'python -' < neotest.py. Problem is, the script imports modules, but I can only transfer one file. So we would have to merge all the Python files into one script. Ugly and awkward, but possible.

Or maybe we could push the Python script and all its contents to the remote via SSH to a temporary directory. First I would take the hashes of all files, and send a script over to the remote. That script would look for a directory under /var/tmp with that hash, and if it does not exist create and the report back success, otherwise failure. On success I would copy over the scripts, on failure I skip this step. Now I can run the test straight from the remote. The hash is used to avoid sending the Python scripts on every test run, we can reuse them as long as they have not changed.

The other issue is with the two temporary files. They need to be created on the remote, then I can copy them over to my local machine via scp to temporary files here. The Python script could take care of creating temporary files and and print their paths to standard output.

As for running the SSH command, this can be done either by shelling out from Neovim, or by using the Fabric library. This would introduce additional Python dependencies though.

[HiPhish]

I have been studying the Python code of neotest-python and I think I have a solid plan now. This will require only the ssh, scp, mktemp and tail commands. We need to achieve the following goals:

• Transfer the neotest-python.py script and all its other scripts to the remote
• Create two temporary files (for streaming and results) on the remote, report their names back to the local machine
• Create two temporary files on the local machine
• Mirror the temporary files from the remote on the local machine while the test is running

Transferring the Python scripts

We can take the hash of multiple files like this:

sha256sum **/*.py | cut -f 1 '-d ' | sha256sum - | cut -f 1 '-d '

This will first compute the hashes of all Python files in the current directory and list them, strip away the file paths, then feed that result into a second hashing command to compute the hash of the result, and finally strip away the - from the output. Note that the output of the second sha256 depends on the contents of the input lines, which might change depending on the working directory.

Once we have the hash we can create a directory on the remote machine and copy over the files. If the hash directory already exists we do not need to copy files.

if ssh user@host "test -d /var/tmp/neotest-pyton-${hash_value}"; then
   ssh user@host "mkdir /var/tmp/neotest-pyton-${hash_value}"
   scp */**.py user@host:/var/tmp/neotest-python-${hash_value}
fi

Creating the temporary files

This is simple, we call mktemp on the remote. Note that we must capture the output locally in Neovim for later.

remote_results=$(ssh user@host mktemp)
remote_stream=$(ssh user@host mktemp)

local_results=$(mktemp)
local_stream=$(mktemp)

Mirroring remote files locally

Neotest needs to read the the stream file as it is written, so we need to mirror it in real time (or at least close enough to real time). This is the most hacky part of it all.

ssh user@host "tail -f ${remote_stream}" > ${local_stream}
ssh user@host "tail -f ${remote_results}" > ${local_results}

These two SSH processes must run in parallel, so they have to be started as separate jobs from within Neovim. The -f option to tail is what prints the contents one line at a time to the standard output.

The only thing I don't know is how to terminate the tail process. I guess I could call jobstop from within Neovim to manually terminate once the Python script is done running.

Running tests on the remote machine

This is a matter of executing the regular command but from inside SSH.

ssh user@host "python /var/temp/neotest-python-${hash_value}/neotest.py --stream ${remote_stream} --results-file ${remote_results} --runner ${runner} -- ..."

I have not actually tested any of this, that comes next. It is clear to me that I cannot write a generic SSH adapter wrapper, that's why I am focusing on Python. I am writing down my thoughts in the hope that other people find them useful in the future.