On the direction of the unix step

[markusdregi]

After getting some experience with the unix step while setting up the two examples currently in place I have some reflections I would like to share.

Embrace unix
The unix step feels the best, and gets the least in my way when it behaves as a shell script in a unix environment. Since we are now in a position of being able to provide multiple step types I think the unix step should be as true to the unix envision as possible.

The commands
It acts naturally as long as I'm using Python scripts as commands, but whenever I want to run some other type of script it becomes awkward. In the SPE1 example I ended up introducing a Python script named bash to be able to run bash commands by adding bash as a prefix. I think this should be the default way of running the commands. So that I could write:

  - echo "I wan't unix" > my_statement
  - cp my_statement your_statement

and it will run without any hassle.

Transportable commands
Currently we offer to transport scripts to the compute side to run them. They are then dumped directly within the runpath folder. I think we should treat these in a more standard unix manner. My suggestion would be to create a natural bin location within the runpath for these scripts that we add to PATH.

Script environment
I think we should get a clear grip on how the compute environment of the unix step is formed. Maybe it is ok to assume that there is a read-only file system where the core environment is defined? Or do we want to be explicit about the entirety of the environment? But are we then closing in to making a container step? And at some point we want to run steps within a Komodo environment. Is that the unix step, a virtualenv like step or a komodoenv step?

[pinkwah]

Embrace unix

<pedantry alert>Linux is not Unix</pedantry>

The distinction is not important, but I'm commenting on every headline so why not be pedantic. Linux is not Unix certified, so... We're technically talking about POSIX steps.

(Imagine we called it a bash step, but then used csh as our shell. Not an important distinction, but technically incorrect)

The commands
So after working with some CI/CD pipelines, my preference is that of Github Actions (and Jenkins kind of), rather than Travis. GA's unix step equivalents allow only a single run: entry in their configurations. Travis, on the other hand, has a list of commands that are executed step-for-step. I think the GA way is neater.

  run: python -c "print('hello')"

or, for multiline (YAML feature, not something we implement)

  run: |
    echo -e "#!/usr/bin/env python3\nprint('hello')" > my_script
    chmod +x my_script
    ./my_script

All such pipeline tools that I know of run the commands in shell mode. As if /usr/bin/bash -c "<user code>". There's no advantage of running in non-shell mode (ie, directly calling the executable). Don't reinvent the shell.

Take inspiration from Github Actions. They also call this a "step": https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idsteps

Transportable commands
Sounds like a cool idea! I always stand by using standard POSIX features.

Script environment
I would target a future where we use docker or some other containerisation software for our environment. We build komodo for docker, then, in the step description, the user specifies the container image (eg. komodo-py36) and the script to execute, as well as some other variables that they might like.

While we can't use docker today, I think we should have separation of environments at a very high priority.

We could also add a environments.yaml, where the user can modify their environments. Eg:

- name: komodo+pytest
  image: komodo-py36:stable
  addons:
    - pytest
  env:
    AWESOME: "yes"

Then they can use it in their code:

  image: komodo+pytest
  run: |
    which pytest
    echo $AWESOME

(The fact that komodoenv is used is an implementation detail)

[markusdregi]

On the name
Good point @dotfloat! I'm not sure whether posix_step will communicate much better, nor whether we'll be able to live up to it. But perhaps shell_step or script_step could be an alternative?

The script
While I agree with you that a single string as the script is very convenient during CI, part of the context here is that due to the long running stages (expect a migrating user to only have a single unix_step to begin with!) we want to be able to monitor the progress within the step. This is already supported, but is depending on a separation, i.e. the list...

Environment
I agree that we don't want to rule out docker, and maybe it should all be encapsulated within an environment or image definition. The stage actually had one in the very beginning, but we removed it again as we had no options for it in the start. If so, I think it should be rich enough that we can naturally offer the standard RHEL environment or a komodoenv as the baseline for the script...

[berland]

shell_step tastes good.

[markusdregi]

@berland brought up the topic of passing parameter values to a script via the command line. I'm tempted to avoid rendering the scripts if we can, as it potentially blows up the complexity of configuration, parsing configuration and the interaction between the client and compute quite a lot.

But, for the unix step it would be natural to be able to pass environment variables that are to be set for the execution. Perhaps something along the lines of:

env:
    RELPERM_INTERPOLATION=reservoir.relperm_interpolation_parameter
    
...
script:
   - ...
   - pyscal ... --int_param_wo $RELPERM_INTERPOLATION

could be a good solution?

[berland]

That could work

[jondequinor]

Why populate the environment with something that's just discarded?

script:
  - pyscal … --int_param_wo ${reservoir.relperm_interpolation_parameter}

[markusdregi]

What do you mean by discarded @jondequinor?

[jondequinor]

Another way to say it, why contaminate the process of pyscal with the environment variable RELPERM_INTERPOLATION when it (as I read this) is only used as an argument in the script directive?

[markusdregi]

Sorry for the really late response @jondequinor! My thinking was that input to a unix_step (or whatever name it ends up with) should be standard POSIX kind of behaviour. And then I was thinking that input data should be communicated either as files (as the records) or via environment variables.

We can of course also introduce a templating language on top of the script, but this adds some complexity as it then needs to be rendered for each of the realisations individually and it brings it further away from what would be standard usage of the shell.

[markusdregi]

shell_step vs unix_step vs posix_step vs something else 🤔