Conda Python Path interaction

[jadenecke]

I am using conda venvs and neurocommand containers via lmod and everythings works as fine, with one unexpected interaction:
When I activate a conda environment and a FSL container the FSL module overrules the python binary from the conda environment (output of which python3 is .../neurocommand/local/containers/fsl_6.0.7.4_20231005/python3).
Basically the module Paths comes first in the PATH variable (i.e. .../neurocommand/local/containers/fsl_6.0.7.4_20231005), while the updated conda path is somewhere after.
I know that this is not a issue with neurocommand, however I dont know how to resolve the issue except loading and unloading the module a lot. I would be very happy if someone might have a solution or could point me in a specific direction.

[stebo85]

Dear @jadenecke,

Yes, that makes sense! We use prepend-path in lmod to add new tools to the path. The idea is that every tool loaded through module load will overwrite previous binaries on the path if they are conflicting. Otherwise a module load wouldn't have an effect if for example fsl is already installed locally and one wants to try a newer version from Neurodesk.

One solution that could work for you is to add the conda environment to the front of the path after you loaded the fsl module:

(base) jovyan@neurodesktop-:~$ echo $PATH
/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jovyan/.local/bin:/opt/conda/bin:/opt/conda/condabin
(base) jovyan@neurodesktop-:~$ ml fsl
(base) jovyan@neurodesktop-:~$ echo $PATH
/cvmfs/neurodesk.ardc.edu.au/containers/fsl_6.0.7.8_20240913:/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jovyan/.local/bin:/opt/conda/bin:/opt/conda/condabin
(base) jovyan@neurodesktop-:~$ which python
/cvmfs/neurodesk.ardc.edu.au/containers/fsl_6.0.7.8_20240913/python

#That's the replication of your problem

# That's a potential solution:
(base) jovyan@neurodesktop-:~$ export PATH=/opt/conda/bin:/opt/conda/condabin:$PATH
(base) jovyan@neurodesktop-:~$ echo $PATH
/opt/conda/bin:/opt/conda/condabin:/cvmfs/neurodesk.ardc.edu.au/containers/fsl_6.0.7.8_20240913:/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jovyan/.local/bin:/opt/conda/bin:/opt/conda/condabin
(base) jovyan@neurodesktop-:~$ which python
/opt/conda/bin/python
(base) jovyan@neurodesktop-:~$ which fslmaths
/cvmfs/neurodesk.ardc.edu.au/containers/fsl_6.0.7.8_20240913/fslmaths

Would that work for you?

[jadenecke]

yes, prepend-path makes sense. I think that solution would work for me, generally I think there are a few ways how to deal with it, however I can't think of a solution that does not require an adjustment in every script, i.e. a way to modify .bashrc or something similar.

While writing, I found a possibly more convenient solution:
One can add a custom script to the conda env that ensures that when loading an environment, the old conda path is removed from $PATH and prepended again. You have to do it once per environment though. Then you just have to remember to load modules first and then the conda env. Lets see what kind of side effects this will have.

Courtesy of chatGPT:

mkdir -p $CONDA_PREFIX/etc/conda/activate.d/
nano $CONDA_PREFIX/etc/conda/activate.d/custom_path.sh

#!/bin/bash
# Remove any existing conda paths from PATH
export PATH=$(echo $PATH | tr ':' '\n' | grep -v 'conda' | tr '\n' ':' | sed 's/:$//')
# Prepend the new conda environment path
export PATH="$CONDA_PREFIX/bin:$PATH"

chmod +x $CONDA_PREFIX/etc/conda/activate.d/custom_path.sh

[stebo85]

Dear @jadenecke - thank you for posting that solution! We will also keep thinking how we could do this better in the future. I currently think that the most elegant, reproducible and sustainable solution would be to also load the conda environment via lmod (and even offer different versions of conda environments and python versions ...). Then the active python environment would be fully controlled by the module load order: If one loads FSL first, and a conda module next, then the python will come from conda. If one loads conda first and then FSL, the python from FSL would be used (although it's questionable why one would do that). We exposed the python from FSL because some users wanted to run notebooks inside the FSL python environment.