Nextflow starts all processes on single slurm node

[alex-d13]

Hi,

I am having some troubles with setting up nextflow+slurm properly. I am using nextflow to run multiple software tools with different input files/parameters and have a slurm cluster setup with 3 nodes to use. The tools are then started in a Docker container (using docker.enabled = true). My problems are:

1. When i start nextflow, all processes are spawned and executed in parallel on a single node and the other 2 nodes are ignored. I can see that by typing sinfo while my pipeline is running and only a single one has the state mix while the other two nodes are idle:
   

2. The pipeline crashes due to an out of memory error. I also think thats because all processes are started at the same time. Is there not an option that jobs are not started at the same time, but rather are placed in a queue until other jobs have finished?
   

I can provide config files and other information if they are helpful, but maybe somebody already knowns where my problem could be.

Cheers and thanks already,
Alex

[mribeirodantas]

I may be wrong, but regarding (1) I think it's a SLURM thing. Check this. If you know how to do it without Nextflow, you can use the clusterOptions directive to make Nextflow do the same.

Regarding (2), you can set the queueSize directive to provide the number of tasks the executor will handle in a parallel manner (default: 100). Also, make sure you're requesting a reasonable amount of resources in your process directives.

Sharing your config file(s), and also the nextflow.log can be helpful 😄 (Remember to remove sensitive data from whatever data you share here)

[alex-d13]

Thanks so much :)
Got it running with these two helpful suggestions. Great way to start into 2023 :D

[mribeirodantas]

Yay! 🥳

Thanks for letting me know, @alex-d13 . Happy New Year 😄

[JLC2141]

Hello,

I've run into a similar issue posted here with our cluster. Can you provide clarification on the solution, please? Based on this proposed solution, "Check this. If you know how to do it without Nextflow, you can use the clusterOptions directive to make Nextflow do the same", do I need to perform both actions for this to work? Meaning: 1) Outside of nextflow, configure the cluster with SelectTypeParameters=CR_LLN and 2) Within nextflow, using the clusterOptions directive, I should also specify SelectTypeParameters=CR_LLN

Thank you!