index.qmd

---
title: Running pose estimation on the SWC HPC system
author: Adam Tyson, Niko Sirmpilatze & Igor Tatarnikov
execute: 
  enabled: true
format:
    revealjs:
        theme: [default, niu-light.scss]
        logo: img/logo_niu_light.png
        footer: "SWC | 2024-10-04"
        slide-number: c
        menu:
            numbers: true
        chalkboard: true
        scrollable: true
        preview-links: false
        view-distance: 10
        mobile-view-distance: 10
        auto-animate: true
        auto-play-media: true
        code-overflow: wrap
        highlight-style: atom-one
        mermaid: 
          theme: neutral
          fontFamily: arial
          curve: linear
    html:
        theme: [default, niu-light.scss]
        logo: img/logo_niu_light.png
        date: "2024-10-04"
        toc: true
        code-overflow: scroll
        highlight-style: atom-one
        mermaid: 
          theme: neutral
          fontFamily: arial
          curve: linear
          margin-left: 0
        embed-resources: true
        page-layout: full
my-custom-stuff:
   my-reuseable-variable: "I can use this wherever I want in the markdown, and change it in only once place :)"
---


## Contents

* Hardware overview
* Introduction to High Performance Computing
* SWC HPC system
* Using the job scheduler
* Running pose estimation on the SWC HPC

## Hardware overview {.smaller}
::: {.fragment}
* CPU (Central Processing Unit)
    * General-purpose
    * Split into cores (typically between 4 and 64)
    * Each core can run a separate process
    * Typically higher clock speed than GPU (~3-5GHz)
:::
::: {.fragment}
* GPU (Graphics Processing Unit)
    * Originally for rendering graphics
    * Thousands of cores
    * Optimised for parallel processing of matrix multiplication
    * Typically lower clock speed than CPU (~1-2GHz)
:::

## Hardware overview {.smaller}
#### Primary storage:

::: {.fragment}
* Cache
    * Small, fast memory
    * Stores frequently accessed data
    * Sits directly on the CPU/GPU
    * Typically in the MB range with multiple levels
:::
::: {.fragment}
* Main memory (RAM/VRAM)
    * Fast storage for data
    * CPU/GPU can access data quickly
    * Lost when machine is powered off
    * Typically 8-512 GB range
:::

## Hardware overview {.smaller}
#### Secondary storage:

::: {.fragment}
* Drive storage (HDD/SSD)
    * Much slower than RAM
    * SSDs faster than HDDs
    * Typically in the GB-TB range
:::
::: {.fragment}
* Network storage (e.g. ceph)
    * Shared storage accessible from multiple machines
    * Typically in the TB-PB range
    * High latency compared to local storage
:::

## Hardware overview {.smaller}
![](img/memory_hierarchy.png){fig-align="center" width="80%"}

::: aside
Source: [Dive into Systems](https://diveintosystems.org/book/C11-MemHierarchy/mem_hierarchy.html)
:::

## Hardware overview {.smaller}
![](img/bandwidth_interfaces.png){fig-align="center" width="60%"}

::: aside
Source: [High Performance Python](https://learning.oreilly.com/library/view/high-performance-python/9781492055013/)
:::


## Performance considerations {.smaller}
::: {.fragment}
* CPU
    * Frequency is important for single-threaded tasks
    * More cores can be better for parallel tasks
    * Sometimes your local machine is faster than the HPC for CPU tasks
:::
::: {.fragment}
* GPU
    * Great for parallel tasks (e.g. machine learning)
    * Memory is important - make sure your data fits in VRAM
    * Generation can be important, a new generation is typically ~10 -- 20% faster
:::
::: {.fragment}
* Storage
    * Best if you can keep data in primary memory (Cache/RAM)
    * If data doesn't fit in memory make sure it's on fast storage (local)
:::

{{< include slides/linux_commands.qmd >}}

## Introduction to High Performance Computing (HPC) {.smaller}
* Lots of meanings
* Often just a system with many machines (nodes) linked together with some/all of:
  * Lots of CPU cores per node
  * Powerful GPUs
  * Lots of memory per node
  * Fast networking to link nodes
  * Fast data storage
  * Standardised software installation

## Why?
* Run jobs too large for desktop workstations
* Run many jobs at once
* Efficiency (cheaper to have central machines running 24/7)

. . .

* In neuroscience, typically used for:
  * Analysing large data (e.g. high memory requirements)
  * Parallelising analysis/modelling (run on many machines at once)


## SWC HPC hardware
(Correct at time of writing)

* Ubuntu 20.04
* 81 nodes
  * 46 CPU nodes
  * 35 GPU nodes
* 3000 CPU cores
* 83 GPUs
* ~20TB RAM

## Logging in

Log into bastion node (not necessary within SWC network)
```bash
ssh <USERNAME>@ssh.swc.ucl.ac.uk
```


. . . 

Log into HPC gateway node
```bash
ssh <USERNAME>@hpc-gw1
```

. . .

This node is fine for light work, but no intensive analyses

::: {.callout-tip}
## More details

See our guide at [howto.neuroinformatics.dev](https://howto.neuroinformatics.dev/programming/SSH-SWC-cluster.html){preview-link="true"}
:::

## File systems  {.smaller}

* `/nfs/nhome/live/<USERNAME>` or `/nfs/ghome/live/<USERNAME>` 
  * "Home drive" (SWC/GCNU), also at `~/`
* `/nfs/winstor/<group>` - Old SWC research data storage
* `/nfs/gatsbystor` - GCNU data storage
* `/ceph/<group>` - Current research data storage
* `/ceph/scratch` - Not backed up, for short-term storage
* `/ceph/apps` - HPC applications

. . .

::: {.callout-note}
You may only be able to "see" a drive if you navigate to it
:::

## 
Navigate to the scratch space
```bash
cd /ceph/scratch
```
. . .

Create a directory for yourself 
```bash
mkdir <USERNAME>
```


## HPC software
All nodes have the same software installed

* Ubuntu 20.04 LTS
* General linux utilities

## Modules
Preinstalled packages available for use, including:


:::: {.columns}

::: {.column width="40%"}
* ANTs
* BrainGlobe
* CUDA
* DeepLabCut
* FSL
* Julia
:::

::: {.column width="60%"}
* Kilosort
* mamba
* MATLAB
* neuron
* miniconda
* SLEAP
:::

::::


## Using modules

List available modules
```bash
module avail
```
. . .

Load a module

```bash
module load SLEAP
```
. . .

Unload a module

```bash
module unload SLEAP
```
. . .

Load a specific version
```bash
module load SLEAP/2024-08-14
```
. . .

List loaded modules
```bash
module list
```


## SLURM
* Simple Linux Utility for Resource Management
* Job scheduler
* Allocates jobs to nodes
* Queues jobs if nodes are busy
* Users must explicitly request resources

## SLURM commands
View a summary of the available resources
```bash
sinfo
```

```
atyson@hpc-gw1:~$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
cpu*         up 10-00:00:0      1   mix# gpu-380-25
cpu*         up 10-00:00:0     31    mix enc1-node[1-14],enc2-node[1-13],enc3-node[6-8],gpu-380-24
cpu*         up 10-00:00:0      4  alloc enc3-node[1-2,4-5]
gpu          up 10-00:00:0      1   mix# gpu-380-15
gpu          up 10-00:00:0      1  down~ gpu-380-16
gpu          up 10-00:00:0     12    mix gpu-350-[01-05], gpu-380-[11,13-14,17-18],gpu-sr670-[20,22]
a100         up 30-00:00:0      2    mix gpu-sr670-[21,23]
lmem         up 10-00:00:0      1  idle~ gpu-380-12
medium       up   12:00:00      1   mix# gpu-380-15
medium       up   12:00:00      1  down~ gpu-380-16
medium       up   12:00:00      7    mix enc3-node[6-8],gpu-380-[11,14,17-18]
medium       up   12:00:00      4  alloc enc3-node[1-2,4-5]
fast         up    3:00:00      2  idle~ enc1-node16,gpu-erlich01
fast         up    3:00:00      4    mix gpu-380-[11,14,17-18]
```

##

View currently running jobs (from everyone)
```bash
squeue
```

```
atyson@hpc-gw1:~$ squeue
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
4036257       cpu     bash   imansd  R 13-01:10:01      1 enc1-node2
4050946       cpu      zsh apezzott  R 1-01:02:30      1 enc2-node11
3921466       cpu     bash   imansd  R 51-03:05:29      1 gpu-380-13
4037613       gpu     bash  pierreg  R 12-05:55:06      1 gpu-sr670-20
4051306       gpu ddpm-vae   jheald  R      15:49      1 gpu-350-01
4051294       gpu  jupyter    samoh  R    1:40:59      1 gpu-sr670-22
4047787       gpu     bash antonins  R 4-18:59:43      1 gpu-sr670-21
4051063_7       gpu    LRsem apezzott  R 1-00:08:32      1 gpu-350-05
4051063_8       gpu    LRsem apezzott  R 1-00:08:32      1 gpu-380-10
4051305       gpu     bash  kjensen  R      18:33      1 gpu-sr670-20
4051297       gpu     bash   slenzi  R    1:15:39      1 gpu-350-01
```

. . .

::: {.callout-tip}
## More details

See our guide at [howto.neuroinformatics.dev](https://howto.neuroinformatics.dev/programming/SLURM-arguments.html){preview-link="true"}
:::

## Partitions

## Interactive job
Start an interactive job (`bash -i`) in the fast partition (`-p fast`) in pseudoterminal mode (`--pty`) with one CPU core (`-n 1`).
```bash
srun -p fast -n 1 --pty bash -i
```
. . .

Always start a job (interactive or batch) before doing anything intensive to spare the gateway node.

## Run some "analysis"

Clone a test script
```bash
cd ~/
git clone https://github.com/neuroinformatics-unit/course-behaviour-hpc
```

. . .

Make the script executable
```bash
cd course-behaviour-hpc/demo
chmod +x multiply.sh
```

. . .

Run the script
```bash
./multiply.sh 10 5
```
. . .

Stop interactive job
```bash
exit
```


## Batch jobs

Check out batch script:
```bash
cd course-behaviour-hpc/demo
cat batch_example.sh
```

```bash
#!/bin/bash

#SBATCH -p fast # partition (queue)
#SBATCH -N 1   # number of nodes
#SBATCH --mem 1G # memory pool for all cores
#SBATCH -n 1 # number of cores
#SBATCH -t 0-0:1 # time (D-HH:MM)
#SBATCH -o slurm_output.out
#SBATCH -e slurm_error.err

for i in {1..5}
do
  ./multiply.sh $i 10
done
```

## 

Run batch job:
```bash
sbatch batch_example.sh
```


## Array jobs


Check out array script:
```bash
cat array_example.sh
```

```bash
#!/bin/bash

#SBATCH -p fast # partition (queue)
#SBATCH -N 1   # number of nodes
#SBATCH --mem 1G # memory pool for all cores
#SBATCH -n 1 # number of cores
#SBATCH -t 0-0:1 # time (D-HH:MM)
#SBATCH -o slurm_array_%A-%a.out
#SBATCH -e slurm_array_%A-%a.err
#SBATCH --array=0-9%4

# Array job runs 10 separate jobs, but not more than four at a time.
# This is flexible and the array ID ($SLURM_ARRAY_TASK_ID) can be used in any way.

echo "Multiplying $SLURM_ARRAY_TASK_ID by 10"
./multiply.sh $SLURM_ARRAY_TASK_ID 10 
```

## 

Run array job:
```bash
sbatch array_example.sh
```


## Using GPUs

Start an interactive job with one GPU:
```bash
srun -p gpu --gres=gpu:1 --pty bash -i
```
. . .

Load TensorFlow & CUDA
```bash
module load tensorflow
module load cuda/11.8
```
. . .

Check GPU 
```bash
python
```

```python
import tensorflow as tf
tf.config.list_physical_devices('GPU')
```

## Useful commands
Cancel a job
```bash
scancel <JOBID>
```
. . .

Cancel all your jobs
```bash
scancel -u <USERNAME>
```

# Example: pose estimation with SLEAP {background-color="#03A062"}

## Pose estimation {.smaller}

![](img/pose_estimation_2D.png){fig-align="center"}

::: {.fragment}
- "easy" in humans - vast amounts of data
- "harder" in animals - less data, more variability
:::

::: aside
Source: [Quantifying behavior to understand the brain](https://www.nature.com/articles/s41593-020-00734-z)
:::

## Pose estimation software {.smaller}

:::: {.columns}

:::{.column width="50%"}
[DeepLabCut](http://www.mackenziemathislab.org/deeplabcut): *transfer learning*
:::

::: {.column width="50%"}
[SLEAP](https://sleap.ai/):*smaller networks*
:::
::::

![source: [sleap.ai](https://sleap.ai/)](img/sleap_movie.gif){fig-align="center" height="400px" style="text-align: center"}

::: aside
Many others: 
[LightningPose](https://github.com/danbider/lightning-pose),
[DeepPoseKit](https://github.com/jgraving/DeepPoseKit),
[Anipose](https://anipose.readthedocs.io/en/latest/),
...
:::

## Top-down pose estimation

![](img/pose_estimation_topdown.png)

## SLEAP workflow

![](img/diagrams/pose-estimation.svg){fig-align=center width=600}

::: {.fragment}
- Training and inference are GPU-intensive
- We can delegate to the HPC cluster's GPU nodes
:::

## Sample data
`/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data/course-hpc-2023`

- Mouse videos from [Loukia Katsouri](https://www.sainsburywellcome.org/web/people/loukia-katsouri)
- SLEAP project with:
  - labeled frames
  - trained models
  - prediction results

## Labeling data locally {.smaller}
![](img/sleap-labeling.png){fig-align=center height=500px}

## Exporting a training job package {.smaller}
![](img/sleap-training.png){fig-align=center height=500px}

::: aside
see also [SLEAP's guide for remote training](https://sleap.ai/guides/remote.html)
:::

## Training job package contents {.smaller}

Copy the unzipped training package to your scratch space and inspect its contents:

```{.bash code-line-numbers="1|2-3"}
cp -r /ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data/course-hpc-2023/labels.v001.slp.training_job /ceph/scratch/$USER/
cd /ceph/scratch/$USER/labels.v001.slp.training_job
ls -1
```

::: {.fragment}
```{.bash code-line-numbers=false filename="labels.v001.slp.training_job"}
labels.v001.pkg.slp     # Copy of labeled frames
centroid.json           # Model configuration
centered_instance.json  # Model configuration
train-script.sh         # Bash script to run training
inference-script.sh     # Bash script to run inference
jobs.yaml               # Summary of all jobs
```
:::

:::: {.fragment}
::: {.callout-warning}
Make sure all scripts are executable
```{.bash  code-line-numbers="false"}
chmod +x *.sh
```
:::
::::

## What's in the SLEAP scripts?

Training
```{.bash code-line-numbers="false"}
cat train-script.sh
```

```{.bash code-line-numbers=false}
#!/bin/bash
sleap-train centroid.json labels.v001.pkg.slp
sleap-train centered_instance.json labels.v001.pkg.slp
```


::: {.fragment}
Inference
```{.bash code-line-numbers="false"}
cat inference-script.sh
```

```{.bash code-line-numbers=false}
#!/bin/bash
```
:::


## Get SLURM to run the script {.smaller}
::: {.panel-tabset}

### Interactive
Suitable for debugging (immediate feedback)

- Start an interactive job with one GPU
  ```{.bash code-line-numbers=false}
  srun -p gpu --gres=gpu:1 --pty bash -i
  ```
- Execute commands one-by-one, e.g.:
  ```{.bash code-line-numbers=false}
  module load SLEAP
  cd /ceph/scratch/$USER/labels.v001.slp.training_job
  bash train-script.sh

  # Stop the session
  exit
  ```

### Batch
Main method for submitting jobs

- Prepare a batch script, e.g. `sleap_train_slurm.sh`
- Submit the job:
  ```{.bash code-line-numbers=false}
  sbatch sleap_train_slurm.sh
  ```
- Monitor job status: 
  ```{.bash code-line-numbers=false}
  squeue --me
  ```
:::

## See example batch scripts

```{.bash code-line-numbers="false"}
cd ~/course-behaviour-hpc/pose-estimation/slurm-scripts
ls
```

:::: {.fragment}

::: {.callout-warning}
Make sure all scripts are executable
```{.bash  code-line-numbers="false"}
chmod +x *.sh
```
:::
::::

::: {.fragment}
Edit a specific script:
```{.bash  code-line-numbers="false"}
nano sleap_train_slurm.sh
```
Save with `Ctrl+O` (followed by `Enter`), exit with `Ctrl+X`
:::



## Batch script for training {.smaller}

```{.bash filename="sleap_train_slurm.sh" code-line-numbers="1-13|15-16|18-20|22-26"}
#!/bin/bash

#SBATCH -J slp_train # job name
#SBATCH -p gpu # partition (queue)
#SBATCH -N 1   # number of nodes
#SBATCH --mem 16G # memory pool for all cores
#SBATCH -n 4 # number of cores
#SBATCH -t 0-06:00 # time (D-HH:MM)
#SBATCH --gres gpu:1 # request 1 GPU (of any kind)
#SBATCH -o slurm.%x.%N.%j.out # STDOUT
#SBATCH -e slurm.%x.%N.%j.err # STDERR
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@domain.com

# Load the SLEAP module
module load SLEAP

# Define the directory of the exported training job package
SLP_JOB_NAME=labels.v001.slp.training_job
SLP_JOB_DIR=/ceph/scratch/$USER/$SLP_JOB_NAME

# Go to the job directory
cd $SLP_JOB_DIR

# Run the training script generated by SLEAP
./train-script.sh
```


## Monitoring the training job {.smaller}

```{.bash code-line-numbers="false"}
sbatch sleap_train_slurm.sh
  Submitted batch job 4232289
```

::: {.panel-tabset}

### squeue
View the status of your queued/running jobs with `squeue --me`

```{.bash code-line-numbers="false"}
squeue --me

JOBID    PARTITION    NAME       USER      ST   TIME  NODES   NODELIST(REASON)
4232289  gpu          slp_trai   sirmpila  R    0:20      1   gpu-380-18
```

### sacct
View status of running/completed jobs with `sacct`:

```{.bash code-line-numbers="false"}
sacct

JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
4232289       slp_train        gpu     swc-ac          4    RUNNING      0:0
4232289.bat+      batch                swc-ac          4    RUNNING      0:0
```

Run `sacct` with some more helpful arguments, e.g. view jobs from the last 24 hours, incl. time elapsed and peak memory usage in KB (MaxRSS):

```{.bash code-line-numbers="false"}
sacct \
  --starttime $(date -d '24 hours ago' +%Y-%m-%dT%H:%M:%S) \
  --endtime $(date +%Y-%m-%dT%H:%M:%S) \
  --format=JobID,JobName,Partition,State,Start,Elapsed,MaxRSS
```

### View the logs
View the contents of standard output and error (the job name, node name and job ID will differ in each case):

```{.bash code-line-numbers="false"}
cat slurm.slp_train.gpu-380-18.4232289.out
cat slurm.slp_train.gpu-380-18.4232289.err
```
:::

## View trained models {.smaller}
While you wait for the training job to finish, you can copy and inspect the trained models from a previous run:
```{.bash code-line-numbers="false"}
cp -R /ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data/course-hpc-2023/labels.v001.slp.training_job/models /ceph/scratch/$USER/labels.v001.slp.training_job/
cd /ceph/scratch/$USER/labels.v001.slp.training_job/models
ls
```

```{.bash code-line-numbers=false}
231130_160757.centroid
231130_160757.centered_instance
```

::: {.fragment}
What's in the model directory?

```{.bash code-line-numbers="false"}
cd 231130_160757.centroid
ls -1
```

```{.bash code-line-numbers="1,9"}
best_model.h5
initial_config.json
labels_gt.train.slp
labels_gt.val.slp
labels_pr.train.slp
labels_pr.val.slp
metrics.train.npz
metrics.val.npz
training_config.json
training_log.csv
```
:::

## Evaluate trained models
![](img/sleap-evaluation.png){fig-align="center" height="500px"}

::: aside
see also the SLEAP [model evaluation notebook](https://sleap.ai/notebooks/Model_evaluation.html){preview-link="true"}
:::


## SLEAP workflow

![](img/diagrams/pose-estimation.svg){fig-align=center width=600}


## Batch script for inference {.smaller}

```{.bash filename="sleap_infer_slurm.sh" code-line-numbers="1-16|18-22|24-28|30-36"}
#!/bin/bash

#SBATCH -J slp_infer # job name
#SBATCH -p gpu # partition
#SBATCH -N 1   # number of nodes
#SBATCH --mem 32G # memory pool for all cores
#SBATCH -n 8 # number of cores
#SBATCH -t 0-01:00 # time (D-HH:MM)
#SBATCH --gres gpu:1 # request 1 GPU
#SBATCH -o slurm.%x.%N.%j.out # write STDOUT
#SBATCH -e slurm.%x.%N.%j.err # write STDERR
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@domain.com

# Load the SLEAP module
module load SLEAP

# Define directories for exported SLEAP job package and videos
SLP_JOB_NAME=labels.v001.slp.training_job
SLP_JOB_DIR=/ceph/scratch/$USER/$SLP_JOB_NAME
VIDEO_DIR=/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data/course-hpc-2023/videos
VIDEO1_PREFIX=sub-01_ses-01_task-EPM_time-165049

# Go to the job directory
cd $SLP_JOB_DIR

# Make a directory to store the predictions
mkdir -p predictions

# Run the inference command
sleap-track $VIDEO_DIR/${VIDEO1_PREFIX}_video.mp4 \
    -m $SLP_JOB_DIR/models/231130_160757.centroid/training_config.json \
    -m $SLP_JOB_DIR/models/231130_160757.centered_instance/training_config.json \
    -o $SLP_JOB_DIR/predictions/${VIDEO1_PREFIX}_predictions.slp \
    --gpu auto \
    --no-empty-frames
```

## Run inference job {.smaller}

1. Edit and save the batch script
```{.bash code-line-numbers="false"}
nano sleap_infer_slurm.sh
```

2. Submit the job
```{.bash code-line-numbers="false"}
sbatch sleap_infer_slurm.sh
```

3. Monitor the job
```{.bash code-line-numbers="false"}
squeue --me
```

## Run inference as an array job {.smaller}

![](img/diagrams/array-jobs.svg){fig-align="center"}

## Batch script for array job {.smaller}

```{.bash filename="sleap_infer_array_slurm.sh" code-line-numbers="14|24-28|36-42"}
#!/bin/bash

#SBATCH -J slp_infer # job name
#SBATCH -p gpu # partition
#SBATCH -N 1   # number of nodes
#SBATCH --mem 32G # memory pool for all cores
#SBATCH -n 8 # number of cores
#SBATCH -t 0-01:00 # time (D-HH:MM)
#SBATCH --gres gpu:1 # request 1 GPU
#SBATCH -o slurm.%x.%N.%j.out # write STDOUT
#SBATCH -e slurm.%x.%N.%j.err # write STDERR
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@domain.com
#SBATCH --array=0-1

# Load the SLEAP module
module load SLEAP

# Define directories for exported SLEAP job package and videos
SLP_JOB_NAME=labels.v001.slp.training_job
SLP_JOB_DIR=/ceph/scratch/$USER/$SLP_JOB_NAME
VIDEO_DIR=/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data/course-hpc-2023/videos

VIDEO1_PREFIX=sub-01_ses-01_task-EPM_time-165049
VIDEO2_PREFIX=sub-02_ses-01_task-EPM_time-185651
VIDEOS_PREFIXES=($VIDEO1_PREFIX $VIDEO2_PREFIX)
CURRENT_VIDEO_PREFIX=${VIDEOS_PREFIXES[$SLURM_ARRAY_TASK_ID]}
echo "Current video prefix: $CURRENT_VIDEO_PREFIX"

# Go to the job directory
cd $SLP_JOB_DIR

# Make a directory to store the predictions
mkdir -p predictions

# Run the inference command
sleap-track $VIDEO_DIR/${CURRENT_VIDEO_PREFIX}_video.mp4 \
    -m $SLP_JOB_DIR/models/231130_160757.centroid/training_config.json \
    -m $SLP_JOB_DIR/models/231130_160757.centered_instance/training_config.json \
    -o $SLP_JOB_DIR/predictions/${CURRENT_VIDEO_PREFIX}_array_predictions.slp \
    --gpu auto \
    --no-empty-frames
```

## Further reading
* [SWC/GCNU Scientific Computing wiki](https://wiki.ucl.ac.uk/display/SSC/High+Performance+Computing)
* [SLURM documentation](https://slurm.schedmd.com/)
* [How to use the SLEAP module on the SWC HPC cluster](https://howto.neuroinformatics.dev/data_analysis/HPC-module-SLEAP.html)