Additional Preferences Configurations: Where is Main Model/Where do Outputs Go

[jmdelahanty]

Hello!

Today I got to chat with @roomrys for a little while and we discussed how we could potentially add in some flexibility for where SLEAP models can be located as well as some customizability for where predictions can end up.

In our lab, some projects record video from the exact same camera setup and are trying to get the same exact features out of the video. In this case, it's for facial expressions of the mouse under head-fixed preparations. Since these projects, at least for a while, will be doing things the same in terms of data collection, it would be really cool to just have one model the lab can communally provide labels across projects/directory structures.

To do this, we would need to have the ability to specify labels that are coming from separate server locations and also make it easy for outputs/predictions be sent to diverse places.

One project in the lab has approximately this structure for both it's raw and processed directories:

subject_id
└── YYYYMMDD
    ├── facial_expressions
    │   ├── hogs.zarr  
    │   │   ├── descriptors
    │   │   └── images
    │   └── sleap
    │       ├── labels
    │       └── predictions
    └── suite2p

Different projects will have their own directory structures/where they want labels and predictions to go. Again, however, since we'll be doing the same analysis on videos that are using a standardized views, there should ideally be one model that's used/updated. Perhaps this model could be organized like this on our servers:

snlkt
└── nn_models 
    └── sleap
        └── facial_expressions
            ├── labels
            └── models

This way there could be one model that everyone can communally update/make better and run from one location. This would also make it much easier (I think) for someone to come in and use the model/deploy the model so it's available as a service or something one day a scheduler could access/run from.

From what @roomrys was telling me, this would require the editing of the preferences.yaml to include:

• Specify input/output locations so they're not all expected in one location
• A couple additional things I don't remember off the top of my head...

And also (I think...) edit the form that builds the pipeline GUI.

Curious about your thoughts and a way forward!

[talmo]

Hi @jmdelahanty,

Not sure if @roomrys followed up with you about this offline, but SLEAP lets you store your models anywhere you want. By default it looks for trained models in the models/ subfolder in the same directory as your labels, but this is not a requirement. If you click the dropdown menu where you select models to use for inference, there is an option that says "Browse for model..." that lets you select a folder in an arbitrary location.

Let me know if that makes sense or if you meant something else.

Talmo

[jmdelahanty]

Cool! I think this is along the lines of what we were talking about! I guess is it allowed to have labels used in multiple directories?

So for example if one project had labels stored in /path/to/project1/labels and another had /path/to/project2/labels can both of these be used to train one model?

[talmo]

You need to merge your labels into a single .slp file before training, and they need to have matching skeletons, but otherwise yes. You can merge from the File menu in the GUI.

[jmdelahanty]

Nice! Since it's in the GUI I'm guessing that there's a way to also just script that as well? Could be cool to basically have a script that goes:

1. Merge labels from projects (since they might have corrections from inferences or something)
2. Train new model
3. Create model image that can be deployed somewhere (if containerization is necessary, which for a lot of this stuff it seems like it might be on cluster systems...)