using a pre-existing model

[bjudkewitz]

We have a tensorflow/keras model, already trained outside SLEAP, which we would like to use as a centroid detector in SLEAP. It's a Unet type architecture that takes grayscale frames as inputs and outputs score maps of the same shape (scores being float32 values between 0 and 1). How can I plug it into the SLEAP pipeline?

[github-actions[bot]]

Thank you for starting a new discussion!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[gitttt-1234]

Hi @bjudkewitz,

Are you trying to set up an inference pipeline with your own model for centroid detection and a SLEAP model for keypoints detection?

Thanks!

Divya

[bjudkewitz]

Hi Divya! Something like that, though not quite a pipeline. We just have a very reliable and sufficiently flexible centroid detector that was trained with much effort, so I'd like to re-use it for further work (centroid inference on new videos and also keypoint detection) in SLEAP.

[gitttt-1234]

Hi @bjudkewitz,

Great, there's a workaround. We can try passing the keras model to the CentroidCrop inference layer.

The main TopDownPredictor has an inference model, TopDownInferenceModel, which has two inference layers: CentroidCrop (to get the centroids and crop the images based on the detected centroids) and FindInstancePeaks (Get the predicted keypoints for each cropped image). Now, both these inference layers take in a keras model and a few other parameters, which are initialized here.

If you pass in your keras model (and other required parameters) to the CentroidCrop layer here, then the predictor will use your custom keras model for centroids and the SLEAP model for centered-instance.

Here's a very rough implementation that might work for your use-case (There are only two things to be added from your side: CENTROID_KERAS_MODEL and IS_CENTROID, which are given in the top of the file). To run inference, python inference_custom_centroid_model.py <cli-args>. The cli_args are the same as those used in sleap-track.

Let me know if this works! (We can provide a more cleaner code based on your requirements).

Thanks!

Divya

[bjudkewitz]

Hi Divya, thanks so much!! This helped me a lot in understanding what happens under the hood, and this approach would likely work for my own use case. However, I am thinking about others in the lab who prefer the GUI. It would be really nice if there was a way to convert any keras centroid or inference detector into a format that the GUI will accept in the model selection dropdown (or have the model dropdown accept a standard tensorflow/keras .h5 or .keras model file). I understand this would likely be without an option to change the architecture, but still very useful.
Best wishes,
Benjamin

[talmo]

Hi @bjudkewitz,

Right, the easiest would probably be to configure and train a centroid model as similar as possible to yours using the SLEAP GUI and then swap out the best_model.h5 in the resulting with your own checkpoint.

There's likely a good number of places where this will need to be adjusted since we make assumptions about layer names, shapes and ordering based on what we parse from the config, but it's a good starting point.

Give it a go and let us know if you run into any errors!

(Based on your experience, we may want to promote this to an enhancement proposal -- and we'd super appreciate contributions!)

Cheers,

Talmo

[bjudkewitz]

Hi Talmo, sorry about the late reply. Since we needed a quick solution, for now, the pragmatic solution was to use a default SLEAP model and retrain. I'd like to revisit this at a later stage and will let you know if we come up with a good solution.

Best wishes,
Benjamin