A few questions about the predict_on_batch method

[murtazahathiyari]

Hi,

I've been trying to set up a closed loop optogenetic stimulation experiment for which I have been using the predict_on_batch method as used in the example notebook for real time inference. I had a few questions about it's usage.

1. How is the predictor.inference_model.predict_on_batch method different than the predictor.predict method? There is a noticeable difference in latency but I am curious to know what goes on under the hood. (Since both methods take a single frame)
2. Is there a way to maintain track identity between predictions? I really like the trail method that is employed in the GUI for offline tracking.
3. Is there a neat way to visualize the tracks for validation similar to the plot method for the sleap.io.dataset.Labels datatype?

Best,
Murtaza

[talmo]

Hi @murtazahathiyari,

Apologies for the delay!

1. How is the predictor.inference_model.predict_on_batch method different than the predictor.predict method? There is a noticeable difference in latency but I am curious to know what goes on under the hood. (Since both methods take a single frame)

The Predictor.predict() method calls tf.keras.Model.predict() under the hood, which in turn recompiles the inference graph on every call, which is super slow. It also packages up the results in SLEAP data structures, which adds a bit to the overhead.

In contrast, Predictor.inference_model.predict_on_batch() calls the lower level tf.keras.Model.predict_on_batch() which will only compile the graph if needed. It also returns raw tensors, optionally without even pulling them down from the GPU to the CPU and converting them to numpy arrays, which adds some latency.

Is there a way to maintain track identity between predictions? I really like the trail method that is employed in the GUI for offline tracking.

Yes, this requires applying the identity tracking. It's harder to do this in realtime without using supervised identity models.

This notebook walks through some of the tracking functionality.

3. Is there a neat way to visualize the tracks for validation similar to the plot method for the sleap.io.dataset.Labels datatype?

Not quite, other than what we have in the GUI, but once you have trajectories pulled out, it should be easy to plot.

I'd recommend checking out the movement library which has examples of plotting and some basic analysis with SLEAP data.

Let us know if you have any questions!

Talmo

[murtazahathiyari]

Hi Talmo,

Thank you for reaching out!

The Predictor.predict() method calls tf.keras.Model.predict() under the hood, which in turn recompiles the inference graph on every call, which is super slow. It also packages up the results in SLEAP data structures, which adds a bit to the overhead.

In contrast, Predictor.inference_model.predict_on_batch() calls the lower level tf.keras.Model.predict_on_batch() which will only compile the graph if needed. It also returns raw tensors, optionally without even pulling them down from the GPU to the CPU and converting them to numpy arrays, which adds some latency.

This makes sense, thank you!

Yes, this requires applying the identity tracking. It's harder to do this in realtime without using supervised identity models.
This notebook walks through some of the tracking functionality.

In the notebook example, the tracker essentially takes a list of instances and the corresponding images as arguments to track the instances across the images. For implementing real time tracking, I might be able to warm start the tracking by collecting a few predictions to track instances on the current frame. Would you recommend doing this or can there be a better/faster method? Additionally, since the sleap.nn.tracking.Tracker.track method takes a list of instances as an argument and the Predictor.inference_model.predict_on_batch() method outputs a RaggedTensor, is there a way to package the list of RaggedTensors into a list of instances?

Not quite, other than what we have in the GUI, but once you have trajectories pulled out, it should be easy to plot.
I'd recommend checking out the movement library which has examples of plotting and some basic analysis with SLEAP data.

Thank for pointing out this resource, I'll check it out!

Best,
Murtaza

[talmo]

Hi @murtazahathiyari,

In the notebook example, the tracker essentially takes a list of instances and the corresponding images as arguments to track the instances across the images. For implementing real time tracking, I might be able to warm start the tracking by collecting a few predictions to track instances on the current frame. Would you recommend doing this or can there be a better/faster method? Additionally, since the sleap.nn.tracking.Tracker.track method takes a list of instances as an argument and the Predictor.inference_model.predict_on_batch() method outputs a RaggedTensor, is there a way to package the list of RaggedTensors into a list of instances?

You could certainly do this, but the issue is that if the tracker makes an identity switch error, it'll be wrong for all subsequent frames (until it switches back again, if ever).

We would actually recommend using the supervised ID model types for these kinds of cases. These models rely on appearance-based cues and no temporal information. In this setting, you rely on having distinguishing visual appearances between animals to assign identities.

In either case, you can then use Bonsai-Rx's SLEAP integration to actually apply it in realtime.

If you do want to try using the tracker in realtime, you can check out how we make instances from the tensors here.

Cheers,

Talmo

[murtazahathiyari]

Hi Talmo,

Appreciate the prompt response! Would SLEAP have the functionality to use supervised ID models that you've described? I tried looking it up on the documentation but was unable to find anything substantial. If not, are there packages you would recommend using that leverage appearance-based cues whilst working with low latency?

Best,
Murtaza

[talmo]

Apologies for the delay @murtazahathiyari!

We don't have this in the docs for a number of reasons, but if you want to give it a go, there's some info you can use in the issue I just opened to track this.

[murtazahathiyari]

Thanks Talmo, this will be very helpful! Looking forward to trying out the supervised ID methods
I'll let you know how it goes.

Best,
Murtaza