Inaccurate prediction for bottom-up view mice

[skyshi0]

Hi,
I'm writing to get some suggestions on how to improve the prediction performance on tracking a mouse monitored by a bottom-up view camera. I used a multi-animal top-down model to train the Unet model with parameters suggested in the previous discussion #1969:

1. Centriod model(anchor to tail base): input scale: 0.5; rotation: -180/180; max stride: 16;
2. Centered instance model: input scale: 1; rotation: -180/180, max stride: 32;

I've labeled 840 frames, but the prediction is still not ideal. Especially, it had a bad prediction for the nose tip(lost or wrong prediction). I have tried other models like resnet and pretrained-encoder, but it returned the "no instance" results again, though the receptive field is the same for Unet and LEAP. Do you have any suggestions for improving the prediction performance? Thank you so much!

[github-actions[bot]]

Thank you for starting a new discussion!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[roomrys]

Hi @skyshi0,

Apologies for the delay - it seems we dropped off in #1969 with

I'm now refining the model to predict the nodes better(currently can't capture the nose tip well, the predicted nose tip is around the mouth instead even though I corrected them several trainings) I also have a follow-up question about the RF, my other videos may have a slight offset of the mouse size, should I set the RF to include the largest one for all videos? Thank you for your help again!

1. Since you were getting better results with top-down - let's refine those parameters.

2. I would not worry too much about the RF being exactly the size of the animal for the top-down view. This matters more for when we do node grouping into animals as a second step - as in bottom-up. With top-down the nodes are grouped into animals through the centered crop (which is the output of the first model in our top-down approach).

3. However, I would make sure to set the crop-size to auto in the centered-instance model which automatically uses you largest labeled annotation to determine how big we should crop about a predicted centroid. If the crop size is too tiny, this might explain missing nodes on the extremities (such as the nose).

Let us know if that helps any.

Thanks,
Liezl

[skyshi0]

Thank you! I tried to improve the models by different ways, and I encountered some errors now:
(1)I opened a new project and used the trained model to predict some random frames in the new videos following https://github.com/talmolab/sleap/discussions/1598 GUI, I corrected the wrong ones, and I wanted to use the trained model + new videos to continue the training. However, it had errors "labels.skeleton can only be used when there is only a single skeleton saved in the labels. Use labels.skeletons instead" when I tried to merge it into the previous model.
(2)I reopened the previous model and added the new videos to the project, predicted and corrected 20 random frames in the new videos. I'm sure that I clicked the 'Auto' crop, however, I have another error saying 'input 0 of layer "model" is incompatible with the layer: expected shape=(None, 160, 160,1), found shape=(1, 120, 120, 1)' similar to https://github.com/talmolab/sleap/issues/1993.

Both the old and new models are not working now, I'd appreciate it if you could give me some guidance to restore my previous model. Thank you!