Unable to View Metrics when Training with Single Instance

[mrnels19]

I have trained a SLEAP model with a single instance using Google Colab. However, training with a single instance does not give me the option to view metrics, such as the error distance, localization error, and object keypoint similarity.

Since the metrics of error distance, localization error, and object keypoint similarity are used to assess the accuracy and validity of models with multiple instances, then I think it would be helpful to include these metrics for models with a single instance. This is the training command for a single instance SLEAP model on Google Colab: !sleap-train single_instance.json filename.pkg.slp.

I thought that I would be able to generated the metrics for a single instance SLEAP model, just as I can for a SLEAP model with multiple instances.

I am unable to load the metrics, as the single instance SLEAP model does not contain the files metrics.train.npz and metrics.val.npz.

I am using SLEAP version 1.3.1 in Google Colab Pro +.

I have included the list of commands I have run in my Google Colab script, which has been adapted from the Google Colab examples on the SLEAP website. I experience an error when I load the last command in this list, which involves a metrics file that does not appear to be generated. I would like SLEAP to generate metrics files for single instance SLEAP models so that I can assess them. Or, can you show me a way to generate figures or other data that can showcase the accuracy and validity of a single instance SLEAP model?

!pip uninstall -y opencv-python opencv-contrib-python
!pip install sleap==1.3.1

from google.colab import drive
drive.mount('/content/drive/')

i```
mport os
os.chdir('/content/drive/My Drive/1.28.24_three_chamber')

from google.colab import drive
drive.mount('/content/drive')

%load_ext tensorboard
%tensorboard --logdir "/content/drive/My Drive/1.28.24_three_chamber" --reload_multifile=true --reload_interval=1 --max_reload_threads=2

`!sleap-train single_instance.json 3.18.23.3.pkg.slp
`
`!ls models
`
`!apt install tree
`
`!tree models/240128_195816.single_instance
`

import sleap
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

mpl.style.use("seaborn-deep")
sleap.versions()

metrics = sleap.load_metrics("models/240128_195816.single_instance", split="val")
print("\n".join(metrics.keys()))

Here is the error I receive after running the last command, which begins with metrics. 
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-11-63d4081fb2d8>](https://localhost:8080/#) in <cell line: 1>()
----> 1 metrics = sleap.load_metrics("models/240128_195816.single_instance", split="val")
      2 print("\n".join(metrics.keys()))

1 frames
[/usr/local/lib/python3.10/dist-packages/numpy/lib/npyio.py](https://localhost:8080/#) in load(file, mmap_mode, allow_pickle, fix_imports, encoding, max_header_size)
    403         if hasattr(file, 'read'):
    404             fid = file
--> 405             own_fid = False
    406         else:
    407             fid = stack.enter_context(open(os_fspath(file), "rb"))

FileNotFoundError: [Errno 2] No such file or directory: 'models/240128_195816.single_instance/metrics.val.npz'

[tabrownresearch]

I'm having the same issues - training on colab (sleap version 1.3.3) isn't generating metric files for my top-down models either. When I was running training on my own machine (sleap version 1.3.2) they were generated. One difference is that I manually stopped training on colab, while it ran until plateauing on my own machine.

[talmo]

Hey folks,

Sorry for the delay here -- this got passed around for a while before we got a chance to just resolve it.

Metrics don't get generated when a model fails to produce accurate results or when you interrupt it since they're only saved upon successful training completion.

Not sure that there's anything to be done here, but if anyone is still experiencing this systematically, let us know and we'll work with you to reproduce it!

Talmo

[JULIANEMI]

Hello, my friend, did you manage to solve the problem you had? I’m facing something similar.