ID changes in good top down model and hungarian tracks

[transkriptase]

Hi,

I use sleap since really long time and so far so good tracks I got but then I recognized that my tracks are flipping in some situation like in pic1

Even my tracks result looks like this:

I run tracking on hpc with this command:

(sleap) okilic@sv2213:~$ sleap-track "/home/okilic/cutting_legs/C-/2/C-_6_2_500 0_21.03.240.mp4" -m "/home/okilic/models/baseline.centroid_2/" -m "/home/okili c/models/baseline_medium_rf.topdown_3/" --tracking.tracker flow --tracking.sim ilarity centroid --tracking.match hungarian --tracking.track_window 20 -o "/ home/okilic/cutting_legs/tracked/6_2_F_C_H.slp"
Started inference at: 2024-06-19 15:45:08.337293
Args:
{
│ 'data_path': '/home/okilic/cutting_legs/C-/2/C-_6_2_5000_21.03.240.mp4',
│ 'models': ['/home/okilic/models/baseline.centroid_2/', '/home/okilic/models/baseline_medium_rf.topdown_3/'],
│ 'frames': '',
│ 'only_labeled_frames': False,
│ 'only_suggested_frames': False,
│ 'output': '/home/okilic/cutting_legs/tracked/6_2_F_C_H.slp',
│ 'no_empty_frames': False,
│ 'verbosity': 'rich',
│ 'video.dataset': None,
│ 'video.input_format': 'channels_last',
│ 'video.index': '',
│ 'cpu': False,
│ 'first_gpu': False,
│ 'last_gpu': False,
│ 'gpu': 'auto',
│ 'max_edge_length_ratio': 0.25,
│ 'dist_penalty_weight': 1.0,
│ 'batch_size': 4,
│ 'open_in_gui': False,
│ 'peak_threshold': 0.2,
│ 'max_instances': None,
│ 'tracking.tracker': 'flow',
│ 'tracking.max_tracking': None,
│ 'tracking.max_tracks': None,
│ 'tracking.target_instance_count': None,
│ 'tracking.pre_cull_to_target': None,
│ 'tracking.pre_cull_iou_threshold': None,
│ 'tracking.post_connect_single_breaks': None,
│ 'tracking.clean_instance_count': None,
│ 'tracking.clean_iou_threshold': None,
│ 'tracking.similarity': 'centroid',
│ 'tracking.match': 'hungarian',
│ 'tracking.robust': None,
│ 'tracking.track_window': 20,
│ 'tracking.min_new_track_points': None,
│ 'tracking.min_match_points': None,
│ 'tracking.img_scale': None,
│ 'tracking.of_window_size': None,
│ 'tracking.of_max_levels': None,
│ 'tracking.save_shifted_instances': None,
│ 'tracking.kf_node_indices': None,
│ 'tracking.kf_init_frame_count': None
}

2024-06-19 15:45:08.405489: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.407732: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.411661: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.413686: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.424260: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.426924: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.429263: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.431409: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.434089: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.436855: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.439228: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.441132: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
INFO:sleap.nn.inference:Auto-selected GPU 0 with 30655 MiB of free memory.
Versions:
SLEAP: 1.3.3
TensorFlow: 2.7.0
Numpy: 1.19.5
Python: 3.7.12
OS: Linux-5.10.0-30-amd64-x86_64-with-debian-11.9

System:
GPUs: 1/4 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True
Device: /physical_device:GPU:1
Available: False
Initalized: False
Memory growth: None
Device: /physical_device:GPU:2
Available: False
Initalized: False
Memory growth: None
Device: /physical_device:GPU:3
Available: False
Initalized: False
Memory growth: None

Video: /home/okilic/cutting_legs/C-/2/C-_6_2_5000_21.03.240.mp4
2024-06-19 15:45:08.631733: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-19 15:45:08.640805: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.643670: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:08.646101: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:10.419222: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:10.421554: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:10.424518: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-06-19 15:45:10.427450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 28541 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:04:00.0, compute capability: 8.0
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% ETA: -:--:-- ?

and the number of flips usually around 10-15 or more in 5 min video. Somehow I can not use --tracking_similarity instance instead of centroid because it takes so lond and hpc never finish tracking. I get this error on terminal:

at 7fae9bee6000 of size 256 next 4589
2024-06-19 15:35:13.714450: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fae9bee6100 of size 256 next 4601
2024-06-19 15:35:13.714456: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fae9bee6200 of size 256 next 4592
2024-06-19 15:35:13.714461: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fae9bee6300 of size 256 next 4591
2024-06-19 15:35:13.714466: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fae9bee6400 of size 256 next 4594
2024-06-19 15:35:13.714479: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fae9bee6500 of size 256 next 4579
2024-06-19 15:35:13.714483: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fae9bee6600 of size 256 next 4617
2024-06-19 15:35:13.714492: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fae9bee6700 of size 256 next 4616
2024-06-19 15:35:13.714499: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fae9bee6800 of size 256 next 4618

could you please help me to at least minimize id flips? I hope there is a clear solutiion for this.

Thank you

[talmo]

Hi @transkriptase,

Apologies for the delay! We're a bit behind on support responses at the moment.

I think your issue is related to the window size: --tracking.track_window 20

While this allows SLEAP to consider linking candidate poses going back up to 20 frames, the further back in time you go, the more likely it is to make false associations. Since you're using the flow tracker, this is partially compensated for by attempting to update the location of previous poses, but with that large of a gap, it's possible that it's updating a previous animal to the location of another one.

I'd recommend trying out a lower track window and consider using our newer track-local queues. This enables bridging over long gaps if you know the exact number of animals that are in your video. You can try it out with these settings:

sleap-track "/home/okilic/cutting_legs/C-/2/C-_6_2_500 0_21.03.240.mp4" -m "/home/okilic/models/baseline.centroid_2/" -m "/home/okili c/models/baseline_medium_rf.topdown_3/" --tracking.tracker simplemaxtracks --tracking.max_tracking 1 --tracking.max_tracks 4 --tracking.track_window 5 -o "/ home/okilic/cutting_legs/tracked/6_2_F_C_H.slp"

Just change --tracking.max_tracks 4 to whatever number of animals you have.

Let us know if that helps!

Cheers,

Talmo

[transkriptase]

Hello Talmo,

thank you so much for your answer. I run another tracking with the command you suggested;

sleap-track "/home/okilic/cutting_legs/C-/2/C-_10_2_5000_21.03.240.mp4" -m "/home/okilic/models/baseline.centroid_2/" -m "/home/okilic/models/baseline_medium_rf.topdown_3/" --tracking.tracker simplemaxtracks --tracking.max_tracking 1 --tracking.max_tracks 54 --tracking.track_window 5 -o "/home/okilic/cutting_legs/tracked/10_2_F_C_H_t_54.slp"

This reduced the number of ID changes, there are still many and even the distance between the individuals is not short, the ID changes exist. I added one example, maybe it helps! For example in photo 1 at frame 1577 track 47 (blue) and track 7 (yellow) switch identities in next frame (1578).

In frame 1578 you can clearly see that track ID replaced even the distance between them.

I hope you have more suggestions because ID changes is extremely big problem for me right now.

Best,
Özge

[talmo]

Hi @transkriptase,

Thanks for the screenshots.

This indeed shouldn't be happening and I'd like to troubleshoot it more carefully.

Would you mind sharing the SLP file and video with us so we can do some troubleshooting on our end?

Cheers,

Talmo

[transkriptase]

Hi Talmo,

I uploaded the file in .zip file to the link you sent. Please write me if you need anything else.

Best wishes,
Özge

[gitttt-1234]

Hi @transkriptase,

Thank you for sharing the data.!

Apologies for the delay. We are looking into the data and trying a few alternate methods. We found that normalizing the keypoints could potentially solve this issue.
We're currently working on having a new normalized_instance_similarity method in this PR.

Thanks,
Divya

[transkriptase]

Hi Divya, Thank you so much for your answer and I hope we find the best solution. I will be waiting for your last feedback. Best wishes, Özge Am Do., 5. Sept. 2024 um 21:05 Uhr schrieb DivyaSesh < ***@***.***>:

…

Hi @transkriptase <https://github.com/transkriptase>, Thank you for sharing the data.! Apologies for the delay. We looked into the data and trying a few alternate methods. We found that using [object_keypoint_similarity]( https://github.com/talmolab/sleap/blob/35463a1ddf7649ab813d36f46680dad5eaf3edfc/sleap/nn/tracker/components.py#L77 ) as the scoring function with hungarian matching could potentially solve the ID switching issue (the current instance_similarity method doesn't apply any normalization, resulting in low - near zero, similarity scores). We're currently testing this method and working on having object_keypoint_similarity method as the default scoring method in this PR <#1939>. Thanks, Divya — Reply to this email directly, view it on GitHub <#1815 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BBGKX672NRNMPSW6MV5MAT3ZVCTPHAVCNFSM6AAAAABJSDMYNWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANJWGEZTQMY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[talmo]

Hi @transkriptase,

Just a quick update that we're working on it and believe we have a good solution in #1939, but we're working through some kinks.

As soon as we have a workable version that you can use from source, @gitttt-1234 will reply here with instructions on how to install from source with the changes optimized for your data.

Thanks for bearing with us!

Talmo

[gitttt-1234]

Hi @transkriptase,

The PR #1939 is all set! Using --tracking.similarity normalized_instance could solve the issue. You can use this new feature by installing sleap from source:

1. git clone https://github.com/talmolab/sleap && cd sleap
2. conda env create -f environment.yml -n sleap_dev
   Ref: conda from source

Let us know if you have any questions!

Thanks!

Divya

[transkriptase]

Hallo Divya,

sadly I am getting same ID switched in same frames. I am not sure if there is less then previous one but they are still exist, even I installed and checked new sleap_dev version of the software.

WhatsApp.Video.2024-10-14.um.13.11.20_f509063a.mp4

And in this version, i also get so much memory error as well. I have no idea why!!!

I try to made changes on memory limitation (made enable) like this;


But still my tracks is never ending and not predict on anything .

The command I run this tracks is as @talmo suggested:

Could you please let me know where I do mistakes. I would be really happy if we can meet online if you have avaliable time. I can also invite my supervisor from Freie University Berlin so we can cooperate on my PhD project, which is all about tracking termites under different conditions and analyse their social behaviour and build an aoutomated system to get continuously videos to track.

Hopefully we can find a better solution.

Best,
Özge

[gitttt-1234]

Hi @transkriptase,

For ID switching, you need to add --tracking.similarity normalized_instance to your command. We added the new normalized instance method but it's not chosen by default. So, we need to explicitly mention in the command. Let me know if this works!
For the memory error, let me look into it and get back soon!

Thanks,

Divya

[transkriptase]

Hello Divya,

thank you for the answer. I add normalized_instance but still get memory error like I used 80 GB memory in GPU

[transkriptase]

Hello Divya,

today, I tried to run a trackking with a local GPU PC (not on hpc) and got same memory error. I will also try same setting with old version of sleap but I guess this is sleap_dev version error. Please let me know how can I handle this problem.

This os the part of the output:

(sleap_dev) C:\Users\okilic\Desktop\cuttung_legs>sleap-track "C-/3/C-_8_3_5000_21.03.240.mp4" -m "models/baseline.centroid_2" -m "models/baseline_medium_rf.topdown_3" -tracking.tracker simplemaxtracks --tracking.similarity normalized_instance --tracking.max_tracking 1 --tracking.max_tracks 58 --tracking.track_window 5 -o "tracked/8_3_simple_58_5_dev.slp"
Started inference at: 2024-11-06 13:59:43.689863
Args:
{
│ 'data_path': 'C-/3/C-_8_3_5000_21.03.240.mp4',
│ 'models': ['models/baseline.centroid_2', 'models/baseline_medium_rf.topdown_3'],
│ 'frames': '',
│ 'only_labeled_frames': False,
│ 'only_suggested_frames': False,
│ 'output': 'tracked/8_3_simple_58_5_dev.slp',
│ 'no_empty_frames': False,
│ 'verbosity': 'rich',
│ 'video.dataset': None,
│ 'video.input_format': 'channels_last',
│ 'video.index': '',
│ 'cpu': False,
│ 'first_gpu': False,
│ 'last_gpu': False,
│ 'gpu': 'auto',
│ 'max_edge_length_ratio': 0.25,
│ 'dist_penalty_weight': 1.0,
│ 'batch_size': 4,
│ 'open_in_gui': False,
│ 'peak_threshold': 0.2,
│ 'max_instances': None,
│ 'tracking.tracker': None,
│ 'tracking.max_tracking': True,
│ 'tracking.max_tracks': 58,
│ 'tracking.target_instance_count': None,
│ 'tracking.pre_cull_to_target': None,
│ 'tracking.pre_cull_iou_threshold': None,
│ 'tracking.post_connect_single_breaks': None,
│ 'tracking.clean_instance_count': None,
│ 'tracking.clean_iou_threshold': None,
│ 'tracking.similarity': 'normalized_instance',
│ 'tracking.match': None,
│ 'tracking.robust': None,
│ 'tracking.track_window': 5,
│ 'tracking.min_new_track_points': None,
│ 'tracking.min_match_points': None,
│ 'tracking.img_scale': None,
│ 'tracking.of_window_size': None,
│ 'tracking.of_max_levels': None,
│ 'tracking.save_shifted_instances': None,
│ 'tracking.kf_node_indices': None,
│ 'tracking.kf_init_frame_count': None,
│ 'tracking.oks_errors': None,
│ 'tracking.oks_score_weighting': None,
│ 'tracking.oks_normalization': None
}

INFO:sleap.nn.inference:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU.
Versions:
SLEAP: 1.4.1a2
TensorFlow: 2.7.0
Numpy: 1.21.5
Python: 3.7.12
OS: Windows-10-10.0.19041-SP0

System:
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initialized: False
Memory growth: True

Video: C-/3/C-_8_3_5000_21.03.240.mp4
2024-11-06 13:59:44.132703: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-06 13:59:44.803231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5992 MB memory: -> device: 0, name: Quadro RTX 4000, pci bus id: 0000:2d:00.0, compute capability: 7.5
Predicting... ---------------------------------------- 0% ETA: -:--:-- ?2024-11-06 13:59:54.008450: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
2024-11-06 13:59:59.913415: W tensorflow/core/common_runtime/bfc_allocator.cc:343] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
Predicting... -------- ------------------------------- 22% ETA: 0:06:50 9.5 FPS2024-11-06 14:03:08.753512: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 17179869184 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:03:08.753748: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 17179869184
Predicting... ---------------- ----------------------- 42% ETA: 0:04:07 11.6 FPS2024-11-06 14:05:15.406063: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:15.406208: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:15.409315: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:15.409396: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:15.439622: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:15.439755: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:15.530290: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:15.530452: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:15.542733: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:15.542906: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:15.544296: E t

........
.......
......
2024-11-06 14:05:18.084218: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.085371: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.086167: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.087192: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.087920: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.098767: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.100617: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.102357: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.102519: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.113215: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.113350: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.114132: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.114608: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.115355: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.115840: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.116622: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.117014: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.118084: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.118274: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.122915: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.123094: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.123749: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.124013: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.124636: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.124904: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.125518: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.125789: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.126408: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.128894: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.136464: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.136536: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.137147: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.137434: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.138136: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.138255: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.138971: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.139110: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:18.153993: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.154116: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
Predicting...2024-11-06 14:05:18.160100: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:18.160259: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
----------------- ---------------------- 43% ETA: 0:05:07 9.3 FPS2024-11-06 14:05:25.417799: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:25.418065: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:05:25.420855: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:05:25.421657: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368

Best,
Özge

[transkriptase]

Hello again Divya,

I got the same memory error with old version of sleap as well

2024-11-06 14:38:33.842811: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:38:33.842923: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:38:33.843719: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:38:33.844549: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:38:33.850671: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:38:33.851067: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:38:33.851593: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:38:33.851910: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
Predicting... ----------------- ---------------------- 43% ETA: 0:04:10 11.4 FPS2024-11-06 14:38:41.708348: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:38:41.708663: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368
2024-11-06 14:38:41.712492: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2024-11-06 14:38:41.713379: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368

(sleap) C:\Users\okilic\Desktop\cuttung_legs>

so, I have no idea now what to do :)