Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When I fuse two perspectives into one video, it works fine during training, but fails during inference. #231

Open
wyclearnpy opened this issue Dec 26, 2024 · 1 comment

Comments

@wyclearnpy
Copy link

My data is like this.
img063

The terminal output is:

Warning: the argument seed shadows a Pipeline constructor argument of the same name.
/opt/dali/dali/operators/reader/nvdecoder/nvdecoder.cc:199: Unable to decode file /home/WYC/datasets/LP_2cam_own/videos/13.mp4
Error executing job with overrides: []
Traceback (most recent call last):
File "/home/WYC/code_repo/lightning-pose/scripts/predict_new_vids.py", line 95, in predict_videos_in_dir
export_predictions_and_labeled_video(
File "/home/WYC/.conda/envs/LP/lib/python3.10/site-packages/typeguard/init.py", line 1033, in wrapper
retval = func(*args, **kwargs)
File "/home/WYC/code_repo/lightning-pose/lightning_pose/utils/scripts.py", line 722, in export_predictions_and_labeled_video
preds_df = predict_single_video(
File "/home/WYC/.conda/envs/LP/lib/python3.10/site-packages/typeguard/init.py", line 1033, in wrapper
retval = func(*args, **kwargs)
File "/home/WYC/code_repo/lightning-pose/lightning_pose/utils/predictions.py", line 425, in predict_single_video
predict_loader = vid_pred_class()
File "/home/WYC/code_repo/lightning-pose/lightning_pose/data/dali.py", line 490, in call
return LitDaliWrapper(pipe, **args[self.train_stage][self.model_type])
File "/home/WYC/code_repo/lightning-pose/lightning_pose/data/dali.py", line 175, in init
super().init(*args, **kwargs)
File "/home/WYC/.conda/envs/LP/lib/python3.10/site-packages/nvidia/dali/plugin/pytorch/init.py", line 224, in init
self._first_batch = DALIGenericIterator.next(self)
File "/home/WYC/.conda/envs/LP/lib/python3.10/site-packages/nvidia/dali/plugin/pytorch/init.py", line 239, in next
outputs = self._get_outputs()
File "/home/WYC/.conda/envs/LP/lib/python3.10/site-packages/nvidia/dali/plugin/base_iterator.py", line 385, in _get_outputs
outputs.append(p.share_outputs())
File "/home/WYC/.conda/envs/LP/lib/python3.10/site-packages/nvidia/dali/pipeline.py", line 1215, in share_outputs
return self._pipe.ShareOutputs()
RuntimeError: Critical error in pipeline:
Error in GPU operator nvidia.dali.fn.readers.video,
which was used in the pipeline definition with the following traceback:

File "/home/WYC/code_repo/lightning-pose/lightning_pose/data/dali.py", line 91, in video_pipe
video = fn.readers.video(

encountered:

Video is too large in at least one dimension.
C++ context: [/opt/dali/dali/operators/reader/nvdecoder/cuvideodecoder.cc:231]
Current pipeline object is no longer valid.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

What is the problem, and how can I solve it?

@themattinthehatt
Copy link
Collaborator

themattinthehatt commented Jan 6, 2025

@wyclearnpy can you run ffprobe /home/WYC/datasets/LP_2cam_own/videos/13.mp4 from the command line and paste the output here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants