MultiThreadedAugmenter.abort_event was set

[Jiachenn99]

Hey there, I'm trying to run nnUNet on the BraTS2020 dataset, however when I run the command nnUNet_train 3d_fullres nnUNetTrainerV2BraTSRegions_DA3_BN Task082_BraTS2020 5 , I obtain the error

RuntimeError: MultiThreadedAugmenter.abort_event was set, something went wrong. Maybe one of your workers crashed. This is not the actual error message! Look further up your stdout to see what caused the error. Please also check whether your RAM was full

I am using the default configurations, with the GPU being a RTX2080Ti . Not too sure what could be the issue.

[FabianIsensee]

Please read this: RuntimeError: MultiThreadedAugmenter.abort_event was set, something went wrong. Maybe one of your workers crashed. This is not the actual error message! Look further up your stdout to see what caused the error. Please also check whether your RAM was full

As it says, this is not the error message. Please look further up the output to see what caused the error

[Jiachenn99]

By checking the traceback, could it be that the dataloader was not initialized properly? FYI my installation is nnUNet as a framework using the command pip install -e .

  File "/cs/home/hfyjc3/nnUNet/bin/nnUNet_train", line 33, in <module>
    sys.exit(load_entry_point('nnunet', 'console_scripts', 'nnUNet_train')())
  File "/cs/home/hfyjc3/nnUNet/nnunet/run/run_training.py", line 160, in main
    trainer.run_training()
  File "/cs/home/hfyjc3/nnUNet/nnunet/training/network_training/nnUNetTrainerV2.py", line 442, in run_training
    ret = super().run_training()
  File "/cs/home/hfyjc3/nnUNet/nnunet/training/network_training/nnUNetTrainer.py", line 319, in run_training
    super(nnUNetTrainer, self).run_training()
  File "/cs/home/hfyjc3/nnUNet/nnunet/training/network_training/network_trainer.py", line 417, in run_training
    _ = self.tr_gen.next()
  File "/cs/home/hfyjc3/nnUNet/lib64/python3.6/site-packages/batchgenerators/dataloading/multi_threaded_augmenter.py", line 181, in next
    return self.__next__()
  File "/cs/home/hfyjc3/nnUNet/lib64/python3.6/site-packages/batchgenerators/dataloading/multi_threaded_augmenter.py", line 205, in __next__
    item = self.__get_next_item()
  File "/cs/home/hfyjc3/nnUNet/lib64/python3.6/site-packages/batchgenerators/dataloading/multi_threaded_augmenter.py", line 189, in __get_next_item
    raise RuntimeError("MultiThreadedAugmenter.abort_event was set, something went wrong. Maybe one of "

[FabianIsensee]

this is still not the error message :-) There must be something further up

[FabianIsensee]

For example if I delete files while training, this is my error:

Exception in background worker 1:
[Errno 2] No such file or directory: '/media/fabian/data/nnUNet_preprocessed/Task004_Hippocampus/nnUNetData_plans_v2.1_stage0/hippocampus_190.npy'
Traceback (most recent call last):
File "/home/fabian/pytorch-v1.7.1_cuda-11.1_cudnn-8.1.0.77/conda_env/bin/nnUNet_train", line 33, in
sys.exit(load_entry_point('nnunet', 'console_scripts', 'nnUNet_train')())
File "/home/fabian/http_git/nnunet/nnunet/run/run_training.py", line 179, in main
trainer.run_training()
File "/home/fabian/http_git/nnunet/nnunet/training/network_training/nnUNetTrainerV2.py", line 442, in run_training
ret = super().run_training()
File "/home/fabian/http_git/nnunet/nnunet/training/network_training/nnUNetTrainer.py", line 319, in run_training
super(nnUNetTrainer, self).run_training()
File "/home/fabian/http_git/nnunet/nnunet/training/network_training/network_trainer.py", line 466, in run_training
l = self.run_iteration(self.val_gen, False, True)
File "/home/fabian/http_git/nnunet/nnunet/training/network_training/nnUNetTrainerV2.py", line 232, in run_iteration
data_dict = next(data_generator)
File "/home/fabian/http_git/dldabg/batchgenerators/dataloading/multi_threaded_augmenter.py", line 205, in next
item = self.__get_next_item()
File "/home/fabian/http_git/dldabg/batchgenerators/dataloading/multi_threaded_augmenter.py", line 189, in __get_next_item
raise RuntimeError("MultiThreadedAugmenter.abort_event was set, something went wrong. Maybe one of "
RuntimeError: MultiThreadedAugmenter.abort_event was set, something went wrong. Maybe one of your workers crashed. This is not the actual error message! Look further up your stdout to see what caused the error. Please also check whether your RAM was full

Out of all this, only

Exception in background worker 1: [Errno 2] No such file or directory: '/media/fabian/data/nnUNet_preprocessed/Task004_Hippocampus/nnUNetData_plans_v2.1_stage0/hippocampus_190.npy'

is relevant

[Jiachenn99]

Ah okay I found out the issue, I just didn't allocate enough memory (RAM) to it. Seems like it does take up a huge chunk of memory. I'm working on a scheduler and so I decided to just allocate more memory to it. Thanks for answering my questions @FabianIsensee

[FabianIsensee]

Yeah that can happen with large data. I am unfortunately bound by the scipy.ndimage implementation as this is the only one I found that offers enough flexibility :-/