2D vs 3D Configuration

[abergsneider]

Hi @FabianIsensee,

TL;DR: 3D configuration should have a loss value closer to -1 than 2D using the compound loss of CE and Dice. My charts show the opposite 0_o

I'm so close to presenting results using the nnU-Net and an expected metric is flipped!! There must be something I'm not seeing right and was hoping I could get some guidance.

I performed complete training using both 2D and 3D configuration and then I submitted my results to KiTS19 and got a higher overall dice score using the 3D configuration as was expected. Now I'm looking at the progress.png files and attempting to make sense of them. The metrics on the learning curve, look reversed to me. 2D goes as far down as ~ -.9 and 3D to ~ -.7. I did notice that 3D starts at a higher than 0 compared to the 2D charts, but is my understanding that they should be as close to -1 as possible for good performance.

Shouldn't the loss reached by 3D be more negative than the one from 2D? Am I missing something here?

The trainings were run using dice and CE as a compound loss. I've appended the image below for reference.

[FabianIsensee]

Nah this is expected behavior. The loss can only be computed on patches/slices whereas KiTS is evaluated on whole images. Also there are some things going on with batch/sample Dice. Don't read anything into absolute values of the loss - it is merely there to show you how much we are overfitting

[abergsneider]

Thanks for the prompt response.

My understanding was that the best training performance is the one that gets closest to -1 (in the case of the compound loss). So that's what I was looking at. If not the absolute values, where can I look at to draw conclusions on the 3D model performing better?

I ran the nnUNet_find_best_configuration -m 2d 3d_fullres -t 121 --strict command as a sanity check and it told me 3d_fullres has better performance. (as expected)

Just interested in what other data other than running the above script I can look at to quantify performance differences.

[FabianIsensee]

nnU-Ner looka at average foreground Dice on the traiing set cross-validation. This is also reported in the stdout of nnUNet_find_best_configuration. And you can find it in the model output folder, for example:

results/nnUNet/3d_fullres/Task002_Heart/nnUNetTrainerV2__nnUNetPlansv2.1/cv_niftis_raw/summary.json

[abergsneider]

Awesome. I've checked the files and its got all I need to check the configuration performance, thanks!
Having said that, I'm just still curious for my own knowledge on the loss metric behaviour from the progress graphs.

I was wondering if since 2D configuration considers the full resolution at each slice, the percentage of background (easier to classify pixels) would be greater and therefore shifting the loss values making the performance look "better"?

On the other hand, 3D configuration takes patches focusing more on the foreground. The percentage of background voxels is much smaller this time, shifting the performance metric to a value less negative, "worse".

[FabianIsensee]

2D is also oversampling foreground slices. Since the batch size is larger, there is more diversity within each batch allowing for a dice loss that is better on average. Also remember that 2D uses batch dice (interpreting all the 2d slices as a single 3d volume for loss computation) which is not always what the 3d configuration does