foreground mask cuda index error

[hanjoonwon]

@niujinshuchong
I trained with
ns-train neus --pipeline.datamanager.train-num-rays-per-batch 2048 --pipeline.model.sdf-field.inside-outside False
--pipeline.model.sdf-field.hidden-dim 256 --pipeline.model.sdf-field.num-layers 2
--pipeline.model.sdf-field.num-layers-color 2 --pipeline.model.sdf-field.use-grid-feature False
--pipeline.model.sdf-field.bias 0.3 --pipeline.model.sdf-field.beta-init 0.3 --pipeline.model.background-model mlp
--pipeline.model.sdf-field.use-appearance-embedding True --trainer.steps-per-eval-image 5000
--trainer.max-num-iterations 100000 --pipeline.model.near-plane 0.05 --pipeline.model.far-plane 2. --vis wandb
--pipeline.model.overwrite-near-far-plane True --pipeline.model.mono-normal-loss-mult 0.01 --viewer.websocket-port 7008 --pipeline.model.mono-depth-loss-mult 0.0
--experiment-name neusmask_bat sdfstudio-data --data /home/xiangyue/joonwon/sdfstudio/data/sdfdata/sdfbat --include-mono-prior True --include_foreground_mask True

[freshlu11]

same problem when I use neuralRecon in the Wild, the command is " ns-train neusW --machine.num-gpus 2 --vis viewer --pipeline.model.sdf-field.inside-outside False --pipeline.model.sdf-field.use-appearance-embedding True heritage-data --data /home/user/NeuralRecon-W/recon_data/IMG_6853/split_0"

[lemondante]

That's because your mask shape is [a, b], but the interface require shape like[a, b, 3] and it will transform it to [a, b, 1]. When you use a numpy array like [a, b], it will become [a, 1], that makes cuda index error.
It's easy to fix, try use the following code in sdfstudio_dataparser.py:

            if self.config.include_foreground_mask:
                assert meta["has_foreground_mask"]
                # load foreground mask
                if self.config.load_dtu_highres:
                    # filenames format is 000.png
                    foreground_mask = np.array(
                        Image.open(
                            self.config.data / "mask" / frame["foreground_mask"].replace("_foreground_mask", "")[-7:]
                        ),
                        dtype="uint8",
                    )
                else:
                    # filenames format is 000000_foreground_mask.png
                    foreground_mask = np.array(Image.open(self.config.data / frame["foreground_mask"]), dtype="uint8")
# here is my fix
                if foreground_mask.ndim == 2:
                    foreground_mask = foreground_mask[:, :, np.newaxis]
                elif foreground_mask.ndim == 3:
                    foreground_mask = foreground_mask[..., :1]
# here ends
                foreground_mask_images.append(torch.from_numpy(foreground_mask).float() / 255.0)