Log full output not just training progress.

[lorinczszabolcs]

Describe the feature
Logging for me only shows the training progress (loss values and eval as a function of training steps).

Motivation

1. It would be nice to keep the full standard output (including weight initialization), to be able to check if it has been done correctly. Many times the output is too long and can not scroll up enough to see if e.g. the backbone is initialized correctly, due to constraint in how much one is able to scroll up in the terminal.

Additional context
The output.log only contains logs from the start of the training, as shown in the picture:

It would be really great if additional logging could be done (e.g. including the contents from the standard output right from the start, but I'm not sure how that could be solved. This feature would also be useful in other openmmlab libraries.

I would be open to make a PR if you could guide me in a direction where that could be solved / a preferred solution that would be optimal for you.

[fangyixiao18]

Hello, thanks for your reporting.
Could you find a file called 'train_xxxx_xxxx', maybe like 'train_20220524_143504.log', here we have output the more information.
e.g.

[lorinczszabolcs]

Hi! Thank you for your quick reply!

I only have a log called 'xxxx_xxxx.log.json' in work_dir, e.g. '20220810_104212.log.json', and it contains the following:

My wandb log folder has the following structure, but none of the files contain anything similar:

I also have a file called INFO, but that also only contains hooks and then the training progress:

No ''train_xxxx_xxxx.log' can be found in either work_dir or the wandb log folder unfortunately. Could it happen that my log config is somehow not correct?

[fangyixiao18]

could you provide your mmcv version and your command to start training?
.log file is supposed to be in the work_dir folder

[lorinczszabolcs]

MMCV version is 1.6.1.

Training is done by a custom script, but the main steps are:

...
# parse some args
...

# load config
cfg = Config.fromfile("...")
...

# create work_dir
cfg.work_dir = "../logs/{}".format(setting)

# set random seed
cfg.seed = 0
set_random_seed(0, deterministic=False)

# create the work directory
mmcv.mkdir_or_exist(osp.abspath(cfg.work_dir))

# set device and gpu
cfg.device = "cuda"
cfg.gpu_ids = range(num_gpus)
# build the algorithm
model = build_algorithm(cfg.model)

# init weights
model.init_weights()

# build the dataset
datasets = [build_dataset(cfg.data.train)]

# start pre-train
train_model(
    model,
    datasets,
    cfg,
    distributed=True if num_gpus > 1 else False,
    timestamp=time.strftime("%Y%m%d_%H%M%S", time.localtime()),
    meta=dict(),
)

[lorinczszabolcs]

Additional info: I am using mmsegmentation as well in another project, and the same issue appears there, only these logs are present in the mentioned directories, nothing similar to what you showed me. Could there be some kind of environment related issue? Both projects are running in docker containers.

[fangyixiao18]

Additional info: I am using mmsegmentation as well in another project, and the same issue appears there, only these logs are present in the mentioned directories, nothing similar to what you showed me. Could there be some kind of environment related issue? Both projects are running in docker containers.

How did you write your model? Did your inherit from BaseModule from mmcv? https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/base_module.py#L15

[lorinczszabolcs]

My model inherits from DenseCL

mmselfsup/mmselfsup/models/algorithms/densecl.py

Line 12 in 8c4f76d

class DenseCL(BaseModel):

, so it should from BaseModule as well. In MMSegmentation my models inherit from BaseSegmentor
https://github.com/open-mmlab/mmsegmentation/blob/dd42fa8d0125632371a41a87c20485494c973535/mmseg/models/segmentors/base.py#L13, so they also should inherit from BaseModule as well.

[fangyixiao18]

you can try initializing logger before build_model, like this https://github.com/open-mmlab/mmselfsup/blob/master/tools/train.py#L150

[lorinczszabolcs]

That solved the problem! Thanks!!