How to load SimMIM Swin-B pretrained model in mmdet? Error with "AssertionError: backbone. is not in the pretrained model"

[foolhard]

Hi,

Thanks for great job.

I try to load SimMIM Swin-B pretrained model in mmdet, but failed with AssertionError: backbone. is not in the pretrained model

How can I load the SimMIM pretrained model in mmdet?

The pretrained model is from https://download.openmmlab.com/mmselfsup/simmim/simmim_swin-base_16xb128-coslr-100e_in1k-192_20220316-1d090125.pth

Error message is:

Traceback (most recent call last):
  File "tools/train.py", line 189, in <module>
    main()
  File "tools/train.py", line 163, in main
    model.init_weights()
  File "/dfs/data/miniconda/envs/mmdet_vitdet/lib/python3.8/site-packages/mmcv/runner/base_module.py", line 116, in init_weights
    m.init_weights()
  File "/dfs/data/open_mmlab/mmclassification/mmcls/models/backbones/swin_transformer.py", line 414, in init_weights
    super(SwinTransformer, self).init_weights()
  File "/dfs/data/miniconda/envs/mmdet_vitdet/lib/python3.8/site-packages/mmcv/runner/base_module.py", line 105, in init_weights
    initialize(self, self.init_cfg)
  File "/dfs/data/miniconda/envs/mmdet_vitdet/lib/python3.8/site-packages/mmcv/cnn/utils/weight_init.py", line 613, in initialize
    _initialize(module, cp_cfg)
  File "/dfs/data/miniconda/envs/mmdet_vitdet/lib/python3.8/site-packages/mmcv/cnn/utils/weight_init.py", line 517, in _initialize
    func(module)
  File "/dfs/data/miniconda/envs/mmdet_vitdet/lib/python3.8/site-packages/mmcv/cnn/utils/weight_init.py", line 499, in __call__
    state_dict = _load_checkpoint_with_prefix(
  File "/dfs/data/miniconda/envs/mmdet_vitdet/lib/python3.8/site-packages/mmcv/runner/checkpoint.py", line 513, in _load_checkpoint_with_prefix
    assert state_dict, f'{prefix} is not in the pretrained model'
AssertionError: backbone. is not in the pretrained model

My environment is

# Name                    Version                   Build  Channel
mmcls                     0.23.1                   pypi_0    pypi
mmcv-full                 1.4.2                    pypi_0    pypi
mmdet                     2.19.1                    dev_0    <develop>
mmselfsup                 0.9.1                     dev_0    <develop>
timm                      0.5.4                    pypi_0    pypi

My mmdet config file is

_base_ = [
    '../_base_/models/mask_rcnn_r50_fpn.py',
    '../_base_/datasets/coco_instance.py',
    '../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
]


# please install mmcls>=0.22.0
# import mmcls.models to trigger register_module in mmcls
custom_imports = dict(imports=['mmcls.models'], allow_failed_imports=False)

checkpoint_file = 'ckpts/simmim_swin-base_16xb128-coslr-100e_in1k-192_20220316-1d090125.pth'

model = dict(
    backbone=dict(
        _delete_=True,
        type='mmcls.SwinTransformer',
        arch='base',
        img_size=224,
        drop_path_rate=0.2,
        out_indices=(0, 1, 2, 3),
        init_cfg=dict(type='Pretrained', checkpoint=checkpoint_file,prefix='backbone.')
    ),
    neck=dict(in_channels=[128, 256, 512, 1024])
)


optimizer = dict(
    _delete_=True,
    type='AdamW',
    lr=0.0001,
    betas=(0.9, 0.999),
    weight_decay=0.05,
    paramwise_cfg=dict(
        custom_keys={
            'absolute_pos_embed': dict(decay_mult=0.),
            'relative_position_bias_table': dict(decay_mult=0.),
            'norm': dict(decay_mult=0.)
        }))
lr_config = dict(warmup_iters=1000, step=[8, 11])
runner = dict(max_epochs=12)

[YuanLiuuuuuu]

The prefix should not be set, since the backbone. prefix has been removed in the provided checkpoint. Thanks!

[foolhard]

I set prefix = None in mmdet config file, and it works.
But found that mmcls Swin-B model does not match with mmselfsup SimMIM Swin-B model exactly.

Does this impact the model accuracy in mmdet?

Below is the log.

_2022-08-02 16:30:04,003 - mmcv - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: mask_token

missing keys in source state_dict: stages.0.blocks.0.attn.w_msa.relative_position_index, stages.0.blocks.1.attn.w_msa.relative_position_index, stages.1.blocks.0.attn.w_msa.relative_position_index, stages.1.blocks.1.attn.w_msa.relative_position_index, stages.2.blocks.0.attn.w_msa.relative_position_index, stages.2.blocks.1.attn.w_msa.relative_position_index, stages.2.blocks.2.attn.w_msa.relative_position_index, stages.2.blocks.3.attn.w_msa.relative_position_index, stages.2.blocks.4.attn.w_msa.relative_position_index, stages.2.blocks.5.attn.w_msa.relative_position_index, stages.2.blocks.6.attn.w_msa.relative_position_index, stages.2.blocks.7.attn.w_msa.relative_position_index, stages.2.blocks.8.attn.w_msa.relative_position_index, stages.2.blocks.9.attn.w_msa.relative_position_index, stages.2.blocks.10.attn.w_msa.relative_position_index, stages.2.blocks.11.attn.w_msa.relative_position_index, stages.2.blocks.12.attn.w_msa.relative_position_index, stages.2.blocks.13.attn.w_msa.relative_position_index, stages.2.blocks.14.attn.w_msa.relative_position_index, stages.2.blocks.15.attn.w_msa.relative_position_index, stages.2.blocks.16.attn.w_msa.relative_position_index, stages.2.blocks.17.attn.w_msa.relative_position_index, stages.3.blocks.0.attn.w_msa.relative_position_index, stages.3.blocks.1.attn.w_msa.relative_position_index, norm0.weight, norm0.bias, norm1.weight, norm1.bias, norm2.weight, norm2.bias_

[YuanLiuuuuuu]

It's OK. Since relative_position_index will be set to the initial value during the whole training process, whether or not loading it has marginal impact on the final result.

[foolhard]

Got it. Thanks a lot for kindly classification.