RuntimeError: Model config for daclip_ViT-B-32 not found.

[Bruce-W12138]

作者您好，您的工作非常精彩，我再尝试训练da-clip的时候，会出现下面的报错，请问是否需要预先下载您的daclip_ViT-B-32.可是我在您readme文件里面并没有找到相关操作。事实上，我如果是要重新训练custom clip 的话应该只需要ViT-B-32即可。
PS：我使用的是Windows操作系统，不过我认为应该没有影响
2024-10-31,16:43:10 | INFO | Running with a single process. Device cuda:0.
2024-10-31,16:43:10 | ERROR | Model config for daclip_ViT-B-32 not found; available models ['coca_base', 'coca_roberta-ViT-B-32', 'coca_ViT-B-32', 'coca_ViT-L-14', 'convnext_base', 'convnext_base_w', 'convnext_base_w_320', 'convnext_large', 'convnext_large_d', 'convnext_large_d_320', 'convnext_small', 'convnext_tiny', 'convnext_xlarge', 'convnext_xxlarge', 'convnext_xxlarge_320', 'EVA01-g-14', 'EVA01-g-14-plus', 'EVA02-B-16', 'EVA02-E-14', 'EVA02-E-14-plus', 'EVA02-L-14', 'EVA02-L-14-336', 'MobileCLIP-B', 'MobileCLIP-S1', 'MobileCLIP-S2', 'mt5-base-ViT-B-32', 'mt5-xl-ViT-H-14', 'nllb-clip-base', 'nllb-clip-base-siglip', 'nllb-clip-large', 'nllb-clip-large-siglip', 'RN50', 'RN50-quickgelu', 'RN50x4', 'RN50x4-quickgelu', 'RN50x16', 'RN50x16-quickgelu', 'RN50x64', 'RN50x64-quickgelu', 'RN101', 'RN101-quickgelu', 'roberta-ViT-B-32', 'swin_base_patch4_window7_224', 'ViT-B-16', 'ViT-B-16-plus', 'ViT-B-16-plus-240', 'ViT-B-16-quickgelu', 'ViT-B-16-SigLIP', 'ViT-B-16-SigLIP-256', 'ViT-B-16-SigLIP-384', 'ViT-B-16-SigLIP-512', 'ViT-B-16-SigLIP-i18n-256', 'ViT-B-32', 'ViT-B-32-256', 'ViT-B-32-plus-256', 'ViT-B-32-quickgelu', 'ViT-bigG-14', 'ViT-bigG-14-CLIPA', 'ViT-bigG-14-CLIPA-336', 'ViT-bigG-14-quickgelu', 'ViT-e-14', 'ViT-g-14', 'ViT-H-14', 'ViT-H-14-378', 'ViT-H-14-378-quickgelu', 'ViT-H-14-CLIPA', 'ViT-H-14-CLIPA-336', 'ViT-H-14-quickgelu', 'ViT-H-16', 'ViT-L-14', 'ViT-L-14-280', 'ViT-L-14-336', 'ViT-L-14-336-quickgelu', 'ViT-L-14-CLIPA', 'ViT-L-14-CLIPA-336', 'ViT-L-14-quickgelu', 'ViT-L-16', 'ViT-L-16-320', 'ViT-L-16-SigLIP-256', 'ViT-L-16-SigLIP-384', 'ViT-M-16', 'ViT-M-16-alt', 'ViT-M-32', 'ViT-M-32-alt', 'ViT-S-16', 'ViT-S-16-alt', 'ViT-S-32', 'ViT-S-32-alt', 'ViT-SO400M-14-SigLIP', 'ViT-SO400M-14-SigLIP-378', 'ViT-SO400M-14-SigLIP-384', 'ViT-SO400M-16-SigLIP-i18n-256', 'vit_medium_patch16_gap_256', 'vit_relpos_medium_patch16_cls_224', 'ViTamin-B', 'ViTamin-B-LTT', 'ViTamin-L', 'ViTamin-L2', 'ViTamin-L2-256', 'ViTamin-L2-336', 'ViTamin-L-256', 'ViTamin-L-336', 'ViTamin-S', 'ViTamin-S-LTT', 'ViTamin-XL-256', 'ViTamin-XL-336', 'ViTamin-XL-384', 'xlm-roberta-base-ViT-B-32', 'xlm-roberta-large-ViT-H-14'].
Traceback (most recent call last):
File "D:\WorkProject\Image_Enhancement\daclip-uir\da-clip\src\training\main.py", line 490, in
main(sys.argv[1:])
File "D:\WorkProject\Image_Enhancement\daclip-uir\da-clip\src\training\main.py", line 218, in main
model, preprocess_train, preprocess_val = create_model_and_transforms(
File "C:\Users\12264.conda\envs\Mamba\lib\site-packages\open_clip\factory.py", line 484, in create_model_and_transforms
model = create_model(
File "C:\Users\12264.conda\envs\Mamba\lib\site-packages\open_clip\factory.py", line 299, in create_model
raise RuntimeError(f'Model config for {model_name} not found.')
RuntimeError: Model config for daclip_ViT-B-32 not found.
[2024-10-31 16:43:10,847] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2024-10-31 16:43:10,847] torch._dynamo.utils: [INFO] Function Runtimes (s)
[2024-10-31 16:43:10,847] torch._dynamo.utils: [INFO] ---------- --------------

[Bruce-W12138]

我可能知道问题在哪了，似乎daclip_ViT-B-32是没有注册？

[Algolzw]

是的，daclip没有merge到open clip里，所以你需要使用我们自己的da-clip文件夹：https://github.com/Algolzw/daclip-uir/tree/main/da-clip

[Bruce-W12138]

我就是在尝试运行src training里的main文件时报错了，我用了您给的参数设置：
--save-frequency
1
--zeroshot-frequency
1
--report-to
tensorboard
--train-data="D:/datasets/universal/daclip_train.csv"
--val-data="D:/datasets/universal/daclip_val.csv"
--csv-img-key
filepath
--csv-caption-key
title
--warmup
100
--batch-size=784
--lr=2e-5
--wd=0.05
--epochs=30
--workers=8
--model
daclip_
ViT-B-32
--name
"daclip_ViT-B-32_b784x1_lr2e-5_e50"
--pretrained
"laion2b_s34b_b79k"
--da

[Bruce-W12138]

不好意思打扰到您，为什么text的token数量设置成了154，这样很难和clip本身处理的77对齐
Traceback (most recent call last):
File "D:\WorkProject\Image_Enhancement\daclip-uir\da-clip\src\training\train.py", line 100, in train_one_epoch
model_out = model(images, texts)
File "C:\Users\12264.conda\envs\Mamba\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\12264.conda\envs\Mamba\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\12264.conda\envs\Mamba\lib\site-packages\open_clip\model.py", line 303, in forward
text_features = self.encode_text(text, normalize=True) if text is not None else None
File "C:\Users\12264.conda\envs\Mamba\lib\site-packages\open_clip\model.py", line 276, in encode_text
x = x + self.positional_embedding.to(cast_dtype)
RuntimeError: The size of tensor a (154) must match the size of tensor b (77) at non-singleton dimension 1
python-BaseException
[2024-10-31 20:10:36,178] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2024-10-31 20:10:36,178] torch._dynamo.utils: [INFO] Function Runtimes (s)
[2024-10-31 20:10:36,178] torch._dynamo.utils: [INFO] ---------- --------------

[Bruce-W12138]

[Bruce-W12138]

sorry，我感到很奇怪的地方在于我加载完batch以后对应text的tensor会变成3x154. 非常凑巧的是154正好是77的两倍，或许是我自己的某些操作失误。但如果可以的话，您能简单帮我看一眼就好，我会给您star

[Algolzw]

不好意思打扰到您，为什么text的token数量设置成了154，这样很难和clip本身处理的77对齐 Traceback (most recent call last): File "D:\WorkProject\Image_Enhancement\daclip-uir\da-clip\src\training\train.py", line 100, in train_one_epoch model_out = model(images, texts) File "C:\Users\12264.conda\envs\Mamba\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "C:\Users\12264.conda\envs\Mamba\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "C:\Users\12264.conda\envs\Mamba\lib\site-packages\open_clip\model.py", line 303, in forward text_features = self.encode_text(text, normalize=True) if text is not None else None File "C:\Users\12264.conda\envs\Mamba\lib\site-packages\open_clip\model.py", line 276, in encode_text x = x + self.positional_embedding.to(cast_dtype) RuntimeError: The size of tensor a (154) must match the size of tensor b (77) at non-singleton dimension 1 python-BaseException [2024-10-31 20:10:36,178] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics: [2024-10-31 20:10:36,178] torch._dynamo.utils: [INFO] Function Runtimes (s) [2024-10-31 20:10:36,178] torch._dynamo.utils: [INFO] ---------- --------------

这里text的token是包括degradation以及caption的所以是154，da-clip在forward函数中会将其分开：

daclip-uir/da-clip/src/open_clip/daclip_model.py

Line 65 in c2cc141

(caption, degradation) = text.chunk(2, dim=-1) if text is not None else (None, None)

.

另外在error信息中我发现你还是调用的原始的clip库，也许你可以新建一个环境或者pip remove open_clip.

[Bruce-W12138]

非常抱歉，事实上，我就是没有在See DA-CLIP.md for details.里面找到相应的步骤产生DaCLIP

我确实是在 https://github.com/Algolzw/daclip-uir/tree/main/da-clip 里面运行的，也许我在什么地方没有注意到。

[Bruce-W12138]

根据您在md的操作介绍，我一直到sigle-process running code这个地方之前都是顺利的，但是在整体运行training里的main函数的时候就提示daclip_ViT-B-32不存在，于是产生了我在这篇issue的第一个问题。是否需要提前下载预训练权重呢？
若您能详细解答这个问题，我将非常感激

[Algolzw]

看起来是默认加载了env里的clip，你尝试一下重新建一个environment或者删掉open clip包呢

[Bruce-W12138]

好的，非常感谢，我把open_clip pip uninstall 以后就可以成功运行了。