Issue with updated import and kernel compatibility for Qwen2_5_VL model

[berkesule]

When running the updated version of the script examples/multimodal_vision/qwen_2_5_vl_example.py using the same model and version as before, I encounter the following error related to linear kernel implementations.

In the older version of the codebase, I was importing the model as:
from llmcompressor.transformers.tracing import TraceableQwen2_5_VLForConditionalGeneration

However, in the new version, the import is changed to:
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

When running the same model and version with this new import, I get the following error related to linear kernel implementation failures.

Is there any way to access the old import that worked previously?

Environment
Include all relevant environment information:

1. OS running google colab
2. Python 3.11.13
3. llmcompressor==0.5.3
4. torch 2.6.0

Errors
ValueError: Failed to find a kernel that can implement the WNA16 linear layer. Reasons:

• MacheteLinearKernel requires capability 90, current compute capability is 80
• AllSparkLinearKernel cannot implement due to: For Ampere GPU, AllSpark does not support group_size = 128. Only group_size = -1 are supported.
• MarlinLinearKernel cannot implement due to: Weight output_size_per_partition = 3420 is not divisible by min_thread_n = 64. Consider reducing tensor_parallel_size or running with --quantization gptq.
• BitBLASLinearKernel cannot implement due to: bitblas is not installed. Please install bitblas by running pip install bitblas>=0.1.0
• ExllamaLinearKernel cannot implement due to: Output features must be a multiple of the pack factor (32 / num_bits) so that we can correctly pack the zero points

[kylesayrs]

Hi @berkesule!

It's likely that the issue you're seeing is a result of huggingface having changed their checkpoint format to use new module paths for the visual modules. This new format can cause the ignore list to fail to match, leading to the vision tower being quantized. Quantized vision towers lack robust runtime support in vllm.

Please ensure your script is using the new ignore list.
See #1545

- ignore=["lm_head", "re:visual.*"],
+ ignore=["lm_head", "re:visual.*", "re:model.visual.*"],

[berkesule]

Hi @berkesule!

It's likely that the issue you're seeing is a result of huggingface having changed their checkpoint format to use new module paths for the visual modules. This new format can cause the ignore list to fail to match, leading to the vision tower being quantized. Quantized vision towers lack robust runtime support in vllm.

Please ensure your script is using the new ignore list. See #1545

• ignore=["lm_head", "re:visual.*"],

• ignore=["lm_head", "re:visual.", "re:model.visual."],

Thank you, but I already use it like this.
recipe = [
GPTQModifier(
targets="Linear",
scheme="W4A16",
ignore=["lm_head", "re:visual.", "re:model.visual."],
),
]

Perform oneshot

oneshot(
model=model,
tokenizer=model_id,
dataset=ds,
recipe=recipe,
max_seq_length=MAX_SEQUENCE_LENGTH,
num_calibration_samples=503,
trust_remote_code_model=True,
data_collator=data_collator,
sequential_targets=["Qwen2_5_VLDecoderLayer"],
)

[berkesule]

I forgot to mention, I’m using a model that I trained myself with LoRA and merged. While I could quantize it without any issues in the old code, now I can’t.

Hi @berkesule!
It's likely that the issue you're seeing is a result of huggingface having changed their checkpoint format to use new module paths for the visual modules. This new format can cause the ignore list to fail to match, leading to the vision tower being quantized. Quantized vision towers lack robust runtime support in vllm.
Please ensure your script is using the new ignore list. See #1545

• ignore=["lm_head", "re:visual.*"],

• ignore=["lm_head", "re:visual.", "re:model.visual."],

Thank you, but I already use it like this. recipe = [ GPTQModifier( targets="Linear", scheme="W4A16", ignore=["lm_head", "re:visual.", "re:model.visual."], ), ]

Perform oneshot

oneshot( model=model, tokenizer=model_id, dataset=ds, recipe=recipe, max_seq_length=MAX_SEQUENCE_LENGTH, num_calibration_samples=503, trust_remote_code_model=True, data_collator=data_collator, sequential_targets=["Qwen2_5_VLDecoderLayer"], )

[kylesayrs]

@berkesule Thanks for the added context. Is it possible that your lora layers are being quantized?

[berkesule]

@berkesule Thanks for the added context. Is it possible that your lora layers are being quantized?

Sorry ı dont know.
Here is my fine-tuned and LoRA-merged model. I use this model in the code: Berkesule/qwen-vl-2.5-7b-merged

Here is the model I quantized with the old quantization code and that works flawlessly in vLLM: Berkesule/qwenvl-2.5-7b-gptq-W4A16-quantize-tr-dpo

Here is the code I quantized with the new quantization code and that causes errors in vLLM: Berkesule/qwenvl-2.5-7b-gptq-W4A16-quantize-tr-dpo-v2

[kylesayrs]

@berkesule These model checkpoints provide a lot more clarity, thanks for providing them. Since your new checkpoint looks correct, this is likely a bug on vllm, potentially with being unable to match the ignore list due to the new checkpoint format.

I'll investigate this further and attempt to replicate on my side. Thank you for calling this to attention!

[berkesule]

Previously, everything worked fine when I used this import:
from llmcompressor.transformers.tracing import (TraceableQwen2_5_VLForConditionalGeneration).
Is it still possible for me to access this?

[kylesayrs]

@berkesule You can find that model definition here. Although this model definition will be out of date w.r.t. the latest transformers changes.

I've been able to replicate your kernel issue, indeed the issue happens for visual.blocks.0.mlp.gate_proj. Still looking into a solution

[kylesayrs]

This is fixed by vllm-project/vllm#20046, although this leads to another error reported here.

[kylesayrs]

The two fixes required for support are here

vllm-project/vllm#20046
vllm-project/vllm#20058