Question-Answering Task

[jrohsc]

Hi,

How can I use Qwen2-Audio model on a speech question-answering task? I tried both providing and not providing the instruction but it seems like the model is only transcribing the audio input.

[hsoftxl]

what is you code?

[jrohsc]

import torch
from transformers import AutoProcessor, Qwen2AudioForConditionalGeneration
import librosa

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Using device:", device)

model = Qwen2AudioForConditionalGeneration.from_pretrained("Qwen/Qwen2-Audio-7B", trust_remote_code=True)
model = model.to(device)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-Audio-7B", trust_remote_code=True)

prompt = "<|audio_bos|><|AUDIO|><|audio_eos|> "
path = "path_to_audio_file.mp3"

audio, sr = librosa.load(path, sr=processor.feature_extractor.sampling_rate)
inputs = processor(text=prompt, audios=audio, return_tensors="pt").to(device)

with torch.no_grad():
generated_ids = model.generate(**inputs, max_length=256)
generated_ids = generated_ids[:, inputs.input_ids.size(1):]
response = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]

print(response)