[Model] Ultravox: Support Llama 4 and Gemma 3 backends

[farzadab]

This is a simplified version of my older PR that was approved by @DarkLight1337 but ended up not working on some backends: https://github.com/vllm-project/vllm/pull/15728/files
This new PR allows Ultravox to support Gemma 3 and Llama 4 backends.

On the Ultravox side, I've made sure that all tokenizers have a new <|audio|> token to allow for better tracking audio placeholder tokens. This is only available on the tokenizer and not the embedding layer. As such, I intercept the input_ids before calling embedding on them and apply safe_input_ids instead.

When using V0, Ultravox has been verified to work on the following backends on an earlier version of this PR: Llama 3, Gemma 3, and Llama 4.

V0 seems to work as verified by evals. I've seen issues on V1 on an earlier version of VLLM, but I'm not sure if that was due to Ultravox or a VLLM V1 bug.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

What issue are you getting on V1?

[farzadab]

@DarkLight1337 When using V1, I noticed that the output was also completely garbled.

After debugging I noticed that when I tried printing input_ids here for the same sample (conditioned on len(input_ids)>1 to avoid decoding tokens), this is what I got:

# with VLLM_USE_V1=0
>>> t.decode([200000, 200005, 15651, 200006, 368, 4662, 583, 262, 19933, 43910, 26, 200008, 200005, 1556, 200006, 368, 4984, 290, 2182, 4097, 38, 7283, 201133, 200008, 200005, 140680, 200006, 368])
'<|begin_of_text|><|header_start|>system<|header_end|>\n\nYou are a helpful assistant.<|eot|><|header_start|>user<|header_end|>\n\nAnswer the following question: \n\n<|vision_reserved_special_token_1047|><|eot|><|header_start|>assistant<|header_end|>\n\n'

# with VLLM_USE_V1=1
>>> t.decode([24, 4984, 290, 2182, 4097, 38, 7283, 201133, 200008, 200005, 140680, 200006, 368])
',Answer the following question: \n\n<|vision_reserved_special_token_1047|><|eot|><|header_start|>assistant<|header_end|>\n\n'

The input_ids in the case of V1 seemed to be missing a part of the beginning.

[farzadab]

I believe I got this issue at around v0.8.4 or 0.8.4. I'll try verifying it on v0.8.5.post1.

[liPatrick]

Resolved by upgrading to v0.9.1

[liPatrick]

Verified that the issue with inference mismatch was indeed a VLLM bug. Upgrading to v0.9.1 fixed the issue and now V1 inference matches V0.

[DarkLight1337]

Nice, let's merge this!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @farzadab.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork