[Model] Support Gemma 3 backend for Ultravox

[farzadab]

This PR enables using Ultravox with Gemma 3 backend.
It also adds a new <|audio|> token to the tokenizer itself instead of relying on the Llama-specific <|reserved_special_token_0|> token.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[farzadab]

@ywang96 @DarkLight1337 PTAL.
I have not made the model public yet. If required, I can open source the 1B model for testing purposes.

[DarkLight1337]

Looks reasonable to me. The tests should be able to catch if this breaks older models.

[DarkLight1337]

Can you merge from main to fix docker build?

[DarkLight1337]

Can you fix the failing test?

[farzadab]

It turns out this method is not working when using V1 on Llama-based backends.

I never noticed this because the embedding size of Gemma-3-4B is larger than their actual text vocab_size.
With V0, I can capture the extra dummy audio_token_id and replace it with 0 before computing inputs_embeds.

I can't do that in V1 however which leads to the error: Token id 128256 is out of vocabulary.

[farzadab]

Closing in favour of #17818