[MoE] Cleanup MoE examples

[kylesayrs]

Purpose

• Update MoE examples to reflect latest MoE models
• Remove redundant moe examples, standardize examples around W4A16

Prerequisites

• [Pipelines] infer model device with optional override #1572
• [MoE] DeepSeek-V3/R1 #1535

Changes

• Just four examples
  • deepseekv2_5_example.py
  • deepseekv3_example.py
  • mixtral_example.py
  • qwen_example.py
• Update examples tests to run mixtral and qwen but not deepseek examples

[gemini-code-assist]

Summary of Changes

Hello @kylesayrs, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on cleaning up and modernizing the Mixture of Experts (MoE) quantization examples. It updates the models used, standardizes quantization configurations, and significantly improves the handling of model input devices within the calibration pipelines, making the examples more robust and aligned with current best practices.

Highlights

• Updated MoE Examples: The DeepSeek and Mixtral quantization examples have been updated to use newer model IDs (DeepSeek-V2.5, Mixtral-8x7B-Instruct-v0.1), reduced calibration sample counts (from 2048 to 512), and standardized W4A16 quantization schemes. Outdated comments and version checks related to transformers library compatibility have also been removed.
• Improved Model Device Handling: Calibration pipelines (basic, layer_sequential, sequential) now dynamically infer the model's input device using a new infer_model_device utility. This enhances flexibility and robustness, especially for models with offloaded parameters, and allows for explicit device specification via a new model_input_device argument in DatasetArguments.
• Standardized Generation and Saving: Sample generation code across examples has been simplified for consistency, using a common input format and max_new_tokens=100. Save directory names now consistently reflect the W4A16 quantization scheme.
• Test Suite Updates: The test suite for MoE examples has been updated to reflect the new example filenames, ensuring that the deepseekv2_5_example.py, mixtral_example.py, and qwen_example.py scripts are properly included in automated testing.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[gemini-code-assist]

Code Review

This pull request provides a nice cleanup for the Mixture of Experts (MoE) examples, updating them to use more recent models and simplifying the quantization recipes. The most significant improvement is the robust handling of device placement for calibration data across various pipelines, which is a great enhancement.

My main feedback is to avoid hardcoding the "cuda" device in the sample generation part of the example scripts. Inferring the device from the model would make the examples more portable and robust. I've left specific suggestions on how to achieve this in the relevant files.

[dsikka]

Just an fyi - both fp8 and w4a16 were popular enough to end up their own examples as a convenience (especially as fp8 can be data free and most popular on vLLM so it’s helpful to have that to point to).

I would keep 1-2 fp8, at least for the most popular models