Skip to content

Fail to run model on CPU using IPEX-XPU installation #815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shira-g opened this issue Apr 24, 2025 · 0 comments
Open

Fail to run model on CPU using IPEX-XPU installation #815

shira-g opened this issue Apr 24, 2025 · 0 comments
Assignees

Comments

@shira-g
Copy link

shira-g commented Apr 24, 2025

We want to use speculative decoding where one model runs on the xpu and another model (significantly smaller) runs on the cpu.
We installed xpu build and run the script from https://github.com/intel/intel-extension-for-pytorch/tree/release/2.6/examples/cpu/llm/inference :

python run.py --benchmark -m microsoft/Phi-3-mini-4k-instruct --input-tokens 1024 --max-new-tokens 128 --token-latency --dtype float32 --ipex

And we get the following error:

Traceback (most recent call last):
File "C:\Users\sdp\shira\single_instance\run_generation.py", line 301, in
model = ipex.llm.optimize(
^^^^^^^^^^^^^^^^^^
File "C:\Users\sdp\miniforge3\envs\shira-ipex\Lib\site-packages\intel_extension_for_pytorch\transformers\optimize.py", line 2157, in optimize
validate_device_avaliable(device)
File "C:\Users\sdp\miniforge3\envs\shira-ipex\Lib\site-packages\intel_extension_for_pytorch\transformers\optimize.py", line 1918, in validate_device_avaliable
error_message(device)
File "C:\Users\sdp\miniforge3\envs\shira-ipex\Lib\site-packages\intel_extension_for_pytorch\transformers\optimize.py", line 1909, in error_message
raise RuntimeError(
RuntimeError: Device [cpu] is not avaliable in your IPEX package, need to re-install IPEX with [cpu] support, exiting...
LLM RUNTIME ERROR: Running generation task failed. Quit.

Can we get support for CPU using XPU build?

@louie-tsai louie-tsai self-assigned this Apr 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants