Should this be working on my iGPU?

[fhaust]

Describe the bug

Short story: One of the latest intel-oneapi-compiler-shared-opencl-cpu.icd versions causes every call to clGetPlatformIDs to hang indefinitely. This is probably not your issue, but I hope that you can suggest a work around.

Long story:

Running the code below just hangs forever on my system. The problem seems to be widespread (1, 2, 3, 4 and I have observed the issue on two different PCs) and can be "resolved" by just deleting the broken intel-oneapi-compiler-shared-opencl-cpu.icd.

Problem is that this does not seem to be a solution for ipex. When I delete intel-oneapi-compiler-shared-opencl-cpu.icd and keep the intel.icd, I get the error message below. The interesting bit is that ipex.optimize succeeds (with some warnings), but then there is a runtime error stating that the kernel has been compiled for a different device and has to be recompiled (followed by many "bf conversion" errors).

My testcode, slightly adapted from the StableLM base code:

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("StabilityAI/stablelm-base-alpha-3b")
model = AutoModelForCausalLM.from_pretrained("StabilityAI/stablelm-base-alpha-3b")

import intel_extension_for_pytorch as ipex
model = model.to('xpu')
model = ipex.optimize(model) 


inputs = tokenizer("What's your mood today?", return_tensors="pt").to('xpu')
tokens = model.generate(
  **inputs,
  max_new_tokens=64,
  temperature=0.7,
  do_sample=True,
)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))

The error trace: https://gist.github.com/fhaust/b1e7dbb7cd506ec1dbf67b0026e03e84
Most noteable the lines:

RuntimeError: The program was built for 1 devices
Build program log for 'Intel(R) UHD Graphics [0x9b41]':
warning: module got recompiled from IR because provided native binary is incompatible with underlying device and/or driver [-Wrecompiled-from-ir]

Versions

(that collect_env.py does not seem to work for me ... somehow it tries to screen capture?

# python -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.13.0a0+gitb1dde16
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (GCC) 12.2.1 20230201
Clang version: 15.0.7
CMake version: version 3.26.3
Libc version: glibc-2.37

Python version: 3.10.10 (main, Mar  5 2023, 22:26:53) [GCC 12.2.1 20230201] (64-bit runtime)
Python platform: Linux-6.2.12-arch1-1-x86_64-with-glibc2.37
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] intel-extension-for-pytorch==1.13.10+xpu
[pip3] numpy==1.23.4
[pip3] torch==1.13.0a0+gitb1dde16
[pip3] torchaudio==0.13.1+b90d798
[conda] No relevant packages

# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 142
model name	: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
stepping	: 12
microcode	: 0xf4
cpu MHz		: 800.273
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust sgx bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple pml ept_mode_based_exec
bugs		: spectre_v1 spectre_v2 spec_store_bypass swapgs itlb_multihit srbds mmio_stale_data retbleed eibrs_pbrsb
bogomips	: 4601.60
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

# lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation CometLake-U GT2 [UHD Graphics] (rev 02)

[gujinghui]

Sorry. For iGPU, we do not have many validation on it.
Its functionality is not fully insured so far.

[fcharras]

I'm also a user interested by iGPU support. I tried to compile from source using Gen11 AOT setting, using the compile_bundle.sh script. And it does compile and run ! But unfortunately it is nearly unusable, it is super slow (each command takes up to minuts).

But it seems to be more of a performance parameter tuning issue rather an issue with kernels. During compilation I've noticed that this warning is printed a lot:

...compiled SIMD32 allocated 128 regs and spilled around 254...

so it seems that for some reason IPEX tries to allocate a lot of registers and it causes a lot of registry spilling ? that might explain the near-frozen performance. @gujinghui do you know if there's a way to tune down the amount of allocated registries ?

[gujinghui]

I'm also a user interested by iGPU support. I tried to compile from source using Gen11 AOT setting, using the compile_bundle.sh script. And it does compile and run ! But unfortunately it is nearly unusable, it is super slow (each command takes up to minuts).

But it seems to be more of a performance parameter tuning issue rather an issue with kernels. During compilation I've noticed that this warning is printed a lot:

...compiled SIMD32 allocated 128 regs and spilled around 254...

so it seems that for some reason IPEX tries to allocate a lot of registers and it causes a lot of registry spilling ? that might explain the near-frozen performance. @gujinghui do you know if there's a way to tune down the amount of allocated registries ?

This warning is not a undoubted hints for kernel performance.
May I know whether the bad performance is only in first iteration run?
For iGPU, our pre-built package does not contain AOT binary, therefore, kernel JIT will happen with huge overhead in first run.

[fcharras]

@gujinghui you're right it's on first iteration run, so it seems to be JIT compiling cost ? but I don't have a warning message that the JIT is triggered, or any prompt that tells what is going on, but it's very long (more than a minut the first time I print a tensor in IPython). I used AOT=Gen11 as suggested by this doc when using compile_bundle.sh. The laptop I'm testing on is supposed to have a Gen11 CPU/iGPU.

Except this cost on first iteration run, some things seems to work well, and other don't:

torch.matmul is about as fast as dpnp.matmul even a bit faster.

However it seems that some torch functions return wrong results. See torch.topk: instead of returning the topk items and their indices, it returns the k first items of the array:

import intel_extension_for_pytorch
import torch
import numpy as np; rng = np.random.default_rng(123)
X = torch.from_numpy(rng.uniform(size=(2**27,)).astype(np.float32)).to("xpu")
torch.topk(X, 100)

gives

torch.return_types.topk(
values=tensor([0.6824, 0.0538, 0.2204, 0.1844, 0.1759, 0.8121, 0.9233, 0.2766, 0.8198,
        0.8899, 0.5130, 0.2450, 0.8242, 0.2138, 0.7415, 0.6299, 0.9274, 0.2319,
        0.7991, 0.5182, 0.2316, 0.1659, 0.4978, 0.5827, 0.1843, 0.0149, 0.4711,
        0.7282, 0.9186, 0.6255, 0.9171, 0.8647, 0.2181, 0.8661, 0.7308, 0.2779,
        0.7970, 0.8652, 0.2994, 0.5270, 0.0715, 0.5832, 0.2379, 0.7650, 0.1736,
        0.3127, 0.0145, 0.0326, 0.4967, 0.4683, 0.1277, 0.2576, 0.0032, 0.3811,
        0.5759, 0.4273, 0.8351, 0.6165, 0.2661, 0.8110, 0.4995, 0.7588, 0.5661,
        0.4374, 0.3962, 0.0222, 0.4694, 0.6236, 0.9461, 0.4353, 0.4856, 0.5191,
        0.4086, 0.5788, 0.0704, 0.4884, 0.6101, 0.7439, 0.4298, 0.3028, 0.0059,
        0.7565, 0.0776, 0.4900, 0.3044, 0.8408, 0.9505, 0.3189, 0.8978, 0.3375,
        0.8121, 0.7988, 0.6553, 0.2287, 0.1377, 0.4244, 0.1515, 0.8733, 0.1791,
        0.0303], device='xpu:0'),
indices=tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
        18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
        36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
        54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
        72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89,
        90, 91, 92, 93, 94, 95, 96, 97, 98, 99], device='xpu:0'))

so this snippet returns good results on Flex GPU but doesn't work on iGPU.

(on Flex:

values=tensor([1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
        1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
        1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
        1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
        1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
        1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
        1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
        1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
        1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
        1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
        1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
        1.0000], device='xpu:0'),
indices=tensor([ 39099291,  77660502,  97313695,   4588695,  19005012,  30310676,
         57071697,  66430123,  72786907,  83991316,  93679705, 111544794,
        114641230, 114753385, 115861539, 118233837, 129170868, 131733669,
          3570195,  23422416,  33457008,  68548533, 114301209, 124352022,
          2874187,  30092597,  56285362,  64226810,  64972397,  90656122,
         95269635,  98095709, 102840308,    795551,   9061400,   9069718,
          9907636,  77622556, 133071033,  18540435,  26138257,  26323696,
         26867357,  33768872,  59779911,  75781878,  93904476,  97679701,
        115697754,  45491637,  66618559,  80479944, 104767033, 108357655,
        131879905,  15536827,  43407600,  57188446,  59874212,  65401081,
         71392848,  72606666,  78973499,  80021152,  81552434,  95489003,
         97550865, 114925643, 129221556, 130119163,  11577994,  14809001,
         39018248,  57241931,  62599777,  83559166,  86154635,  86334627,
         88525641,  92830700, 112137861, 133193128,  12337235,  32563419,
         33002077,  72544265,  76133101,  90494135,  15598220,  19757407,
         26765893,  73635330,  79412528,  86531093, 123573669, 130055479,
         11408583,  19946137,  23811083,  42690506], device='xpu:0'))

[gujinghui]

Can you confirm whether the AOT is enabled correctly by below cmd?

1. export SYCL_PI_TRACE=-1
2. run workload to find below log lines.

	---> piProgramCreateWithBinary(
	<unknown> : 0xxxxxxxxxxxxxxx
	<unknown> : d
	<unknown> : 0xxxxxxxxxxxxxxx
	<unknown> : 0xxxxxxxxxxxxxxx
	<unknown> : 0xxxxxxxxxxxxxxx
	<unknown> : 0xxxxxxxxxxxxxxx
	) ---> pi_result : PI_SUCCESS

Or, you can run below cmd to figure out what kind of AOT is supported by current compiler.
ocloc compile --help

[gujinghui]

For topK, we will take a look soon. @arthuryuan1987

[fcharras]

For topK, we will take a look soon. @arthuryuan1987

Thanks!

Can you confirm whether the AOT is enabled correctly by below cmd?

I think I made a mistake and my device AOT setting should be tgllp or Gen12LP but I used Gen11. I try to compile again with both those AOT settings and will report back. (device is Intel Corporation TigerLake-LP GT2 [Iris Xe Graphics] (rev 01))

[fengyuan14]

The warning you saw is a mention from compiler to notify registers spill at the step of compilation, register allocation. It is not an indication to say bad performance, but a mention to kernel developers, whether register usage is expected. To evaluate expectation, developers need take something (both functionality and performance factors) into account, including,

1. Register addressing is not supported in some ISA. Have to spill and generate memory addressing instruction.
2. Temporary local variables usage in kernel is necessary or not, and performant or not. If the usage locality is guaranteed, these spill should always hit L1.
3. Trad-off concurrency and registers usage. SIMD instructions bring more powerful concurrency, but also bring more registers usage.

Re correctness of TopK on iGPU, we will follow up.

[fcharras]

@gujinghui compiling with the right AOT flag fixed the lag on first run. The good AOT for my device is one of Gen12LP or tgllp. Very happy that it can compile and run for igpus.

matmul is 6x faster than the CPU it's embedded with. That can really help for local work on laptops for a lot of people. Let me know if a fix for topk if possible, I'm very curious about the topk performance on igpu.

[fengyuan14]

@gujinghui compiling with the right AOT flag fixed the lag on first run. The good AOT for my device is one of Gen12LP or tgllp. Very happy that it can compile and run for igpus.

matmul is 6x faster than the CPU it's embedded with. That can really help for local work on laptops for a lot of people. Let me know if a fix for topk if possible, I'm very curious about the topk performance on igpu.

Sure. We have reproduced the failure on Gen9. We are fixing.

[fcharras]

Reporting some more on this issue, this bug probably only affects cases that fall back on the sort kernel, that you mentionned in #345 (comment) .

With less data, the topk works on iGPU, and the performance is satisfying from what I can tell. For instance torch.topk(data, 100) with data dtype float32 and shape (100, 1000000) runs in about 130ms, which is about equal the timings I get on the cpu it's embedded with.

[fengyuan14]

Reporting some more on this issue, this bug probably only affects cases that fall back on the sort kernel, that you mentionned in #345 (comment) .

With less data, the topk works on iGPU, and the performance is satisfying from what I can tell. For instance torch.topk(data, 100) with data dtype float32 and shape (100, 1000000) runs in about 130ms, which is about equal the timings I get on the cpu it's embedded with.

It's a platform compatibility issue. We have got a fixing in internal repository. You may pick it after next xpu-master synchronization.

[tye1]

The issue has been fixed in latest xpu-master, see a42e131. Please help to confirm and close this issue.