jllllll/exllama

[techbutler]

Since jllllll/exllama doesn't have discussions enabled for that fork, I'm hoping someone that has installed that python module might be able to help me. I've installed all of the dependencies and Windows prerequisites, but I keep getting the below error when trying to install. Thanks in advance.

Collecting git+https://github.com/jllllll/exllama
Cloning https://github.com/jllllll/exllama to c:\users\11a8b\appdata\local\temp\pip-req-build-clohc0yd
Running command git clone --filter=blob:none --quiet https://github.com/jllllll/exllama 'C:\Users\11a8b\AppData\Local\Temp\pip-req-build-clohc0yd'
Resolved https://github.com/jllllll/exllama to commit 1dd02af
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [21 lines of output]
Traceback (most recent call last):
File "C:\Users\11a8b\AppData\Local\Programs\Python\Python311\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 353, in
main()
File "C:\Users\11a8b\AppData\Local\Programs\Python\Python311\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\11a8b\AppData\Local\Programs\Python\Python311\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\11a8b\AppData\Local\Temp\pip-build-env-2oat8h83\overlay\Lib\site-packages\setuptools\build_meta.py", line 341, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\11a8b\AppData\Local\Temp\pip-build-env-2oat8h83\overlay\Lib\site-packages\setuptools\build_meta.py", line 323, in _get_build_requires
self.run_setup()
File "C:\Users\11a8b\AppData\Local\Temp\pip-build-env-2oat8h83\overlay\Lib\site-packages\setuptools\build_meta.py", line 488, in run_setup
self).run_setup(setup_script=setup_script)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\11a8b\AppData\Local\Temp\pip-build-env-2oat8h83\overlay\Lib\site-packages\setuptools\build_meta.py", line 338, in run_setup
exec(code, locals())
File "", line 2, in
ModuleNotFoundError: No module named 'torch'
[end of output]

[turboderp]

I'm not sure what's going on with how they package the CUDA extension as an installable module, but by the looks of that error message it's not finding Torch. Have you installed PyTorch? Make sure the version you install matches your CUDA version.

[techbutler]

The PyTorch website has sample code to run to verify it was installed correctly. It ran as it was supposed to. I'll keep digging around.

[turboderp]

But installing PyTorch correctly doesn't mean you've installed the right version for your CUDA toolkit. PyTorch will run quite happily even without any GPU support. That's why the website has that whole process for picking which version to install.