Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
NOTE: Requires this llama.cpp branch, which is where most of the changes are: https://github.com/nomic-ai/llama.cpp/tree/update-llamacpp-base
We will have to reset our llama.cpp fork's master branch to that commit before we can properly merge this.
What I did:
It seems to run fine with app.py on my RX 7800 XT. 0cc4m tested a slightly older version of the changes to llama.cpp and also had success (after some initial failures that we couldn't reproduce).