Run_any_gguf_model_in_TextGen_webui.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": [],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/seyf1elislam/LocalLLM_OneClick_Colab/blob/main/Run_any_gguf_model_in_TextGen_webui.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# $\\color{#19ABEA}{\\text{Run any gguf quantized models in Latest Version TextGen-webui :}}$\n",
        "### `version 10/2024`\n",
        "> changes :  \n",
        "> fixed the '**' in the fs path.  \n",
        "> add new TextWebUI versions  .\n",
        "> add new models to the list\n",
        "\n",
        "this [Notebook](https://github.com/seyf1elislam/LocalLLM_OneClick_Colab) allow u run  any quantized gguf file from any Hugging face repository as long as it fits the colab Vram and ram.\n",
        "\n",
        ">follow this [githubRepository](https://github.com/seyf1elislam/LocalLLM_OneClick_Colab) to get the updates of this noteboook.\n",
        "\n",
        "> Created by :  \n",
        "> 😇 github :   [@seyf1elislam](https://github.com/seyf1elislam)  \n",
        "> 🤗 Huggingface : [@seyf1elislam](https://huggingface.co/seyf1elislam)\n",
        "\n",
        "## Quantized models Sources :\n",
        "> you can check those 🤗 huggingface repos :\n",
        "- [mradermacher](https://huggingface.co/mradermacher)  gguf source ,contains thousand of gguf quantized models.\n",
        "- [bartowski](https://huggingface.co/bartowski) #contains thousand of gguf quantized models\n",
        "- [LongStriker](https://huggingface.co/LoneStriker)  gguf source ,contains hundreds of quantized models.(exl2,gguf)\n",
        "- [QuantFactory](https://huggingface.co/QuantFactory) contains some gguf models.  \n",
        "- [using search](https://huggingface.co/models?sort=trending&search=gguf) here u can find all gguf files on  huggingface.\n",
        "\n",
        "## Good models to try :\n",
        "> you can try those :  \n",
        "- [QuantFactory/Mistral-Nemo-Instruct-2407-GGUF](https://huggingface.co/QuantFactory/Mistral-Nemo-Instruct-2407-GGUF) 12B model Q5_K_M / Q4_K_M (⭐🔥) .\n",
        "- [bartowski/Mistral-Small-Instruct-2409-GGUF](https://huggingface.co/bartowski/Mistral-Small-Instruct-2409-GGUF) this is 22B u can use it 3KM in 15g vram (⭐🔥) .\n",
        "- [Meta-Llama-3.1-8B-Instruct-GGUF](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF) Q8_0 (⭐🔥) .\n",
        "- [Meta-Llama-3-8B-Instruct-GGUF](https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF)  Q8_0  (⭐🔥) .\n",
        "- [gemma-2-9b-it-GGUF](https://huggingface.co/bartowski/gemma-2-9b-it-GGUF)  Q8_0/Q6   (⭐🔥) .\n",
        "\n",
        "## Some Tips :\n",
        "> in free colab gpu T4  (15G vram) you can use :\n",
        "> - 22b model quantized upto Q3_K_M(`context up to 8K`)\n",
        "> - 12b model quantized upto Q5_K_M(`context up to 16K`)\n",
        "> - 8b model quantized upto Q8_0(`context up to 16K or even more if ur model stable with long context`)\n",
        "\n",
        "> most  models goes with 8k context length if u want to use longer context you need to make sure the model support longer context.\n",
        "\n",
        "> if you want to run model higher then 20B (such as 20B,4x7b..) on colab you may need to reduce the offloaded gpu layers  to split the ram usage between gpu vram and system ram. (slower but it works 😉)"
      ],
      "metadata": {
        "id": "KW0lK7R8yp_i"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H_bLc7-6yVQS"
      },
      "outputs": [],
      "source": [
        "#@title # $\\color{#19ABEA}{\\text{Download and install requirements}}$ {display-mode: \"form\"}\n",
        "#@markdown $\\color{#19ABEA}{\\text{run this only once at the start}}$\n",
        "\n",
        "## @markdown `Note :no need to check these checkboxes unless u facing a problem`\n",
        "#@markdown ---\n",
        "#@markdown `check this if you need to change an installed version only `\n",
        "delete_existing_texgen_webui_folder =False #@param {type:\"boolean\"}\n",
        "if delete_existing_texgen_webui_folder :\n",
        "  !rm -rf /content/text-generation-webui\n",
        "#@markdown ---\n",
        "#@markdown ### TextGen webui version :\n",
        "#@markdown `v1.15(latest) has been released on 1 oct 2024`\n",
        "#@markdown `to get  more info about the newer versions  check the TextGenUI releases here:` https://github.com/oobabooga/text-generation-webui/releases\n",
        "\n",
        "%cd /content\n",
        "branch = \"v1.15\" # @param [ \"snapshot-2024-01-07\", \"snapshot-2024-01-14\", \"snapshot-2024-01-28\", \"snapshot-2024-02-11\", \"snapshot-2024-02-25\", \"snapshot-2024-03-03\", \"snapshot-2024-03-10\", \"snapshot-2024-03-17\", \"snapshot-2024-03-24\", \"snapshot-2024-03-31\", \"snapshot-2024-04-07\", \"snapshot-2024-04-14\", \"snapshot-2024-04-21\", \"snapshot-2024-04-28\",\"v1.8\",\"v1.12\",\"v1.14\",\"v1.13\",\"v1.15\",\"dev\"]\n",
        "branch = f\"-b {branch}\"\n",
        "#@markdown ----\n",
        "#@markdown `Noe :use this to download the latest version of TextGen Webui use use_latest ,this will ignore the branch/version name and download the latest .`\n",
        "use_latest =True #@param {type:\"boolean\"}\n",
        "\n",
        "if use_latest :\n",
        "  branch = \"\"\n",
        "!git clone {branch} https://github.com/oobabooga/text-generation-webui\n",
        "\n",
        "%cd /content/text-generation-webui\n",
        "!pip install -r requirements.txt\n",
        "!pip install git+https://github.com/UWUplus/flask-cloudflared\n",
        "!pip install -r extensions/openai/requirements.txt --upgrade\n",
        "!apt-get -y install -qq aria2\n",
        "#@markdown ----\n",
        "\n",
        "from IPython.display import clear_output\n",
        "clear_output()\n",
        "print(\"Done\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@title #  $\\color{#19ABEA}{Download Model}$ {display-mode: \"form\"}\n",
        "#@markdown ## $\\color{#19ABEA}{\\text{Copy the RepoName here :}}$\n",
        "\n",
        "#----------\n",
        "from huggingface_hub import HfFileSystem\n",
        "fs = HfFileSystem()\n",
        "\n",
        "def get_donwloadlink_and_filename(repo_name,quant='Q5_K_M'):\n",
        "  gguf_files = fs.glob(f\"{repo_name}/*.gguf\")\n",
        "  filtered_files = [file for file in gguf_files if (quant in file or (quant.lower()) or  (quant.upper()) in file)]\n",
        "\n",
        "  if len(filtered_files) ==0 : # quant not existed\n",
        "    print( \"=\" * 10 +\"\\navailable quants:\\n\",gguf_files)\n",
        "    return None,None\n",
        "  file_path= filtered_files[0]\n",
        "  *repo_name , file_name = file_path.split(\"/\");\n",
        "  repo_name='/'.join(repo_name)\n",
        "  return f\"https://huggingface.co/{repo_name}/resolve/main/{file_name}?download=true\" ,file_name\n",
        "\n",
        "#----------\n",
        "\n",
        "#@markdown  Example :`bartowski/Meta-Llama-3.1-8B-Instruct-GGUFF`\n",
        "repo_name = \"QuantFactory/Mistral-Nemo-Instruct-2407-GGUF\" # @param [\"QuantFactory/Meta-Llama-3-8B-Instruct-GGUF\",\"bartowski/gemma-2-9b-it-GGUF\",\"QuantFactory/Mistral-Nemo-Instruct-2407-GGUF\",\"bartowski/Mistral-Small-Instruct-2409-GGUF\",\"bartowski/Meta-Llama-3.1-8B-Instruct-GGUF\"] {\"allow-input\":true}\n",
        "\n",
        "#@markdown in free colab gpu T4  (15G vram) you can use :\n",
        "#@markdown - 22b model quantized upto Q3_K_M(`context up to 8K`)\n",
        "#@markdown - 12b model quantized upto Q5_K_M(`context up to 16K`)\n",
        "#@markdown - 8b model quantized upto Q8_0(`context up to 16K`)\n",
        "\n",
        "quant =\"Q5_K_M\" #@param ['Q2_K', 'Q3_K_L', 'Q3_K_M', 'Q3_K_S', 'Q4_0', 'Q4_1', 'Q4_K_M', 'Q4_K_S', 'Q5_0', 'Q5_1', 'Q5_K_M', 'Q5_K_S', 'Q6_K', 'Q8_0']\n",
        "model_download_url,model_file_name =get_donwloadlink_and_filename(repo_name,quant=quant)\n",
        "\n",
        "if model_download_url is not None and model_file_name is not None :\n",
        "  !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M {model_download_url} -d /content/text-generation-webui/models/ -o {model_file_name}\n",
        "else :\n",
        "  print(\"======================\")\n",
        "  print(\"please check the repo name or check availability of the quantized file inside the repo\")"
      ],
      "metadata": {
        "id": "9AsZFjNe0Y7_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@title ##  $\\color{#19ABEA}{RunOnly}$ {display-mode: \"form\"}\n",
        "# %cd /content/text-generation-webui\n",
        "#@markdown  $\\color{#19ABEA}{\\text{Enter ur Context size or let it unchecked to use the default size}}$\n",
        "params =\" \"\n",
        "#@markdown _____\n",
        "api =True # @param {type:\"boolean\"}\n",
        "if api :params +=\" --api --public-api --listen \"\n",
        "api_only_mode_nowebui =False # @param {type:\"boolean\"}\n",
        "if api_only_mode_nowebui : params+=\" --nowebui \"\n",
        "#@markdown _____\n",
        "use_custom_context_len = True # @param {type:\"boolean\"}\n",
        "context_len=8192 #@param\n",
        "if use_custom_context_len : params += f\" --n_ctx  {context_len} \"\n",
        "#@markdown _____\n",
        "use_custom_params = True # @param {type:\"boolean\"}\n",
        "custom_params = \"\" # @param {type:\"string\"}\n",
        "if use_custom_params : params +=custom_params\n",
        "\n",
        "#this if u want to use RoPE scale\n",
        "# params +=\" --alpha_value 1.5\"\n",
        "\n",
        "#@markdown _____\n",
        "# params\n",
        "!python server.py   --share  --n-gpu-layers 100000  \t--model $model_file_name   $params"
      ],
      "metadata": {
        "id": "8jBKRxrXVceJ"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}