convert-ckpt-to-ggml.py : error

[raphael10-collab]

I want to convert this https://huggingface.co/speechbrain/asr-whisper-medium-commonvoice-it ckpt model to ggml :

Downloaded the model:

(.venv) raphy@raohy:~/whisper.cpp/models$ cat download-asr-w-medium-commonvoice-it.py
from huggingface_hub import snapshot_download
model_id="speechbrain/asr-whisper-medium-commonvoice-it"
snapshot_download(repo_id=model_id, local_dir="asr-w-medium-commonvoice-it",
                  local_dir_use_symlinks=False, revision="main")



(.venv) raphy@raohy:~/whisper.cpp/models$ python3 download-asr-w-medium-commonvoice-it.py
/home/raphy/whisper.cpp/models/.venv/lib/python3.12/site-packages/huggingface_hub/file_download.py:832: UserWarning: `local_dir_use_symlinks` parameter is deprecated and will be ignored. The process to download files to a local folder has been updated and do not rely on symlinks anymore. You only need to pass a destination folder as`local_dir`.
For more details, check out https://huggingface.co/docs/huggingface_hub/main/en/guides/download#download-files-to-local-folder.
  warnings.warn(
example-it.wav: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 136k/136k [00:00<00:00, 1.35MB/s]
config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 43.0/43.0 [00:00<00:00, 133kB/s]
hyperparams.yaml: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.58k/1.58k [00:00<00:00, 8.58MB/s]
.gitattributes: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.52k/1.52k [00:00<00:00, 9.23MB/s]
README.md: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.51k/4.51k [00:00<00:00, 17.3MB/s]
whisper.ckpt: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.06G/3.06G [00:54<00:00, 56.4MB/s]
Fetching 6 files: 100%|█

Downloaded convert-ckpt-to-ggml.py from :

https://github.com/ggerganov/ggml/blob/master/examples/gpt-2/convert-ckpt-to-ggml.py

And tried to convert the file , but got FileNotFoundError: [Errno 2] No such file or directory: './encoder.json' error :

(.venv) raphy@raohy:~/whisper.cpp/models/asr-w-medium-commonvoice-it$ python3 ../llama.cpp/convert-ckpt-to-ggml.py . --outfile asr-w-medium-commonvoice-it.ggml --outtype f16
2025-01-14 17:32:46.776224: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1736872366.881500   48970 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1736872366.910231   48970 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-14 17:32:47.152768: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Traceback (most recent call last):
  File "/home/raphy/whisper.cpp/models/asr-w-medium-commonvoice-it/../llama.cpp/convert-ckpt-to-ggml.py", line 66, in <module>
    with open(dir_model + "/encoder.json", "r", encoding="utf-8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './encoder.json'

How to make it work?

[ssakkout]

The convert-ckpt-to-ggml.py script you initially tried is for gpt models, not Whisper. It expects:

• encoder.json: contains token-to-index mappings for gpt.
• hparams.json: contains model hyperparameters.
  the thing is these files are not part of Whisper checkpoints.

Get the Whisper-specific conversion script:

wget https://raw.githubusercontent.com/ggerganov/whisper.cpp/master/models/convert-pt-to-ggml.py
then make sure you have the right version of torch, numpy and transformers

convert the speechbrain checkpoint to standard whisper format

import torch

# Load SpeechBrain checkpoint
ckpt = torch.load("asr-w-medium-commonvoice-it/whisper.ckpt", map_location="cpu")

# Create standard Whisper format
whisper_state = {
    "dims": {
        "n_vocab": 51865,  # Standard Whisper multilingual vocab size
        "n_audio_ctx": 1500,
        "n_audio_state": 1024,  # medium model size
        "n_audio_head": 16,     
        "n_audio_layer": 24,   
        "n_text_ctx": 448,
        "n_text_state": 1024,   
        "n_text_head": 16,      
        "n_text_layer": 24,     
        "n_mels": 80
    },
    "model_state_dict": {}
}

# Map SpeechBrain parameters to Whisper format
for key, value in ckpt["state"].items():
    # Remove 'model.' prefix if present
    key = key.replace('model.', '')
    whisper_state["model_state_dict"][key] = value

# Save converted checkpoint
torch.save(whisper_state, "whisper_converted.pt")

Clone the official whisper repo https://github.com/openai/whisper then run the final conversion with the parameters:

python convert-pt-to-ggml.py whisper_converted.pt ./whisper ./models/asr-whisper-medium-commonvoice-it

[raphael10-collab]

Hi @ssakkout ,

Tried to execute the Speechbrain 's checkpoints conversion to Whisper's checkpoints :

(.venv) riccardo@riccardo-HP-Laptop-15-da0xxx:~/Downloads/whisper.cpp/models$ cat SpeechbrainCheckpointsToWhisperCheckpointsConversion.py 
import torch

# Load SpeechBrain checkpoint
ckpt = torch.load("asr-whisper-medium-commonvoice-it/whisper.ckpt", map_location="cpu")

# Create standard Whisper format
whisper_state = {
    "dims": {
        "n_vocab": 51865,  # Standard Whisper multilingual vocab size
        "n_audio_ctx": 1500,
        "n_audio_state": 1024,  # medium model size
        "n_audio_head": 16,
        "n_audio_layer": 24,
        "n_text_ctx": 448,
        "n_text_state": 1024,
        "n_text_head": 16,
        "n_text_layer": 24,
        "n_mels": 80
    },
    "model_state_dict": {}
}

# Map SpeechBrain parameters to Whisper format
for key, value in ckpt["state"].items():
    # Remove 'model.' prefix if present
    key = key.replace('model.', '')
    whisper_state["model_state_dict"][key] = value

# Save converted checkpoint
torch.save(whisper_state, "whisper_converted.pt")

I get :

(.venv) riccardo@riccardo-HP-Laptop-15-da0xxx:~/Downloads/whisper.cpp/models$ python3 SpeechbrainCheckpointsToWhisperCheckpointsConversion.py 
/home/riccardo/Downloads/whisper.cpp/models/SpeechbrainCheckpointsToWhisperCheckpointsConversion.py:4: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  ckpt = torch.load("asr-whisper-medium-commonvoice-it/whisper.ckpt", map_location="cpu")
Traceback (most recent call last):
  File "/home/riccardo/Downloads/whisper.cpp/models/SpeechbrainCheckpointsToWhisperCheckpointsConversion.py", line 24, in <module>
    for key, value in ckpt["state"].items():
                      ~~~~^^^^^^^^^
KeyError: 'state'