Skip to content

Latest commit

 

History

History
239 lines (180 loc) · 8.28 KB

README.md

File metadata and controls

239 lines (180 loc) · 8.28 KB

Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement

arXiv hf WebPage

We present our reproduction of Vevo, a versatile zero-shot voice imitation framework with controllable timbre and style. We invite you to explore the audio samples to experience Vevo's capabilities firsthand.



We have included the following pre-trained Vevo models at Amphion:

  • Vevo-Timbre: It can conduct style-preserved voice conversion.
  • Vevo-Style: It can conduct style conversion, such as accent conversion and emotion conversion.
  • Vevo-Voice: It can conduct style-converted voice conversion.
  • Vevo-TTS: It can conduct style and timbre controllable TTS.

Besides, we also release the content tokenizer and content-style tokenizer proposed by Vevo. Notably, all these pre-trained models are trained on Emilia, containing 101k hours of speech data among six languages (English, Chinese, German, French, Japanese, and Korean).

Quickstart (Inference Only)

To run this model, you need to follow the steps below:

  1. Clone the repository and install the environment.
  2. Run the inference script.

Clone and Environment Setup

1. Clone the repository

git clone https://github.com/open-mmlab/Amphion.git
cd Amphion

2. Install the environment

Before start installing, making sure you are under the Amphion directory. If not, use cd to enter.

Since we use phonemizer to convert text to phoneme, you need to install espeak-ng first. More details can be found here. Choose the correct installation command according to your operating system:

# For Debian-like distribution (e.g. Ubuntu, Mint, etc.)
sudo apt-get install espeak-ng
# For RedHat-like distribution (e.g. CentOS, Fedora, etc.) 
sudo yum install espeak-ng

# For Windows
# Please visit https://github.com/espeak-ng/espeak-ng/releases to download .msi installer

Now, we are going to install the environment. It is recommended to use conda to configure:

conda create -n vevo python=3.10
conda activate vevo

pip install -r models/vc/vevo/requirements.txt

Inference Script

# Vevo-Timbre
python -m models.vc.vevo.infer_vevotimbre

# Vevo-Style
python -m models.vc.vevo.infer_vevostyle

# Vevo-Voice
python -m models.vc.vevo.infer_vevovoice

# Vevo-TTS
python -m models.vc.vevo.infer_vevotts

Running this will automatically download the pretrained model from HuggingFace and start the inference process. The result audio is by default saved in models/vc/vevo/wav/output*.wav, you can change this in the scripts models/vc/vevo/infer_vevo*.py

Training Recipe

For advanced users, we provide the following training recipe:

Emilia data preparation

  1. Please download the dataset following the official instructions provided by Emilia.

  2. Due to Emilia's substantial storage requirements, data loading logic may vary slightly depending on storage configuration. We provide a reference implementation for local disk loading in this file. After downloading the Emilia dataset, please adapt the data loading logic accordingly. In most cases, only modifying the paths specified in Lines 36-37 should be sufficient:

    MNT_PATH = "[Please fill out your emilia data root path]"
CACHE_PATH = "[Please fill out your emilia cache path]"

Launch Training

Train the Vevo tokenizers, the auto-regressive model, and the flow-matching model, respectively:

Note: You need to run the following commands under the Amphion root path:

git clone https://github.com/open-mmlab/Amphion.git
cd Amphion

Tokenizers

Run the following script:

# Content Tokenizer (Vocab = 32)
sh egs/codec/vevo/fvq32.sh

# Content-Style Tokenizer (Vocab = 8192)
sh egs/codec/vevo/fvq8192.sh

If you want to try different vocabulary sizes, just specify it in the egs/codec/vevo/fvq*.json:

{
    ...
     "model": {
        "repcodec": {
            "codebook_size": 8192, // Specify the vocabulary size here.
            ...
        },
        ...
    },
    ...
}

Auto-regressive Transformer

Specify the content tokenizer and content-style tokenizer paths in the egs/vc/AutoregressiveTransformer/ar_conversion.json:

{
    ...
    "model": {
        "input_repcodec": {
            "codebook_size": 32,
            "hidden_size": 1024, // Representations Dim
            "codebook_dim": 8,
            "vocos_dim": 384,
            "vocos_intermediate_dim": 2048,
            "vocos_num_layers": 12,
            "pretrained_path": "[Please fill out your pretrained model path]/model.safetensors" // The pre-trained content tokenizer
        },
        "output_repcodec": {
            "codebook_size": 8192, // VQ Codebook Size
            "hidden_size": 1024, // Representations Dim
            "codebook_dim": 8,
            "vocos_dim": 384,
            "vocos_intermediate_dim": 2048,
            "vocos_num_layers": 12,
            "pretrained_path": "[Please fill out your pretrained model path]/model.safetensors" // The pre-trained content-style tokenizer
        }
    },
    ...
}

Run the following script:

sh egs/vc/AutoregressiveTransformer/ar_conversion.sh

Similarly, you can run the following script for Vevo-TTS training:

sh egs/vc/AutoregressiveTransformer/ar_synthesis.sh

Flow-matching Transformer

Specify the pre-trained content-style tokenizer path in the egs/vc/FlowMatchingTransformer/fm_contentstyle.json:

{
    ...
    "model": {
        "repcodec": {
            "codebook_size": 8192, // VQ Codebook Size
            "hidden_size": 1024, // Representations Dim
            "codebook_dim": 8,
            "vocos_dim": 384,
            "vocos_intermediate_dim": 2048,
            "vocos_num_layers": 12,
            "pretrained_path": "[Please fill out your pretrained model path]/model.safetensors" // The pre-trained content-style tokenizer
        }
    },
    ...
}

Run the following script:

sh egs/vc/FlowMatchingTransformer/fm_contentstyle.sh

Vocoder

We provide a unified vocos-based vocoder training recipe for both speech and singing voice. See our Vevo1.5 framework for the details.

Citations

If you find this work useful for your research, please cite our paper:

@inproceedings{vevo,
  author       = {Xueyao Zhang and Xiaohui Zhang and Kainan Peng and Zhenyu Tang and Vimal Manohar and Yingru Liu and Jeff Hwang and Dangna Li and Yuhao Wang and Julian Chan and Yuan Huang and Zhizheng Wu and Mingbo Ma},
  title        = {Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement},
  booktitle    = {{ICLR}},
  publisher    = {OpenReview.net},
  year         = {2025}
}

If you use the Vevo pre-trained models or training recipe of Amphion, please also cite:

@article{amphion2,
  title        = {Overview of the Amphion Toolkit (v0.2)},
  author       = {Jiaqi Li and Xueyao Zhang and Yuancheng Wang and Haorui He and Chaoren Wang and Li Wang and Huan Liao and Junyi Ao and Zeyu Xie and Yiqiao Huang and Junan Zhang and Zhizheng Wu},
  year         = {2025},
  journal      = {arXiv preprint arXiv:2501.15442},
}

@inproceedings{amphion,
    author={Xueyao Zhang and Liumeng Xue and Yicheng Gu and Yuancheng Wang and Jiaqi Li and Haorui He and Chaoren Wang and Ting Song and Xi Chen and Zihao Fang and Haopeng Chen and Junan Zhang and Tze Ying Tang and Lexiao Zou and Mingxuan Wang and Jun Han and Kai Chen and Haizhou Li and Zhizheng Wu},
    title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit},
    booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024},
    year={2024}
}