Live_Portrait_Monitor

If you find this project useful, please give it a star ❤️❤️

Monitor results

video6105138957194890677.mp4

video6105138957194890678.mp4

video6105138957194890680.mp4

video6105138957194890682.mp4

video6105138957194890681.mp4

Webcam result

My_img.mp4

🔥 Updates

• 2024/07/10: 🔥 I released the initial version of the inference code for webcam.
• 2024/07/16: 🔥 I released the initial version of the inference code for monitor.
• 2024/07/26: 🔥 I released windows package for faster inference speed. Continuous updates, stay tuned!

Introduction

This repo, named Webcam Live Portrait, Author paper LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control. I am actively updating and improving this repository for research only. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) 💖.

🔥 Getting Started

1. Clone the code and prepare the environment

git clone https://github.com/Mrkomiljon/Live_Portrait_Monitor.git
cd Live_Portrait_Monitor

# create env using conda
conda create -n LivePortrait python==3.9.18
conda activate LivePortrait
# install dependencies with pip
pip install -r requirements.txt
pip install mss # for using monitor

2. Download pretrained weights

Download pretrained LivePortrait weights and face detection models of InsightFace from Google Drive or Baidu Yun. All weights have packed in one directory 😊. Unzip and place them in ./pretrained_weights ensuring the directory structure is as follows:

pretrained_weights
├── insightface
│   └── models
│       └── buffalo_l
│           ├── 2d106det.onnx
│           └── det_10g.onnx
└── liveportrait
    ├── base_models
    │   ├── appearance_feature_extractor.pth
    │   ├── motion_extractor.pth
    │   ├── spade_generator.pth
    │   └── warping_module.pth
    ├── landmark.onnx
    └── retargeting_models
        └── stitching_retargeting_module.pth

3. Inference 🚀

python inference_org.py

If the script runs successfully, you will get an output mp4 file named animations/s6--d0_concat.mp4. This file includes the following results: driving video, input image, and generated result.

Unrealtime result

My_photo--d6_concat.mp4

Or, you can change the input by specifying the -s and -d arguments come from webcam:

# to use monitor
python inference_monitor.py -s assets/examples/source/MY_photo.jpg 

# or disable pasting back
python inference_monitor.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 --no_flag_pasteback

# to use original code for inference 
python inference_org.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 --no_flag_pasteback

# more options to see
python inference_org.py -h

# For good result, please play with padding (src/crop.py) for driver head.
    y1 = max(0, y1 - 320)                # Upper part of the head
    y2 = min(frame.shape[0], y2 + 550)   # Lower part of the head
    x1 = max(0, x1 - 360)                # Both sides of the head
    x2 = min(frame.shape[1], x2 + 360)   # Both sides of the head

5. Improved inference speed 🚀🚀🚀

IMG_0.MOV

Windows_results.MOV

• 2024/07/26: Windows integrated package, one-click run without installation, supporting TensorRT. Thanks to author for supporting TensorRT.
• [Optional] If you have already installed CUDA and cuDNN on your Windows computer, please skip this step. I have only verified this on CUDA 12.4. If you haven't installed CUDA or if you encounter CUDA-related errors, you need to follow - these steps for installation::
• Download cuda12.2, double-click the exe and follow the default settings to install.
• Download the cudnn zip file, extract it and copy the lib, bin, and include folders from the cudnn folder to the CUDA12.4 folder (default is C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4)
• [Optional] Install ffmpeg
• Download Live_Portrait_Monitor zip file and extract it.
• Enter FasterLivePortrait_High_FPS
• For real-time Monitor operation, double-click Monitor.bat. If you want to change the target image, run in command line: camera.bat assets/examples/source/s1.jpg
• Note!!! There should be a driver face video on Monitor[1] or [2].
• I have adapted the code for readability on monitors and introduced some functionalities. You can easily use it.

Acknowledgements

I would like to thank the contributors of FOMM, Open Facevid2vid, SPADE, InsightFace repositories, for their open research and main authors.

（Some demo images/videos above are sourced from image websites/repos. If there is any infringement, I will immediately remove them and apologize.）