DPIR

DPIR, or Plug-and-Play Image Restoration with Deep Denoiser Prior, is a denoise and deblocking neural network. See also https://github.com/HolyWu/vs-dpir.

DPIR requires a strength parameter.

Link:

• (stable) https://github.com/AmusementClub/vs-mlrt/releases/download/model-20211209/dpir_v3.7z

Includes these models:

• Denoise models, default sigma is 5.0
  • drunet_gray: GRAY denoise
  • drunet_color: RGB denoise
• Deblocking models, default sigma is 50.0
  • drunet_deblocking_grayscale: GRAY deblocking
  • drunet_deblocking_color: RGB deblocking

Requirements & Parameters

1. block_w and block_h (tile size) must be multiples of 8.
2. All DPIR models require a strength parameter, or sigma, and you need to pass that in the form of a GRAYS clip (with normalization factor 1.0/255), see examples below for details.

vsmlrt.py wrapper Usage

In order to simplify usage, we provided a Python wrapper module vsmlrt that provides a more Pythonic interface:

from vsmlrt import DPIR, DPIRModel, Backend

src = core.std.BlankClip(format=vs.RGBS) # or vs.GRAYS for gray only models

# backend could be:
#  - CPU Backend.OV_CPU(): the recommended CPU backend; generally faster than ORT-CPU.
#  - CPU Backend.ORT_CPU(num_streams=1, verbosity=2): vs-ort cpu backend.
#  - GPU Backend.ORT_CUDA(device_id=0, cudnn_benchmark=True, num_streams=1, verbosity=2)
#     - use device_id to select device
#     - set cudnn_benchmark=False to reduce script reload latency when debugging, but with slight throughput performance penalty.
# DPIR is a huge model and GPU backend is highly recommended.
# If the model runs out of GPU memory, increase the tiles parameter.
flt = DPIR(src, strength=5, model=DPIRModel.drunet_color, tiles=2, backend=Backend.ORT_CUDA())

If you want to use variable strength, you can also pass a GRAYS clip as strength parameter that has the same dimension as the input clip where each pixel stores the DPIR strength for that pixel.

Raw Model Usage

src = core.std.BlankClip(width=640, height=360, format=vs.GRAYS)
sigma = 2.0
flt = core.ov.Model([src, core.std.BlankClip(src, color=sigma/255.0)], "drunet_gray.onnx")

Notes

DPIR is a huge network and it is extremely slow when running on CPU (e.g. for 360p input, you might see 0.05fps/cpu).

Benchmarking

Measurements: FPS / Device Memory (GB)

Device memory:

• GPU: device memory including context

Tesla V100

Software: VapourSynth R57, Windows Server 2019, Graphics Driver 511.23

Input size: 1920x1080

Backends

1. vs-mlrt v6
2. vs-dpir v1.7.1, PyTorch 1.10.1+cu113, TensorRT 8.2.2, torch2trt 2732b35

Performance

FP32

Model	[1] ort-cuda	[1] trt	[1] trt (2 streams)	[2] cuda	[2] trt
gray	2.45 / 5188	2.59 / 3979		2.27 / 11552	2.45 / 3959
color	2.39 / 5220	2.51 / 4011	2.56 / 7187	2.12 / 11558	2.26 / 3979

FP16

Model	[1] ort-cuda	[1] trt	[1] trt (2 streams)	[2] cuda	[2] trt
gray	5.20 / 3018	8.09 / 2831		4.55 / 11289	6.93 / 3461
color	4.95 / 3058	7.54 / 2863	7.70 / 4135	4.29 / 11302	5.60 / 3473

Tesla A10

Software: VapourSynth R57, Windows Server 2019, Graphics Driver 511.23

Input size: 1920x1080

Backends

1. vs-mlrt v6
2. vs-dpir v1.7.1, PyTorch 1.10.1+cu113, TensorRT 8.2.2, torch2trt 2732b35

Performance

FP32

Model	[1] ort-cuda	[1] trt	[2] cuda	[2] trt
gray	2.34 / 5791	2.75 / 4015	2.20 / 11837	2.67 / 4189
color	2.29 / 5823	2.73 / 4075	2.12 / 11853	2.54 / 4209

FP16

Model	[1] ort-cuda	[1] trt	[2] cuda	[2] trt
gray	3.73 / 3621	6.67 / 3437	3.72 / 11853	6.17 / 4079
color	3.65 / 3661	6.26 / 3423	3.45 / 11597	5.25 / 4103