Propose to update & upgrade SkyReels-V2

[tolgacangoz]

• Fixed the visualisation and enhanced the documentation: Previously, HTML rendering at the docs page looked very bad, see: https://huggingface.co/docs/diffusers/main/en/api/pipelines/skyreels_v2#a-visual-demonstration.
• Reorganized the code by moving components into the attention dispatcher because the original repo uses "_native_cudnn" for self-attentions and "flash_varlen" or "_flash_varlen_3" for cross-attentions.
• Added support for pipeline.transformer.compile_repeated_blocks(fullgraph=True) and looking forward to be merged torch.compile compatibility with varlen APIs #11970.
• Took Wan's RoPE directly to be able to compile: Remember Use real-valued instead of complex tensors in Wan2.1 RoPE #11649.

Skywork/SkyReels-V2-DF-1.3B-540P
seed=0
main: ~14 min.	Wan's RoPE
main.mp4	Wan.s_RoPE.mp4
Wan's RoPE + compile_repeated_blocks(fullgraph=True): ~12 min.	Wan's RoPE + compile_repeated_blocks(fullgraph=True) + "_native_cudnn" for attn1 and "flash" for attn2, FA=2.8.3: ~8 min.
Wan.s_RoPE+regional.mp4	Wan.s_RoPE+regional+FA.mp4

Reproducer

!uv pip install git+https://github.com/tolgacangoz/diffusers.git@update-skyreels-v2

import torch, os
from diffusers import AutoModel, SkyReelsV2DiffusionForcingPipeline, UniPCMultistepScheduler
from diffusers.utils import export_to_video

# For faster loading into the GPU
os.environ["HF_ENABLE_PARALLEL_LOADING"] = "YES"

model_id = "Skywork/SkyReels-V2-DF-1.3B-540P-Diffusers"
vae = AutoModel.from_pretrained(model_id,
                                subfolder="vae",
                                torch_dtype=torch.float32,
                                device_map="cuda")
pipeline = SkyReelsV2DiffusionForcingPipeline.from_pretrained(
    model_id,
    vae=vae,
    torch_dtype=torch.bfloat16,
    device_map="cuda"
)
flow_shift = 8.0  # 8.0 for T2V, 5.0 for I2V
pipeline.scheduler = UniPCMultistepScheduler.from_config(pipeline.scheduler.config, flow_shift=flow_shift)

# Some acceleration helpers
# Be sure to install Flash Attention: https://github.com/Dao-AILab/flash-attention#installation-and-features
#for block in pipeline.transformer.blocks:
#    block.attn1.set_attention_backend("_native_cudnn")
#    block.attn2.set_attention_backend("flash")
#pipeline.transformer.compile_repeated_blocks(fullgraph=True)

prompt = "A cat and a dog baking a cake together in a kitchen. The cat is carefully measuring flour, while the dog is stirring the batter with a wooden spoon. The kitchen is cozy, with sunlight streaming through the window."

output = pipeline(
    prompt=prompt,
    num_inference_steps=30,
    height=544,  # 720 for 720P
    width=960,   # 1280 for 720P
    num_frames=97,
    base_num_frames=97,  # 121 for 720P
    ar_step=5,  # Controls asynchronous inference (0 for synchronous mode)
    causal_block_size=5,  # Number of frames in each block for asynchronous processing
    overlap_history=None,  # Number of frames to overlap for smooth transitions in long videos; 17 for long video generations
    addnoise_condition=20,  # Improves consistency in long video generation
    generator=torch.Generator("cpu").manual_seed(0)
).frames[0]
export_to_video(output, "T2V.mp4", fps=24, quality=8)

Environment

- 🤗 Diffusers version: 0.35.0 or this branch
- Platform: Linux-4.4.0-x86_64-with-glibc2.36
- Running on Google Colab?: No
- Python version: 3.12.6
- PyTorch version (GPU?): 2.8.0+cu126 (True)
- Flax version (CPU?/GPU?/TPU?): 0.11.0 (gpu)
- Jax version: 0.7.0
- JaxLib version: 0.7.0
- Huggingface_hub version: 0.34.3
- Transformers version: 4.55.0
- Accelerate version: 1.9.0
- PEFT version: not installed
- Bitsandbytes version: not installed
- Safetensors version: 0.6.1
- xFormers version: not installed
- Accelerator: NVIDIA A100-SXM4-40GB, 40960 MiB

@a-r-r-o-w @yiyixuxu @stevhliu

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[stevhliu]

Thanks for improving the docs!

[stevhliu]

Thanks for improving! I think the text block should only be used for the graph and chart visuals.

[tolgacangoz]

I modified accordingly. I used backticks for rows representation too, otherwise the columns don't seem as aligned. How is it now?

[stevhliu]

Why is the LoRA example being removed?

[tolgacangoz]

This part was copied from Wan's page. Since I didn't test anything with LoRAs, I removed the specific example. But SkyReels-V2 and Wan have almost the same architecture. I preserved - SkyReels-V2 supports LoRAs with [~loaders.SkyReelsV2LoraLoaderMixin.load_lora_weights].

[stevhliu]

Thanks, the docs LGTM!

Let's wait for @a-r-r-o-w to chime in on the other changes before we merge :)

[tolgacangoz]

Alright, thanks for your review!

[a-r-r-o-w]

Nice @tolgacangoz! Thanks for propagating the attention backend changes to SkyReels

[a-r-r-o-w]

This is nice! We can use this instead of iterating the blocks:

diffusers/src/diffusers/models/modeling_utils.py

Line 586 in 673d435

def set_attention_backend(self, backend: str) -> None:

[tolgacangoz]

Can we set different attention backends for attn1 and attn2?

[a-r-r-o-w]

Not sure why we need to showcase different backends though? Is there a specific reason?

Examples should showcase usage via the official expected APIs. There are usually multiple side-effects when not doing so. In this case, one instance could be the following warning not being raised:

• diffusers/src/diffusers/models/modeling_utils.py

  Line 603 in 673d435

  logger.warning("Attention backends are an experimental feature and the API may be subject to change.")

Another problem is the following check will be skipped. When skipped, not-so-technical users will be baffled by the traceback:

• diffusers/src/diffusers/models/modeling_utils.py

  Line 610 in 673d435

  _check_attention_backend_requirements(backend)

Custom uses are okay, but usually only best done by power-users who have a better understanding of the library

[tolgacangoz]

SkyReels-V2's self-attentions use "_native_cudnn" with a custom attention mask, and its cross-attentions use varlen types. We cannot set flash attention for all because it doesn't support a custom mask, right?

I agree to do this at the expected API. I opened a feature request for this, but then I thought it was not much necessary: #12210.

[tolgacangoz]

I can remove this # Some acceleration helpers comment section completely for now. Since this model is a bit slow, I just wanted to add several speedup helpers. I plan to profile and optimize this model.

[tolgacangoz]

Removed them.