Add VisualCloze #11377

lzyhha · 2025-04-21T12:06:54Z

What does this PR do?

Add VisualCloze: A Universal Image Generation Framework via Visual In-Context Learning, an in-context learning based universal image generation framework, along with corresponding tests and documentation.

Here are some test codes and their results: Model Card.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

sayakpaul · 2025-04-21T12:17:13Z

@lzyhha thanks for your contribution. Could you please add some code snippets and results to the thread?

sayakpaul · 2025-04-21T12:17:29Z

Cc: @asomoza as well for testing if possible.

lzyhha · 2025-04-21T12:50:44Z

@lzyhha thanks for your contribution. Could you please add some code snippets and results to the thread?

Hello, here are some test codes and their results: Model Card.

HuggingFaceDocBuilderDev · 2025-04-21T12:59:08Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

asomoza · 2025-04-21T13:30:18Z

Hi, really nice and thank you for your work. Currently diffusers doesn't have einops as a dependency. Is it possible that you refactor all the rearrange calls to just use a torch equivalent without the need of external libraries?

lzyhha · 2025-04-21T13:56:12Z

Hi, really nice and thank you for your work. Currently diffusers doesn't have einops as a dependency. Is it possible that you refactor all the rearrange calls to just use a torch equivalent without the need of external libraries?

Okay, I will make the necessary modifications. Additionally, I noticed that the call method is not functioning properly in the documentation. Could you please help check the cause?

docs/source/en/api/pipelines/visualcloze.md

Co-authored-by: Álvaro Somoza <[email protected]>

lzyhha · 2025-04-22T01:57:21Z

Hello, we have removed einops from the code while ensuring the correctness of the results. @asomoza

sayakpaul

Thanks for working on this. I just added few minor comments.

I am unsure about self.denoise(). On one hand I see its value but since it deviates from our usual pipeline implementations, I will defer the decision to the other reviewers.

docs/source/en/api/pipelines/visualcloze.md

src/diffusers/pipelines/visualcloze/processor_visualcloze.py

src/diffusers/pipelines/visualcloze/pipeline_visualcloze.py

Co-authored-by: Sayak Paul <[email protected]>

lzyhha · 2025-04-23T06:06:51Z

Hello, we have made changes to the code based on your suggestions. @sayakpaul

sayakpaul

Thank you! I left some further comments and I will let the other reviewers comment here.

sayakpaul · 2025-04-23T16:53:05Z

docs/source/en/api/pipelines/overview.md

@@ -89,6 +89,7 @@ The table below lists all the pipelines currently available in 🤗 Diffusers an
 | [UniDiffuser](unidiffuser) | text2image, image2text, image variation, text variation, unconditional image generation, unconditional audio generation |
 | [Value-guided planning](value_guided_sampling) | value guided sampling |
 | [Wuerstchen](wuerstchen) | text2image |
+| [VisualCloze](visualcloze) | text2image, image2image, subject driven generation, inpainting, style transfer, image restoration, image editing, [depth,normal,edge,pose]2image, [depth,normal,edge,pose]-estimation, virtual try-on, image relighting |


cc: @stevhliu do you think it's alright?

src/diffusers/image_processor.py

tests/pipelines/visualcloze/test_pipeline_visualcloze.py

src/diffusers/pipelines/visualcloze/pipeline_visualcloze.py

Co-authored-by: Sayak Paul <[email protected]>

a-r-r-o-w

Thanks for the awesome PR, and congrats on the release of your work! Just some minor changes that are needed before we can proceed to merge

src/diffusers/image_processor.py

a-r-r-o-w · 2025-04-25T13:42:29Z

src/diffusers/pipelines/visualcloze/pipeline_visualcloze.py

+        ...     # in-context examples
+        ...     [
+        ...         load_image(
+        ...             "https://github.com/lzyhha/VisualCloze/tree/main/examples/examples/5bf755ed9dbb9b3e223e7ba35232b06e/5bf755ed9dbb9b3e223e7ba35232b06e_depth-anything-v2_Large.jpg"


Could you open a pull request to https://huggingface.co/datasets/huggingface/documentation-images/tree/main/diffusers and create a folder named visualcloze containing all necessary assets?

Hello, I have created a PR at https://huggingface.co/datasets/huggingface/documentation-images/discussions/483.
After it is merged, I will update the file URLs in the documentation.

Thanks for the PR! I've merged it

Thanks, I have updated the image URLs for the examples.

src/diffusers/pipelines/visualcloze/pipeline_visualcloze.py

a-r-r-o-w · 2025-04-25T13:49:20Z

src/diffusers/pipelines/visualcloze/pipeline_visualcloze.py

+
+        # Generate the target image latents by denoising the initial noise
+        # using the provided prompts and guidance scale
+        cloze_latents = self.denoise(


@yiyixuxu This is quite different from our usual pipeline design, but there is benefit to having it here to reduce duplicated code. Could you review this part as well?

src/diffusers/pipelines/visualcloze/pipeline_visualcloze.py

a-r-r-o-w · 2025-04-28T01:40:05Z

@bot /style

a-r-r-o-w

Thank you, the PR looks good to merge! I'll run the example scripts to verify and try to merge by tomorrow

lzyhha and others added 6 commits April 20, 2025 23:26

VisualCloze

4746498

Merge branch 'main' of https://github.com/lzyhha/diffusers into main

6507b60

style quality

fbf534e

add docs

c2f5771

add docs

4a5dbe2

Merge branch 'main' into main

7c6b26c

sayakpaul requested a review from a-r-r-o-w April 21, 2025 12:17

lzyhha added 2 commits April 21, 2025 20:54

typo

97943bb

Merge branch 'main' of https://github.com/lzyhha/diffusers

d1459d5

asomoza reviewed Apr 21, 2025

View reviewed changes

docs/source/en/api/pipelines/visualcloze.md Outdated Show resolved Hide resolved

yiyixuxu added this to Diffusers Roadmap 0.34 Apr 21, 2025

github-project-automation bot moved this to In Progress in Diffusers Roadmap 0.34 Apr 21, 2025

lzyhha and others added 3 commits April 22, 2025 08:50

Update docs/source/en/api/pipelines/visualcloze.md

b93152f

Co-authored-by: Álvaro Somoza <[email protected]>

delete einops

c70520c

style quality

7fe072b

Merge branch 'main' into main

c98b148

sayakpaul reviewed Apr 22, 2025

View reviewed changes

lzyhha and others added 6 commits April 22, 2025 20:59

Update src/diffusers/pipelines/visualcloze/pipeline_visualcloze.py

8a6f7b6

Co-authored-by: Sayak Paul <[email protected]>

reorg

0ef118c

Merge branch 'main' of https://github.com/lzyhha/diffusers into main

206d531

refine doc

cef2e24

style quality

060abe8

typo

0ea3343

Merge branch 'main' into main

94e7d87

typo

c420015

sayakpaul reviewed Apr 23, 2025

View reviewed changes

lzyhha and others added 4 commits April 24, 2025 09:10

Update src/diffusers/image_processor.py

72e021f

Co-authored-by: Sayak Paul <[email protected]>

add comment

0973942

test

ba3a528

style

73f95d0

a-r-r-o-w reviewed Apr 25, 2025

View reviewed changes

lzyhha and others added 4 commits April 26, 2025 06:54

Modified based on review

415aadb

style

6666dd7

restore image_processor

adea49d

Merge branch 'main' into main

3a016ab

lzyhha and others added 4 commits April 28, 2025 17:19

update example url

7378e28

style

01330a0

Merge branch 'main' of https://github.com/lzyhha/diffusers into main

e8f90b8

Merge branch 'main' into main

9dac423

a-r-r-o-w approved these changes Apr 28, 2025

View reviewed changes

fix-copies

096a2da

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add VisualCloze #11377

Add VisualCloze #11377

lzyhha commented Apr 21, 2025 •

edited

Loading

sayakpaul commented Apr 21, 2025

sayakpaul commented Apr 21, 2025

lzyhha commented Apr 21, 2025

HuggingFaceDocBuilderDev commented Apr 21, 2025

asomoza commented Apr 21, 2025

lzyhha commented Apr 21, 2025

lzyhha commented Apr 22, 2025

sayakpaul left a comment

lzyhha commented Apr 23, 2025

sayakpaul left a comment

sayakpaul Apr 23, 2025

a-r-r-o-w left a comment

a-r-r-o-w Apr 25, 2025

lzyhha Apr 25, 2025

a-r-r-o-w Apr 28, 2025

lzyhha Apr 28, 2025

a-r-r-o-w Apr 25, 2025

a-r-r-o-w commented Apr 28, 2025

a-r-r-o-w left a comment

Add VisualCloze #11377

Are you sure you want to change the base?

Add VisualCloze #11377

Conversation

lzyhha commented Apr 21, 2025 • edited Loading

What does this PR do?

Before submitting

Who can review?

sayakpaul commented Apr 21, 2025

sayakpaul commented Apr 21, 2025

lzyhha commented Apr 21, 2025

HuggingFaceDocBuilderDev commented Apr 21, 2025

asomoza commented Apr 21, 2025

lzyhha commented Apr 21, 2025

lzyhha commented Apr 22, 2025

sayakpaul left a comment

Choose a reason for hiding this comment

lzyhha commented Apr 23, 2025

sayakpaul left a comment

Choose a reason for hiding this comment

sayakpaul Apr 23, 2025

Choose a reason for hiding this comment

a-r-r-o-w left a comment

Choose a reason for hiding this comment

a-r-r-o-w Apr 25, 2025

Choose a reason for hiding this comment

lzyhha Apr 25, 2025

Choose a reason for hiding this comment

a-r-r-o-w Apr 28, 2025

Choose a reason for hiding this comment

lzyhha Apr 28, 2025

Choose a reason for hiding this comment

a-r-r-o-w Apr 25, 2025

Choose a reason for hiding this comment

a-r-r-o-w commented Apr 28, 2025

a-r-r-o-w left a comment

Choose a reason for hiding this comment

lzyhha commented Apr 21, 2025 •

edited

Loading