Cant load IP Adapter to FLUX

[AeroDEmi]

Is there a problem with the latest version of Diffusers?

I'm trying to run the following snippet with Diffusers 0.33.1 and Transformers 4.51.3

from diffusers import FluxPipeline
from diffusers.utils import load_image

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16
).to("cuda")

image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flux_ip_adapter_input.jpg").resize((1024, 1024))

pipe.load_ip_adapter(
    "XLabs-AI/flux-ip-adapter",
    weight_name="ip_adapter.safetensors",
    image_encoder_pretrained_model_name_or_path="openai/clip-vit-large-patch14"
)
pipe.set_ip_adapter_scale(1.0)

It breaks when loading the ip adapter:
TypeError: CLIPVisionModelWithProjection.__init__() got an unexpected keyword argument 'dtype'

[AeroDEmi]

I will also add a comment on the transformers github.
Edit: Link

[AeroDEmi]

Here is the error:
3e59d53

[AeroDEmi]

I fixed with this:

import torch
from diffusers import FluxPipeline, LCMScheduler
from transformers import CLIPVisionModelWithProjection

image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    "openai/clip-vit-large-patch14",
    torch_dtype=torch.bfloat16
)

pipe: FluxPipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    image_encoder=image_encoder,
    torch_dtype=torch.bfloat16,
).to("cuda")

pipe.load_ip_adapter("XLabs-AI/flux-ip-adapter", weight_name="ip_adapter.safetensors")