Patterns for dealing with images?

[Gerkinfeltser]

I have Ollama running locally with the vlm "llava" & I'd like to make a few patterns for getting it to describe images. The issue is I'm not sure how to pass in an image. Is this currently possible or is this text only atm?

Thanks for fabric. Its super fun & inspiring to work with!

[thadius83]

I have Ollama running locally with the vlm "llava" & I'd like to make a few patterns for getting it to describe images. The issue is I'm not sure how to pass in an image. Is this currently possible or is this text only atm?

Thanks for fabric. Its super fun & inspiring to work with!

Did you ever get a response on this?

[LaBalaDeOro]

I have a similar need. Here's how to send an image to the OpenAI API https://platform.openai.com/docs/api-reference/images/createEdit
Using the https://github.com/danielmiessler/fabric/tree/main/patterns/create_coding_project might the easiest way to get this functionality now.

[LaBalaDeOro]

Looks like description for image files is a new feature as of November 8, 02024:

Multimodal Support: You can now use -a (attachment) for Multimodal submissions to OpenAI models that support it. Example: fabric -a https://path/to/image "Give me a description of this image."

Prolly doesn't work with local images so would need to be extended to support that use case. Might have a crack at it.

[LaBalaDeOro]

Just tested the -a flag and it does work with local images using both relative and absolute file paths!
I was only able to get a text response from the "Give me a description of this image." prompt however.

go run . -a "../../../Catalina Whale Mixer/catalina-whale-475.png" "Give me a description of this image."

The image features a cartoonish character that resembles a fish or a whale with a large, grinning face. The character has a green head, a prominent white mouth, and sharp teeth, giving it a mischievous appearance. It is wearing a bright orange beanie and stylish glasses, which add to its trendy look. The character is dressed in a gray hoodie layered under a denim jacket, set against a light blue background. The overall vibe is playful and humorous.

I tried image creating prompts but they did not return created images, only errors:

go run . -m dall-e-3 -a "../../../Catalina Whale Mixer/catalina-whale-475.png" "Add fluffy clouds to the background"

error, status code: 403, status: 403 Forbidden, message: You are not allowed to sample from this model
exit status 1