Ollama also supports multimodal models, which can interact with (but not create) images.
We start by loading the package:
After loading the package, we need to pull a model that can handle
images. For example, the llava
model. Using pull_model("llava") will download the model,
or just load it if it has already been downloaded before.
We can use textual and visual input together. For instance, we can
ask a question and provide a link to a picture or a local file path,
such as images = "/home/user/Pictures/IMG_4561.jpg".
In the first example, we ask the model to describe the logo of this package:
query("Excitedly desscribe this logo", model = "llava",
images = "https://raw.githubusercontent.com/JBGruber/rollama/master/man/figures/logo.png")
#>
#> ── Answer from llava ─────────────────────────────────────────────────
#> The image you've shared is a vibrant and playful logo. At the center
#> of the design, there's an animated character that appears to be a
#> white, cat-like creature with blue eyes and ears. This character
#> seems to be in a relaxed state, laying on its stomach with its head
#> resting comfortably on one arm while the other arm is stretched out,
#> adding to the overall whimsical feel of the logo.
#>
#> Above this character, there's a blue circular element with some sort
#> of design or text, but it's not clear enough for me to describe.
#> Below the character, the word "ROLLAM" is prominently displayed in
#> bold black letters, suggesting that this could be the name of the
#> entity represented by the logo.
#>
#> The background of the logo features a light blue color, providing a
#> soft contrast to the central character and text elements. The overall
#> design of the logo suggests it might be for a gaming or
#> entertainment-related company or product, given the animated
#> character and playful aesthetic.The second example asks a classification question:
query("Which animal is in this image: a llama, dog, or walrus?",
model = "llava",
images = "https://raw.githubusercontent.com/JBGruber/rollama/master/man/figures/logo.png")
#>
#> ── Answer from llava ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> The image features a character that appears to be a llama wearing a
#> blue helmet, lying on grass.