Introduction

After you installed Ollama on your machine and downloaded the package rollama you can load the package and pull a model. The default model (llama3.1), is a good all-round chat model. For annotation, however, the instruction tuned llama models are often better suited, as they follow instructions more diligently and are less likely to trail off into a conversation. By changing the option rollama_model, we can change which model is used by default in the current session:

library(rollama)
options(rollama_model = "llama3.2:3b-instruct-q8_0")
pull_model()
#> ✔ model llama3.2:3b-instruct-q8_0 pulled succesfully

Prompting Strategies

If you want to annotate textual data, you can use various prompting strategies. For an overview of common approaches, you can read a paper by Weber and Reichardt (2023). These strategies primarily differ in whether or how many examples are given (Zero-shot, One-shot, or Few-shot) and whether reasoning is involved (Chain-of-Thought).

When writing a prompt we can give the model content for the system part, user part and assistant part. The system message typically includes instructions or context that guides the interaction, setting the stage for how the user and the assistant should interact. For an annotation task we could write: “You assign texts into categories. Answer with just the correct category.” The table below summarizes different prompting strategies for annotating textual data. Each strategy varies in the number of examples given and the incorporation of reasoning.

Prompting Strategy	Example Structure
Zero-shot	`{"role": "system", "content": "Text of System Prompt"},` `{"role": "user", "content": "(Text to classify) + classification question"}`
One-shot	`{"role": "system", "content": "Text of System Prompt"},` `{"role": "user", "content": "(Example text) + classification question"},` `{"role": "assistant", "content": "Example classification"},` `{"role": "user", "content": "(Text to classify) + classification question"}`
Few-shot	`{"role": "system", "content": "Text of System Prompt"},` `{"role": "user", "content": "(Example text) + classification question"},` `{"role": "assistant", "content": "Example classification"},` `{"role": "user", "content": "(Example text) + classification question"},` `{"role": "assistant", "content": "Example classification"},` `. . . more examples` `{"role": "user", "content": "(Text to classify) + classification question"}`
Chain-of-Thought	`{"role": "system", "content": "Text of System Prompt"},` `{"role": "user", "content": "(Text to classify) + reasoning question"},` `{"role": "assistant", "content": "Reasoning"},` `{"role": "user", "content": "Classification question"}`

Zero-shot

In this approach, no prior examples are given. The structure includes a system prompt providing instructions and a user prompt with the text to classify and the classification question (in this example we only provide the categories).

library(tibble)
library(purrr)
q <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. Answer with just the correct category.",
  "user",   "text: the pizza tastes terrible\ncategories: positive, neutral, negative"
)
query(q)
#> 
#> ── Answer from llama3.2:3b-instruct-q8_0 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> negative

One-shot

This involves giving a single example before the actual task. The structure includes a system prompt, followed by a user prompt with an example text and classification question, the assistant’s example classification, and then another user prompt with the new text to classify.

q <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. Answer with just the correct category.",
  "user", "text: the pizza tastes terrible\ncategories: positive, neutral, negative",
  "assistant", "Category: Negative",
  "user", "text: the service is great\ncategories: positive, neutral, negative"
)
query(q)
#> 
#> ── Answer from llama3.2:3b-instruct-q8_0 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> Category: Positive

A nice side effect of the one-shot strategy (and all n>0-strategies) is that you can tune the format the model uses in its replies. For example, if you want to have an output that easy to parse, you could change the assistant message to "{'Category':'Negative'}"

q <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. Answer with just the correct category.",
  "user", "text: the pizza tastes terrible\ncategories: positive, neutral, negative",
  "assistant", "{'Category':'Negative'}",
  "user", "text: the service is great\ncategories: positive, neutral, negative"
)
answer <- query(q)
#> 
#> ── Answer from llama3.2:3b-instruct-q8_0 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> {'Category':'Positive'}

This is a valid JSON return and can be parsed into a list with, e.g., jsonlite::fromJSON(). Using this logic, we could request a more informative output:

q <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. Provide the following information: category, confidence, and the word that is most important for your coding decision.",
  "user", "text: the pizza tastes terrible\ncategories: positive, neutral, negative",
  "assistant", "{'Category':'Negative','Confidence':'100%','Important':'terrible'}",
  "user", "text: the service is great\ncategories: positive, neutral, negative"
)
answer <- query(q)
#> 
#> ── Answer from llama3.2:3b-instruct-q8_0 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> {'Category':'Positive','Confidence':'100%','Important':'great'}

By using pluck(answer, "message", "content"), you can directly extract the result and don’t need to copy it from screen.

Few-shot

This strategy includes multiple examples (more than one). The structure is similar to one-shot but with several iterations of user and assistant messages providing examples before the final text to classify.

q <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. Answer with just the correct category.",
  "user", "text: the pizza tastes terrible\ncategories: positive, neutral, negative",
  "assistant", "Category: Negative",
  "user", "text: the service is great\ncategories: positive, neutral, negative",
  "assistant", "Category: Positive",
  "user", "text: I once came here with my wife\ncategories: positive, neutral, negative",
  "assistant", "Category: Neutral",
  "user", "text: I once ate pizza\ncategories: positive, neutral, negative"
)
query(q)
#> 
#> ── Answer from llama3.2:3b-instruct-q8_0 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> Category: Neutral

Chain-of-Thought

This approach involves at least one reasoning step. The structure here starts with the system prompt, then a user prompt with a text to classify and a reasoning question.

q_thought <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. ",
  "user",   "text: the pizza tastes terrible\nWhat sentiment (positive, neutral, or negative) would you assign? Provide some thoughts."
)
output_thought <- query(q_thought, output = "text")
#> 
#> ── Answer from llama3.2:3b-instruct-q8_0 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> I would assign the sentiment of this text as "negative".
#> 
#> The phrase "tastes terrible" is a strong expression of dislike for the pizza, implying that it's unpalatable and unpleasant to eat. The use of the word "terrible" also indicates a strong negative emotion, rather than a mild
#> disappointment or dissatisfaction.
#> 
#> Additionally, there's no balance or mitigating language in the text, which further reinforces the negative sentiment. There are no words like "it could be better" or "I'm not sure if I like it", which would suggest a more neutral or
#> even positive tone.
#> 
#> Overall, the text conveys a strong negative opinion about the pizza, making "negative" the most suitable sentiment assignment.

In the next step we can use the assistant’s reasoning and a user prompt with the classification question.

q <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. ",
  "user",   "text: the pizza tastes terrible\nWhat sentiment (positive, neutral, or negative) would you assign? Provide some thoughts.",
  "assistant", output_thought,
  "user",   "Now answer with just the correct category (positive, neutral, or negative)"
)
resps <- query(q)
#> 
#> ── Answer from llama3.2:3b-instruct-q8_0 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> Negative

Batch annotation

In practice, you probably never want to annotate just one text. In this section, we show you how you can wrap rollama::query() into another function to ask the model to annotate a batch of texts. We might add this function to the package in the future, but at the moment, we want to keep it simple.

Function to create a query

The create_query function is designed to facilitate the creation of a structured query for text classification.

Components:

System Message: Provides context or instructions for the classification task.
Examples: Prior examples consisting of user messages and assistant responses (for one-shot and few-shot learning).
Text to Classify: The new text to be categorized.
Classification Question: Lists the possible categories for classification.

create_query <- function(systemmsg, examples, texttoclassify, classification_question) {
  # Start with the system message
  q <- tribble(
    ~role,    ~content,
    "system", systemmsg
  )

  # Add examples (if any), appending the classification question to the user messages
  for(example in examples) {
    usermsg_with_question <- paste(example$usermsg, "\n", classification_question)
    q <- add_row(q, role = "user", content = usermsg_with_question)
    q <- add_row(q, role = "assistant", content = example$assistantmsg)
  }

  # Add the current text to classify along with the classification question
  usermsg_final <- paste("text:", texttoclassify, "\n", classification_question)
  q <- add_row(q, role = "user", content = usermsg_final)

  return(q)
}

Example usage

Zero-shot example

In this example, the function is used without any examples.

systemmsg <- "You assign texts into categories. Answer with just the correct category."
q_zs <- create_query(systemmsg, examples = list(), "the pizza tastes terrible", "Categories: positive, neutral, negative")
query(q_zs)
#> 
#> ── Answer from llama3.2:3b-instruct-q8_0 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> negative

One-shot example with one example

Here, one prior example is provided to aid the classification:

examples_os <- list(
  list(
    usermsg = "text: the pizza tastes terrible",
    assistantmsg = "Category: Negative"
  )
)
q_os <- create_query(systemmsg, examples_os, "the service is great", "Categories: positive, neutral, negative")
query(q_os)
#> 
#> ── Answer from llama3.2:3b-instruct-q8_0 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> Category: Positive

Few-shot example with multiple examples

This scenario uses multiple examples to enrich the context for the new classification:

examples_fs <- list(
  list(
    usermsg = "text: the pizza tastes terrible",
    assistantmsg = "{Negative}"
  ),
  list(
    usermsg = "text: the service is great",
    assistantmsg = "{Positive}"
  ),
  list(
    usermsg = "text: I once came here with my wife",
    assistantmsg = "{Neutral}"
  )
)
q_fs <- create_query(systemmsg, examples_fs, "I once ate pizza", "Categories: positive, neutral, negative")
query(q_fs)
#> 
#> ── Answer from llama3.2:3b-instruct-q8_0 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> {Neutral}

Another example using a dataframe

This example demonstrates how to perform sentiment analysis on a set of movie reviews. The process involves creating a dataframe of reviews, processing each review to classify its sentiment, and appending the results as a new column in the dataframe.

We create a dataframe named movie_reviews with two columns:

# Create an example dataframe with 5 movie reviews
movie_reviews <- tibble(
  review_id = 1:5,
  review = c("A stunning visual spectacle with a gripping storyline.",
             "The plot was predictable, but the acting was superb.",
             "An overrated film with underwhelming performances.",
             "A beautiful tale of love and adventure, beautifully shot.",
             "The movie lacked depth, but the special effects were incredible.")
)
# Print the initial dataframe
movie_reviews
#> # A tibble: 5 × 2
#>   review_id review                                                          
#>       <int> <chr>                                                           
#> 1         1 A stunning visual spectacle with a gripping storyline.          
#> 2         2 The plot was predictable, but the acting was superb.            
#> 3         3 An overrated film with underwhelming performances.              
#> 4         4 A beautiful tale of love and adventure, beautifully shot.       
#> 5         5 The movie lacked depth, but the special effects were incredible.

We can use create_query again to define a query for each of these reviews. What we want to do is to perform a sentiment analysis, guided by a system message and a classification question.

systemmsg <- "Classify the sentiment of the movie review. Answer with just the correct category."
classification_question <- "Categories: positive, neutral, negative"

queries <- lapply(movie_reviews$review, function(t) create_query(
  systemmsg = systemmsg,
  examples = examples_fs,
  texttoclassify = t,
  classification_question = classification_question
))

The query function accepts lists of queries, so we can get the annotations simply using:

# Process and annotate the movie reviews
movie_reviews$annotation <- query(queries, screen = FALSE, output = "text")

# Print the annotated dataframe
movie_reviews
#> # A tibble: 5 × 3
#>   review_id review                                                    annotation
#>       <int> <chr>                                                     <chr>     
#> 1         1 A stunning visual spectacle with a gripping storyline.    {Positive}
#> 2         2 The plot was predictable, but the acting was superb.      {Neutral} 
#> 3         3 An overrated film with underwhelming performances.        {Negative}
#> 4         4 A beautiful tale of love and adventure, beautifully shot. {Positive}
#> 5         5 The movie lacked depth, but the special effects were inc… {Neutral}

This takes a little longer than classic supervised machine learning or even classification with transformer models. However, the advantage is that instructions can be provided using plain English, the models need very few examples to perform surprisingly well, and the best models, like mixtral, can often deal more complex categories than other approaches.