Scikit-LLM Revolutionizes Multi-Label Classification with LLMs

Scikit-LLM: A Breakthrough in Multi-Label Classification

Introduction

Text classification is often viewed as a binary task, where reviews are categorized as "positive" or "negative," or where requests are classified into distinct categories. However, human emotions are rarely so simple to describe. A single sentence can express multiple emotions, such as joy and anger simultaneously. Take, for example: "I absolutely love the improved battery life, but the new design is incredibly awful." Here, multi-label classification comes into play, allowing multiple categories to be assigned to the same text.

Creating such classifiers typically requires large volumes of labeled data and complex neural network architectures. However, large language models (LLMs) offer an alternative solution due to their reasoning capabilities, particularly in zero-shot scenarios. With libraries like scikit-LLM, it is now possible to integrate these models into a traditional machine learning workflow, such as scikit-learn. This article explores how to tackle a multi-label sentiment classification problem using an open-source dataset.

Step-by-Step Guide

Scikit-LLM stands out for its ability to simplify the use of LLMs for inference, without requiring intensive training. It also allows the use of free and open-source LLMs without quota restrictions. We will illustrate how to load, adapt, and leverage a pre-trained LLM for a multi-label classification task.

To begin, you need to import the required libraries:

pip install scikit-llm datasets

We will use a free LLM from Groq, known for its fast inference capabilities. After signing up on their website, you will receive an API key. This key, once generated, should be copied and inserted into the following code:

from skllm.config import SKLLMConfig
from skllm.models.gpt.classification.zero_shot import MultiLabelZeroShotGPTClassifier

# 1. Define your API key (use "any_string" if local)
SKLLMConfig.set_openai_key("YOUR_FREE_API_KEY")

# 2. Set the custom endpoint URL
SKLLMConfig.set_gpt_url("https://api.groq.com/[openai](/dossier/openai)/v1/")

# 3. Initialize the classifier.
clf = MultiLabelZeroShotGPTClassifier(model="custom_url::[llama](/dossier/meta-ia)-3.3-70b-versatile", max_labels=3)

Here, we have created an object of the MultiLabelZeroShotGPTClassifier class to host our pre-trained model from Groq.

Next, we import a dataset. Hugging Face offers a wide range of datasets, and we will specifically use the go_emotions dataset, which is ideal for our task. Depending on your environment, a Hugging Face API key may be required, which can be easily obtained by signing up on their site.

from datasets import load_dataset
import pandas as pd

dataset = load_dataset("google-research-datasets/go_emotions", split="train[:100]")
df = dataset.to_pandas()

texts = df['text'].tolist()
print(f"Loaded {len(texts)} comments.")
print(f"Example: '{texts[0]}'")

You will see an output indicating an example from the loaded dataset:

Loaded 100 comments.
Example: 'My favorite dish is anything I didn't have to cook myself.'

To "train" the LLM, we simply need to define our specific domain label set. The model then adapts to classify the texts using these labels. Here is the set of labels we will use:

candidate_labels = [
    "admiration", "amusement", "anger", "annoyance",
    "approval", "curiosity", "disappointment", "joy",
    "sadness", "surprise"
]

This is not traditional training: we are merely exposing the model to our label set to define the problem. Here’s how to proceed:

clf.fit(None, [candidate_labels])

After completing these steps, you are ready to make predictions on a few text examples. Let’s try with five texts from the dataset:

predictions = clf.predict(texts)

for i in range(5):
    print(f"Comment: {texts[i]}")
    print(f"Predicted Sentiments: {predictions[i]}")
    print("-" * 50)

Results

Here is an excerpt of the results — two of the five predictions are shown:

Comment: My favorite dish is anything I didn't have to cook myself.
Predicted Sentiments: ['amusement' 'joy' '']
--------------------------------------------------
Comment: Now, if he commits suicide, everyone will think he's joking by mocking people instead of actually being dead.
Predicted Sentiments: ['anger' 'annoyance' 'surprise']

Note that multiple labels can be assigned to a single text during prediction.

Do not be surprised if the prediction process takes time. This is normal, as using these LLMs locally is resource-intensive. Inference is often slower than model fitting, as no actual training is performed, only a scenario definition with the labels.

Conclusion

This article has demonstrated how to perform multi-label text classification with scikit-LLM, leveraging the capabilities of pre-trained LLMs as traditional machine learning models with scikit-learn.

To go further, you could expand the label set to better cover the emotional spectrum of your domain or test other models hosted by Groq to compare predictions. Scikit-LLM also supports other zero-shot and few-shot classification strategies, allowing for refined predictions with a small number of labeled examples. Finally, for production use, it is crucial to establish an evaluation loop to measure precision and recall, in order to identify the model's strengths and weaknesses.