Scikit-LLM and Ollama: Explaining Text Embeddings

A tutorial details how to generate embeddings from 1,000 IMDB reviews using a local all-minilm model and submit them for diagnosis via logistic regression. The performance reaches an accuracy of 0.77, while UMAP and SHAP are employed to examine the semantic structure and the influence of latent dimensions.

A linear model achieves 0.77 accuracy on 200 tests

On the test sample of 200 reviews, the accuracy reaches 0.77. For class 0, precision is 0.77, recall is 0.76, and F1 is 0.76, with a support of 100. For class 1, precision is 0.76, recall is 0.77, and F1 is 0.77, with a support of 100. The macro averages indicate a precision of 0.77, a recall of 0.77, and an F1 of 0.76, for a total support of 200. The weighted averages display the same levels of precision and recall (0.77) and an F1 of 0.76. These results are presented as respectable for logistic regression, considering the limited dataset size relative to the dimension of the embeddings.

UMAP and SHAP employed for embedding interpretation

UMAP visualization is used to inspect the semantic structure captured by the embeddings. UMAP is described as a projection-based dimensionality reduction technique, commonly used for visualization, with a 2-dimensional projection here and the use of cosine similarity as the distance metric, which is standard for text embeddings. The application of SHAP values is intended to identify the latent dimensions that most influence the predictions of a classifier.

Assessing the quality of vectors via an exploratory classifier

The exploratory classifier serves as a diagnostic tool to inspect the learned representations. The quality of the embeddings in separating positive and negative reviews is evaluated using a simple model such as logistic regression and standard metrics. When precision, recall, and F1 are satisfactory with a shallow model, it indicates that the vectors are rich enough for the task. The use of a simple model helps isolate the contribution of the embeddings. The tutorial thus trains a LogisticRegression with a random_state of 42 and a max_iter of 1000.

Balanced IMDB dataset and stratified 80/20 split

The corpus used is the public IMDB movie reviews dataset. One thousand texts are selected, evenly split between 500 positive and 500 negative to create a balanced sample, with labels 1 for positives and 0 for negatives. The preparation involves concatenating the two subsets, shuffling them, and then extracting the texts and labels. The split between training and testing is stratified, with 80% of the examples for training and 20% for evaluation, a test_size of 0.2, and a random_state of 42.

Free local chain: Ollama and Scikit-LLM for 1,000 embeddings

The tutorial is executable in Google Colab and requires the latest version of Scikit-LLM. It offers a local and free setup, with installation via pip for scikit-llm, umap-learn, and shap. In Colab, an installation of zstd via apt-get may be necessary, to be handled with caution depending on the environment. Ollama is installed via script and launched in the background, with a wait of 5 seconds, after which the all-minilm embedding model is retrieved. The libraries numpy, pandas, matplotlib.pyplot, umap, and shap are imported, along with SKLLMConfig and GPTVectorizer, and scikit-learn utilities for training and evaluation, plus load_dataset. Scikit-LLM is configured on a local server at the URL http://localhost:11434/v1/, with a dummy OpenAI key used for formatting but ignored locally. The embeddings for 1,000 texts are produced with GPTVectorizer and the all-minilm model, using an API similar to scikit-learn (fit_transform for training, transform for testing). In Colab, the generation may take about 5 to 10 minutes as it involves 1,000 calls to a local LLM. The entire setup serves to generate embeddings from movie reviews with Scikit-LLM and a local Ollama model, then to train a logistic regression classifier to evaluate their quality, combining exploratory classifiers, UMAP, and SHAP to interpret the textual representation produced by LLMs, in a context where they are powerful yet inherently less interpretable, and where text classification has long relied on machine learning and deep networks.