Fixed Embeddings and Random Forest: 82% on AG News with Qwen

On a subset of AG News, fixed embeddings extracted from a Qwen2.5-1.5B LLM and a Random Forest achieve 82.0% accuracy. The majority of the time is spent on vector extraction (96.5 s), with training taking only 2.866 s. A fine-tuning via LoRA is being considered to compare performances, with no results communicated at this stage.

82.0% Accuracy and Measured Times of 96.5 s and 2.866 s

The strategy based on fixed embeddings achieves 82.0% accuracy. The feature extraction time reaches 96.5 seconds, while training the classifier takes only 2.866 seconds. The computational load is therefore concentrated on extraction, with downstream model learning remaining brief. Evaluations are conducted on 800 training examples and 200 test examples, both shuffled with a seed of 42.

A 4-bit Quantized LLM Qwen2.5-1.5B and a 4-Class Task

The experiment relies on Qwen/Qwen2.5-1.5B configured for four-label classification. The model is loaded with 4-bit quantization of type nf4 and bfloat16 computations, relying on an auto device_map. The model's pad_token_id is aligned with that of the tokenizer, which is initialized from the same identifier. The end-of-sequence token is used as a padding token, which is indicated as necessary for decoder-only architectures. The texts come from the ag_news dataset, which targets four categories of articles.

Extraction by Masked Averaging of the Last Hidden State and Random Forest

The LLM is used as a feature extractor, and its weights remain fixed. The embeddings are produced in batches of size 8 in evaluation mode, with tokenization that activates padding and truncation up to 128 tokens. Inference is done without gradients, requesting the hidden states from the model. The aggregation retains the last hidden state and calculates a weighted average using the attention mask, with a clamp of 1e-9 on the denominator for numerical stability, before converting to float32 vectors on CPU. On these representations, a RandomForestClassifier with 100 estimators and a seed of 42 is trained. The embeddings and labels are extracted separately for training and testing, and the extraction and learning times are measured independently.

Fine-Tuning via LoRA Considered, Cost/Performance Comparison in View

The next step of the protocol plans to test fine-tuning of the LLM with LoRA, a low-rank adaptation method to efficiently update weights, to examine whether the 82% accuracy can be surpassed. The recommended environment on Colab indicates the use of a T4 GPU and the installation of bitsandbytes, peft, accelerate, and evaluate, while in other Python environments, scikit-learn, transformers, and datasets are required. The stated goal is to shed light on the choice between fixed embeddings and fine-tuning in terms of costs and performance, particularly for use cases with limited budget and resources.