Lightweight LLMs and Tabular Data: A Unified Scikit-learn Pipeline

A Scikit-learn pipeline can aggregate textual embeddings from a lightweight open-source model and tabular variables within the same flow. The detailed method relies on sentence-transformers, ColumnTransformer, and a Random Forest classifier. It illustrates the approach with a real SMS dataset augmented with synthetic variables, without resorting to paid APIs or giant LLMs.

Three Parallel Branches and a Random Forest to Conclude

The pipeline assembles three processing branches executed in parallel, one of which is for textual embedding. A ColumnTransformer orchestrates these branches and maintains the integrity of the dataset without manual splitting or recombination. Numeric columns undergo standardization, while categorical ones are processed through one-hot encoding. The chain concludes with a RandomForestClassifier. The embedding model chosen in the example is all-MiniLM-L6-v2.

A Real SMS Dataset Enriched with Synthetic Variables

The demonstration relies on the SMS Spam Collection dataset, loaded from a public URL and read in tab-separated format without a header, containing the columns label and message. The target is encoded as 0 for ham and 1 for spam. The generation of synthetic variables is made reproducible by a seed of 42. Account age is simulated between 1 and 365 days for the spam class and between 1 and 1500 days for the ham class. The is_premium status follows distinct probabilities based on the class: for spam, 0.95 for no and 0.05 for yes; for ham, 0.80 for no and 0.20 for yes. A priority score is drawn uniformly between 0.4 and 1.0 for spam, and between 0.0 and 0.7 for ham. These distributions intentionally introduce noise and significant overlap to avoid unrealistic learning. A preview of the first three rows allows for checking overall consistency. This data setup serves as the basis for a scenario of unsubscribing or customer triage.

A Text Transformer Designed for Scikit-learn

The textual embedding component is a class TextEmbedder that inherits from TransformerMixin and BaseEstimator. Its constructor accepts a model parameter initialized to all-MiniLM-L6-v2. The fit method instantiates a SentenceTransformer if necessary to comply with Scikit-learn's cloning rules. The transform method handles pandas DataFrames by extracting the first column as strings or converting the input to Series if applicable. It returns a 2D array of embeddings produced by model.encode with the progress bar disabled. This encapsulation makes the generation of embeddings compatible with existing Scikit-learn pipelines.

Unified Orchestration of Text and Tabular Data

The ColumnTransformer allows for the coexistence of textual embeddings and preprocessing dedicated to numeric and categorical variables within the same flow, orchestrating parallel branches. The operations rely solely on standard classes from the library, facilitating the assembly and evaluation of a deployment-ready pipeline. The presentation includes a diagram to visualize the unified architecture and synchronization of the branches.

Use Cases and Technical Choices Without Paid APIs

Tasks such as ticket triage or churn prediction require the integration of text and tabular data within the same model. The approach implements a unified pipeline that processes each data type according to its needs while keeping the overall structure coherent. The illustrated case targets the detection of spam users, but the method generalizes to other contexts where textual messages or descriptions complement structured attributes. The technical choices favor lightweight open-source LLMs, avoiding paid APIs and massive models like LLaMA 3. The installation can rely on sentence-transformers, scikit-learn, pandas, and numpy via a simple pip command. In a local IDE, the notebook-specific prefix should be omitted. The central goal is to combine textual embeddings and tabular variables within a single classification pipeline.