LLM and Feature Engineering: A Revolution in Python

Feature Engineering with LLMs: Techniques and Examples in Python

What is Feature Engineering with LLMs?

Feature engineering with LLMs (Large Language Models) represents a significant advancement in the field of machine learning. By leveraging these large language models, engineers can transform raw data into structured input features, thereby optimizing the performance of learning systems. Unlike traditional methods that rely on manual transformations, LLMs allow for the extraction of semantic and structured signals, enriching models with contextual information.

This new approach enables engineers to develop machine learning models through various methods, including both numerical transformations and context-based representations. Pre-trained language models are used to convert raw inputs into high-dimensional structured representations, which helps models achieve better performance. The models utilize context to determine relationships between elements while creating features that express meaning beyond simple statistical patterns.

How Does This Differ from Traditional Feature Engineering?

Traditional feature engineering relies on rules and manual transformations to create features. In contrast, LLMs capture the meaning and intentions of users, as well as the relationships between data, often missed by manual encoding. Traditional methods, such as TF-IDF, treat words as separate entities, thus losing relationships and emotional meanings. LLMs, on the other hand, use their training on vast textual databases to understand linguistic context and extract semantic features.

The Transition: From Manual Features to Semantic Features

Machine learning develops models through the use of handcrafted features, which include one-hot vectors and standardized numerical values. Handcrafted features have limitations as they do not account for context and require specialized knowledge, while failing to manage subtle differences. The TF-IDF method treats words as separate entities, leading to a loss of relationships between words and their emotional significance.

The limitations of traditional methods include the need for constant connections to the system and domain-specific expertise. The system fails to incorporate both general knowledge and complex connections. A bag-of-words model requires more knowledge than simply "cold food" to recognize negative sentiments. Human resources must spend considerable time identifying all exceptional situations.

LLMs operate within their respective contexts by using their training from vast textual databases to acquire knowledge and recognize patterns. The system understands linguistic context through their world knowledge and ability to comprehend hidden messages. The system extracts semantic features from the data via LLMs, which create automatic features identifying data elements such as sentiment, subject, and risk categories.

The importance of this transition lies in its ability to demonstrate that semantic features yield better results than human-created features when it comes to complex tasks. The system requires fewer feature heuristics for its operations, resulting in faster testing processes.

Key Techniques in Feature Engineering with LLMs

This section will illustrate key methods with code examples. We will generate small sample data and show how features are derived.

Embeddings as Features

LLMs produce dense semantic vectors from text. The extracted embeddings function as numerical features that allow the model to understand a meaning that goes beyond simple word frequencies. We can use a transformer model to create 384-dimensional sentence embeddings through sentence encoding.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')
sentences = ["I love machine learning", "The movie was fantastic"]
embeddings = model.encode(sentences)
print("Shape of embeddings:", embeddings.shape)

The output shape (2, 384) shows that two sentences are mapped into dense vectors of 384 dimensions (one for each sentence). The vectors represent semantic properties of the text, including related meanings and emotional expressions.

When to use embeddings vs traditional features:

from sklearn.feature_extraction.text import TfidfVectorizer

docs = [
    "The cat is sitting on the mat",
    "The dog ate the cat",
]

# Traditional TF-IDF: sparse bag of words
tfidf = TfidfVectorizer()
X_tfidf = tfidf.fit_transform(docs)

# LLM embeddings: dense semantic features
X_emb = model.encode(docs)
print("Shape of TF-IDF features:", X_tfidf.shape)
print("Shape of [LLM](/dossier/llm) embeddings features:", X_emb.shape)

The shape of the TF-IDF features creates a sparse matrix of (2×6) containing six unique terms, while the LLM embeddings exist as dense vectors of (2×384). The embeddings convey the meaning of words in their context as they show how synonyms relate to each other, for example, "cat" and "dog." Use semantic features from embeddings, while traditional features work for simple numerical data and high-frequency categorical data requiring sparse encoding.

Prompt-Based Feature Extraction

We can prompt the LLM to extract specific structured information from the text. The model outputs can be analyzed into features.

from transformers import pipeline

extractor = pipeline("text2text-generation", model="google/flan-t5-base")

# Example text
text = "The phone's battery lasts all day and the performance is smooth"
result = extractor(prompt, max_length=50)
print(result[0]["generated_text"])

We use the LLM prompt that states "Extract sentiment (positive/negative), product issue, and performance from this review." The model returns structured features in a dictionary format similar to JSON. Sentiment, subject, and urgency features now exist as separate columns that we can integrate into our classification system.

Schema-Guided Extraction

A JSON schema can be applied during invocation to ensure consistent outputs. For example:

# Extract in JSON format
result = extractor(prompt, max_length=100)
print(result[0]["generated_text"])

Generating Semantic Features

LLMs generate new descriptive attributes that can be applied to both individual rows and individual data values.

{"review": "Great camera quality but the battery drains quickly"},
{"review": "Affordable and durable, good for daily use"}

Generate a new feature called 'user_intent' from this review:

result = extractor(prompt, max_length=50)
print(result[0]["generated_text"])

The LLM extracts the user's intent from the review through its text analysis. The system transforms raw text into structured features that show the user's preference for cameras and their concern about battery life. The system allows users to add new columns that enhance the model's understanding of user activity patterns.

Creating Contextual Features

LLMs can generate textual features when they use their knowledge to analyze the value of a feature in specific situations. The LLM uses postal code information to explain the corresponding geographical area.

result = extractor(prompt, max_length=50)
print(result[0]['generated_text'])

The LLM uses customer review information to determine which customer group the reviewer belongs to. The system transforms the input text into a normalized label that displays the two main user preferences for affordable and durable products. The system allows users to implement a new feature that enables models to categorize users based on their behavioral patterns and specific preferences.

Hybrid Feature Spaces (Multimodal Pipelines)

Multimodal pipelines combine different data sources to enrich the extracted features, providing a more comprehensive and accurate view of the analyzed data.