AI Redefines the Data Stack: Outdated Architecture No Longer Fits

The Old Data Stack Facing the Challenges of AI

Traditional data architectures, centered around warehouses like Redshift or BigQuery, are struggling to meet the new demands of AI. These systems, designed to handle structured data with simple SQL queries, find it difficult to address the complex needs of LLM pipelines. The latter require the management of unstructured data, real-time computations, and complete traceability—tasks that the old stack cannot handle efficiently.

The traditional stack, often powered by Airflow DAGs, was initially designed for a simple workflow: structured data comes in, SQL queries go out, and BI dashboards get updated. However, AI fundamentally alters this contract, necessitating major adjustments. For instance, an LLM pipeline may require raw, unstructured text in S3 alongside clean relational data, embeddings calculated from 50 million-row tables in real-time, and feature vectors generated in milliseconds for a model serving endpoint.

None of these elements fit neatly into the old paradigm. The warehouse assumes structure, while the data lake has structure but lacks transactions. Neither was designed to serve a vector database with 10 ms latency. The Lakehouse architecture, integrating solutions like Delta Lake, Apache Iceberg, or Apache Hudi, offers increased flexibility while maintaining transactional integrity, thus better meeting current needs.

The Importance of the Platinum Level in the Medallion Architecture

The Medallion architecture, often simplified to three levels (Bronze, Silver, Gold), frequently overlooks a crucial fourth level: the Platinum or AI-native level. This level is essential for preparing data for AI, organizing and tracking embeddings and fine-tuning datasets. Ignoring this step from the outset complicates later integration into structures like Iceberg.

Embeddings need to be calculated and stored quickly, and fine-tuning datasets must be organized, versioned, and tracked. Feature vectors for real-time ML models need to be pre-materialized to avoid computations at query time. One thing I want to emphasize here: many teams treat this fourth level as optional, something to add later. In practice, if you don't design it from the start, adding it to an existing Iceberg table structure is genuinely painful. You will end up with embedding tables that are not linked to their source rows in the traceability graph, making auditing nearly impossible.

End of the Lambda Debate: A New Unified Approach

The debate between Lambda and Kappa architectures has long divided data engineers. Today, a unified approach using Iceberg as a central point seems to be emerging. This method allows for the simultaneous management of real-time and batch processing, ensuring safe ACID level writes and providing a consistent view to downstream consumers.

Latency SLOs must be explicitly stated. The real-time path must meet latency requirements measured in seconds, while the batch path is measured in hours. Both pipelines require different monitoring, different alerts, and often different teams to own them. I have seen organizations merge these two SLOs into a single call rotation, and it ends badly.

Data Governance: A Fundamental Necessity

Data governance is often relegated to the background, perceived as a regulatory obligation. Yet, it is crucial to avoid costly mistakes, such as the accidental ingestion of PII. Column-level traceability, data contracts, and automation of PII classification are essential elements for effective and secure data management.

OpenLineage is the open standard for traceability, and tools like Airflow and dbt can emit traceability events that feed into tools like Marquez or DataHub. Data contracts are the real unlock. The idea is simple enough: before a team of producers changes the schema or SLA of a table, they must negotiate this change with all downstream consumers. In practice, this means defining a YAML contract (Avro schemas work well here) that specifies column names, types, nullability guarantees, and the expected freshness SLA. Break the contract, break the build. Tools like Soda Core or custom Great Expectations suites can enforce this at ingestion time.

PII classification was once manual. Now, it no longer needs to be. An LLM tagging pipeline, yes, using an LLM to govern data that could be fed into an LLM, can scan new tables, identify columns that resemble names, emails, phone numbers, or health identifiers, and automatically apply the relevant governance tags. These tags then propagate downstream to trigger column-level masking policies in Unity Catalog or whichever metastore you are using.

The AI Layer: Challenges and Solutions

Integrating an LLM into a data platform goes far beyond simple installation. Models like Text-to-SQL and RAG pipelines require strict governance to prevent leaks of sensitive data. Tools like Vanna.AI and DSPY from Stanford enhance SQL generation in production. Text-to-SQL is deceptively tricky. Demonstrations are always impressive; you type "what was our revenue by country last quarter," and it outputs beautiful SQL. But in production, the quality of SQL generation degrades sharply when your schema is wide, when column names are ambiguous (is status the order status or the payment status?), or when the query requires joining more than three or four tables. Tools like Vanna.AI tackle this with a feedback loop, storing validated query-SQL pairs as examples with few examples. DSPY from Stanford takes a more systematic approach to prompt chain optimization using a small labeled dataset. Neither is perfect, but both are significantly more reliable than a simple prompt "here's my schema, generate SQL."

RAG pipelines also need governance. When your RAG system retrieves snippets from an internal knowledge base, it effectively bypasses all the row-level security you have integrated into your lakehouse. A user who should not be able to see customer PII can, in theory, ask a question that leads the RAG system to retrieve and display that information. Properly resolving this requires filtering your vector search results by the same access policies you apply to your structured data, meaning your embedding pipeline must store metadata with each vector indicating which roles are allowed to see it.

The feature store is still underutilized. Most teams I speak with have at least heard of feature stores but have not built one properly. The value proposition is as follows: you compute a feature, for example, a user's purchase frequency over 30 days, once, store it, and serve it both to online model serving endpoints (with millisecond latency via Redis) and to offline training pipelines (from the lakehouse). Without a feature store, this calculation is duplicated, often subtly differently, between your training pipeline and your serving pipeline. This inconsistency is one of the most common sources of performance degradation for models in production.

Observability: The Part That Decides Whether Your Platform Survives

Data observability has become its own category. Tools like Monte Carlo, Acceldata, and Metaplane essentially perform health checks on your pipelines and alert you when something silently breaks. And "silently breaking" is the killer. A pipeline that fails is annoying. A pipeline that runs correctly but produces subtly incorrect numbers for three weeks is catastrophic.

The minimum viable observability setup for a production platform:

• Volume checks: if a table receives 10% of the normal daily row count, trigger an alert.

• Freshness checks: if a Gold table has not been updated within 6 hours of its SLA, trigger an alert.

• Distribution checks: if the average of a numeric column changes by more than two standard deviations from its 30-day moving average, report it.

• Schema drift detection: if a source...