The Crucial Choices of AI Engineers Often Overlooked

Build or Buy: A Persistent Dilemma for AI Engineers

With the rapid evolution of large language models (LLMs), artificial intelligence engineers face a complex and crucial choice: should they use an API, fine-tune an open-source model, or develop their own infrastructure? This question, once straightforward, took on new dimensions in 2026. According to a survey conducted by Omdia in 2025, which polled 376 technical and business stakeholders, 95% of respondents believe that building their own model offers more customization and control. However, 91% of the same respondents acknowledge that pre-built platforms allow for faster deployment. Both statements, while true, present a difficult dilemma to resolve.

In practical terms, the choice largely depends on scale. For companies handling fewer than 100,000 daily requests, calling an API like GPT-4o Mini is generally the best option. This allows for low overhead and rapid iteration. However, when the volume of requests exceeds one million per day, token costs begin to weigh heavily on margins.

A 2024 analysis highlighted an often-overlooked aspect: personnel accounts for 70 to 80% of self-hosting costs, while hardware and electricity only make up 20 to 30%. Teams tend to underestimate these costs, frequently leading to budget overruns of around 340%. These overruns are typically due to a lack of tenant usage tracking and the absence of cost attribution at the request level, rather than the token rate itself.

Framework lock-in is another issue that manifests later. For example, Hugging Face's text generation inference went into maintenance mode at the end of 2025, forcing teams that had relied on this technology to migrate to other solutions. In contrast, teams using an API did not face this type of forced migration.

Model Complexity and Maintainability: A Delicate Balance

AI engineers must also juggle model complexity and maintainability. A well-known Google paper introduced the CACE principle: Changing anything changes everything. In machine learning systems, a small adjustment in one part of the pipeline can lead to surprising changes elsewhere. This rarely happens with linear regression but is common with ensembles and neural networks.

Research on technical debt in machine learning shows that data dependency is more costly than code dependency. Why? Because data is harder to track, version, and explain to anyone inheriting the system six months down the line. The original paper estimated that the actual model code is only a small fraction of a real ML system. The essence lies in feature stores, pipeline logic, monitoring, retraining triggers, and the connections between all these elements.

In practice, teams often choose a more complex model for a 2% gain in accuracy but pay for that choice for 18 months in debugging time, retraining overhead, and the tax of "no one remembers why we did this." The question to ask before deploying a complex model is: who will be responsible for it in a year? If the honest answer is "uncertain," that's the decision point.

Quantity of Data vs. Quality of Data: The Data Swamp Trap

In applied machine learning, more data does not always mean better performance. Research shows that beyond a certain noise threshold, adding more low-quality data flattens or degrades model performance. This means that the relationship between sample size and accuracy deteriorates once the noise exceeds a certain level.

The "data swamp" problem is what this looks like in companies. Teams collect everything because storage is cheap and they assume it will be useful someday. Without governance, you end up with a dataset that takes weeks to clean, increases storage and pipeline costs, and slows down experimentation without improving outcomes.

Medical AI is the clearest case. Small datasets with expert-verified labels have consistently outperformed larger datasets with unreliable annotations. The model learned the right patterns from less data because the signal was clear.

The question I find more useful in practice is: what is the noise level of what we have, and what does an additional hour of cleaning bring us compared to an extra day of collection?

Throughput vs. Latency: Choosing Between Batch and Real-Time

Batch inference and real-time inference are two different system architectures. Choosing the wrong one leads to infrastructure, cost, and user experience decisions that are difficult to reverse later.

Batch inference involves generating predictions on a schedule (hourly, daily), which are then stored in a database and served from there. This method is less expensive, with simpler infrastructure that is easier to debug. However, predictions may become outdated.

In contrast, real-time inference generates predictions on demand, in milliseconds to a few seconds. While always current, this method is more costly due to the need for 24/7 availability. It also has more moving parts and is harder to monitor.

The system-level tension is that larger batch sizes yield higher throughput but higher latency per request. Real-time systems use a batch size of 1, which provides speed but may lose efficiency.

The mistake I see most often is that teams default to real-time because it seems more impressive. But most business problems do not require predictions in under a second. Nightly churn scores, weekly recommendation refreshes, daily fraud model updates—these are batch problems that are over-engineered as real-time issues, and the cost difference at scale is significant.

The practical signal is that if your users do not notice whether the prediction is 5 minutes or 5 milliseconds late, use batch inference instead of real-time.

Prompt Engineering vs. Fine-Tuning: Two Distinct Approaches

The decision logic here has become clearer in recent months. Prompt engineering is quick, inexpensive, and flexible. It can take hours to days to iterate and works well for most tasks, especially with capable state-of-the-art models.

The downside is fragility, as small input changes produce inconsistent outputs, and long prompts with complex formatting rules tend to fail in edge cases.

Fine-tuning is costly upfront in computation, data preparation, and engineering time. It is reliable and consistent at scale once the work is done. A real example I have seen cited: fine-tuning GPT-4o for a customer support chatbot cost about $10,000 in computation and 6 weeks of data preparation. The RAG alternative was delivered in 2 weeks.

My view on current advice from practitioners is to start with prompts. Move to fine-tuning only when you hit failure modes that prompts cannot resolve. Below 100,000 requests, prompts are almost always the right choice. Fine-tuning has been shown to be cost-effective at high volume when the task is stable and well-defined.

A 2025 analysis revealed that optimizing prompts with tools like DSPy outperformed fine-tuning by 6 to 19 points on certain benchmarks, using 35 times fewer deployments. It seems the gap is narrowing year by year. Fine-tuning has become a final step in most stacks I see, used after prompts have clearly hit their ceiling.

The hybrid model is increasingly common in production: a fine-tuned model on domain style and tone, combined with RAG for factual anchoring. Both techniques solve different problems.

Automation vs. Human Oversight: Where to Draw the Line?

How much do you trust the model to act on its own? The useful question in production is: what is the cost of a wrong decision, and who absorbs it?

Human-in-the-loop (HITL) exists on a spectrum. At one end, humans review every AI output before it acts. At the other, complete automation with humans only monitoring anomalies.

Most production systems fall somewhere in between, routing low-confidence predictions to humans and passing through high-confidence ones. But the operational cost of HITL is real: reviewing every model decision does not scale.

The truth is that real-time human intervention slows the system down, and the inconsistency of reviewers degrades label quality. The working model is selective HITL: human review is triggered only for edge cases, low-confidence outputs, and high-stakes decisions.

In fields like healthcare, finance, and law, HITL is often a compliance requirement. A radiologist reviewing tumors flagged by AI or a lawyer examining contract clauses flagged by AI. These are cases where the cost of an error is too high to be fully automated.

One way to think about the division is that AI handles volume, speed, and pattern recognition, while humans manage irreversibility. The design question is exactly where that line should be drawn.