Satya Nadella: AI Models, Hidden Trojan Horses

Satya Nadella: AI Models, Hidden Trojan Horses

Among all the debates about the potential downsides of AI, one concern raises the most alarm among AI enthusiasts in Silicon Valley. Their fear is that the major AI labs selling proprietary models act as Trojan horses.

The worry is that as startups and companies use AI models from labs like OpenAI and Anthropic, these labs gain increasingly significant access to the most sensitive business information of these companies. The model creators can then use this knowledge to their own advantage, potentially becoming competitors to their own clients. Those issuing such warnings range from venture capitalists like Jason Calacanis to Palantir CEO Alex Karp.

In a surprising blog post published on Sunday, Microsoft CEO Satya Nadella joined this group. Nadella warns that AI users (the "buyers," as he calls them) are paying twice. They consciously spend money on AI token usage, but they also unknowingly transmit valuable data in the process.

"You are essentially paying for intelligence twice, once with money, and again with something even more valuable: the proprietary knowledge you must reveal to make that intelligence useful. The more you want the model to perform well, the more of that knowledge you have to provide!" he writes.

More dangerously, companies are literally teaching the models the nuances of their operations, he argues.

"The models learn from the 'exhaust,' the prompts that people write, the tools that agents use, and especially the corrections that people make when the model gets it wrong. Every correction is distilled into institutional knowledge," he writes.

This is "the kind of knowledge that a competitor could never buy," and yet companies are passing it along.

Nadella argues that while AI companies can freely scour the internet to train their models, it is only fair that businesses can study—or "distill"—these models in return. "Distillation" is the practice of using a model's own outputs to learn how it works and to train a new, often less costly model based on that information. In February, Anthropic accused Chinese open-source models of sending millions of prompts to Claude to improve their own models and urged the U.S. government to crack down on export controls.

Nadella's point is that model creators cannot have their cake and eat it too. It is hypocritical for them to train freely on the world's data while restricting others from doing the same with their models.

"While the great innovation that arises from fair use rights of model providers to train models on public data is necessary, I find it ironic that the status quo is to turn around and impose restrictive conditions on distillation," Nadella writes.

Nadella is particularly concerned when model creators "reserve the right to learn from customer usage and interaction data."

Nadella's solution is the kind of thing a CEO of a major cloud provider might suggest. He wants companies to "retain ownership" of their data, including prompts, feedback, etc. He urges them to build their own "proprietary learning environments" in the cloud (where their data is likely already stored anyway, and conveniently, this could mean Microsoft’s cloud, Azure). He also wants companies to integrate what he calls "orchestration layers"—essentially, a way to easily switch from one AI model to another provider rather than being locked into a single one. Tools like AI "gateways" that allow companies to do just that have become increasingly popular.

Although Nadella never uses the words "open source" as a method for retaining ownership, it is an obvious subtext. Yet, there is another subtext.

Large companies, many of which still have some of their own data centers in addition to using the cloud, are already turning to open-source models installed on their premises ("on-prem," in industry jargon). Idit Levine, founder and CEO of Solo.io—who makes networking and security software that helps companies manage AI systems—says she is seeing exactly this shift happen with her own clients. After experimenting with proprietary model creators, they are starting to ask, "Can I take an open-source model and run it on-site? It will do almost 90% of what the large model does. It will cost a lot less," she tells TechCrunch. "They understand this, and they can control it."

Solo.io's technology was selected last year to power the Linux Foundation's Agentgateway project. Her company has clients such as T-Mobile, ADP, and SAP. She sees that companies are increasingly installing open-source models on-site and considers this the next big wave in enterprise AI usage.

She is not alone. Vercel (best known as a website creation and hosting platform, which recently added AI model-switching tools) and OpenRouter (a company that helps developers route requests across different AI models) are both seeing an increase in traffic to open-source models. In fact, open-source models accounted for 29% of all traffic routed through Vercel's gateway last month.

With the CEO of Microsoft, a company that has invested in both OpenAI and Anthropic, now openly urging companies to be wary of using proprietary models, we bet this trend will continue to grow. "By consuming intelligence, you create intelligence. And what you create should belong to you," writes Nadella.