Amazon Confronts AI Failures: A Fragile Balance

The Impact of AI Agents on Businesses

Recent disruptions at Amazon, caused by an artificial intelligence tool, illustrate the challenges companies face when integrating these technologies. A notable outage resulted in the loss of nearly 120,000 orders, highlighting the risks associated with a hasty adoption of AI. This situation is not isolated; other companies have also encountered similar issues with AI agents or AI-generated code.

To mitigate these risks, organizations are implementing safeguards and conducting audits. However, the Silicon Valley ethos, which advocates for rapid innovation, can sometimes lead to unintended consequences. Matt Rosenbaum, a senior researcher at the Conference Board, emphasizes the importance of knowing one's risk tolerance and understanding how to respond in case of problems.

The Challenges of Verifying AI-Generated Code

According to Todd Olson, CEO of Pendo, the evolution of software development practices, where developers spend more time reviewing AI-generated code than writing it, presents new challenges. This transition requires different skills and can lead to errors if AI outputs are not carefully verified.

A study conducted by KPMG and the University of Melbourne reveals that two-thirds of workers accept AI results without thorough verification, and 72% admit to reducing their efforts due to AI. These behaviors increase the risk of errors, as noted by Lauren Buitta of Girl Security, who warns against the dangers of speed without analytical discipline.

Lessons Learned from AI Errors

Despite the incidents, mistakes can offer learning opportunities. Todd Olson believes that the Amazon outage likely allowed the company to identify new test cases to improve its systems. Andrew Filev, CEO of Zencoder, argues that these incidents, while minor, are beneficial if addressed internally.

It is crucial to remind employees of the importance of reporting AI-generated errors to prevent major incidents. Filev recommends a combined approach of human and AI audits to achieve reliable autonomy, ensuring that AI review is as rigorous as human review.

The Balance Between Innovation and Safety

Recent incidents involving AI agents highlight the delicate balance employers must navigate. An error related to an AI tool was a major factor in an outage at Amazon, underscoring the risks associated with rapid technology adoption. The tech giant is not alone in facing issues with AI agents or AI-generated code.

Organizations are implementing safeguards and conducting audits to balance AI experimentation and risks. In the age of AI, the Silicon Valley ethic of "move fast and break things" is proving to be literally true.

Earlier this week, Business Insider reported that Amazon had established new safeguards following a series of outages, one primarily caused by its AI coding tool, resulting in nearly 120,000 lost orders. Similar errors have affected other companies during their AI adoption. In January, a founder of an events company reported that an AI agent made four mistakes in a single week, including distributing free tickets. Last summer, the CEO of a browser-based coding platform apologized after an AI agent erased a client's codebase and lied about it.

These incidents underscore a delicate balance for employers eager to leverage AI. If too many restrictions are placed on workers, experimentation suffers. Conversely, if the reins are loosened too much, the risks associated with errant AI agents or poorly evaluated code can quickly multiply.

"You need to know your own risk tolerance," said Matt Rosenbaum, a senior researcher at the Conference Board, a provider of data and insights for business leaders. "You also need to know what to do if things go wrong and what to change to prevent it from happening again."

Speed and Power, Without Control

Part of the challenge lies in the fact that software developers are no longer expected to write as much code as before, said Todd Olson, CEO and co-founder of Pendo, an AI startup that helps companies improve their user experience. Now, a significant portion of developers' work involves reviewing code written by AI, he noted.

"These are very different skills and habits," Olson told Business Insider. Another issue: as AI can generate code in seconds, workers pressed to meet deadlines may be tempted to accept the output as is, increasing the risk that errors go unnoticed.

About two-thirds of workers accepted AI-generated results without carefully verifying them, and 72% reported putting in less effort in their tasks due to AI, according to a global study conducted by KPMG and the University of Melbourne. The findings are based on a survey of over 30,000 workers between November 2024 and January 2025.

"The lesson companies are learning is that speed without large-scale analytical discipline can create systemic exposure," said Lauren Buitta, founder and CEO of Girl Security, a nonprofit that prepares young women for careers in national security.

The uncertainty surrounding the rapidly expanding capabilities of AI adds an additional layer of complexity. As tools become more powerful and accessible, employees may test their limits without fully understanding the downstream consequences.

"Just because you can do something doesn't mean you should do it," said Kevin Serwatka, founder of the recruitment intelligence platform Benchmarket, who previously held leadership roles in recruitment at companies like Google, Meta, and Robinhood. A lesson to be learned from these mistakes, he said, is not to discourage experimentation, "but to put safeguards around what that looks like in your company."

A Positive Aspect

Olson stated that the Amazon outage likely served as a painful lesson for the company. "They probably found a bunch of test cases they can train the AI on, so the AI can review these items in the future," he said.

Other companies using AI to write code are also likely to make mistakes, and this is an integral part of experimentation, said Andrew Filev, founder and CEO of the coding agent company Zencoder. "Small incidents are actually beneficial," he said, although ideally, they are identified and addressed internally rather than exposed to customers. "People will learn and improve their safeguards and systems."

Reminding workers of the importance of reporting any AI-generated errors is crucial, Filev said, as ignoring a problem could lead to an "incident where the blast radius is much larger."

Filev stated that achieving AI autonomy requires starting with a combination of AI and human audits. "You want both processes to work in parallel for a while," he said, until "the AI review is at least as good as human review."