Raffi Krikorian Warns of Tesla's FSD Dangers After Crash

⚡

Key Takeaways

1Raffi Krikorian, former head of Uber, was involved in an accident while in FSD mode with his Tesla, highlighting the risks of overconfidence in AI.

2Tesla is under investigation for incidents related to FSD, including driving errors and fatal accidents.

3Ford and Uber have also faced similar issues with their driver assistance technologies.

💡Why it matters — Excessive trust in autonomous driving systems can lead to serious accidents, raising questions about the safety and regulation of these technologies.

Raffi Krikorian and His Experience with Tesla's FSD

Raffi Krikorian, who led Uber's autonomous driving program from 2015 to 2017, recently shared a significant personal experience regarding artificial intelligence systems. In an article published in The Atlantic, he revealed that his Tesla was severely damaged in an accident that occurred while he was using the Full Self-Driving (FSD) mode last year.

Krikorian, well aware of the potential dangers of entrusting control to AI, explained that the impressive performance of the technology made it difficult not to trust it. The accident involving his Tesla Model X, which crashed into a wall, served as a lesson on the risks of over-reliance on AI. In his essay, he describes how the vehicle suddenly made unexpected movements, forcing him to take control without understanding what the system had attempted to do.

A Personal Warning

The accident caused Krikorian a concussion, although no one else was injured. The incident occurred while he was heading to a Boy Scouts meeting in San Francisco. Despite a request for comment, Tesla did not respond. Krikorian, now Chief Technology Officer at Mozilla, emphasized how easy it is to mentally disconnect when systems seem almost perfect.

He described the difficulty of instantly switching from the role of passenger to that of driver, a challenge compounded by months of accumulated trust in the system. Modern systems like Tesla's FSD are designed to handle most driving situations, but they are not infallible. Drivers must remain vigilant and ready to intervene at any moment.

The Dangers of Nearly Perfect Systems

Krikorian highlighted the danger of machines that operate almost perfectly. Unlike an infallible machine that requires no supervision, a nearly perfect machine can create a false sense of security. This excessive trust can have disastrous consequences.

History of FSD-Related Incidents

The errors made by Tesla's FSD system are well documented. The company is facing regulatory investigations following incidents where vehicles have driven the wrong way, run red lights, or attempted to cross train tracks inappropriately.

Tesla has also been involved in several lawsuits following fatal accidents related to FSD and its predecessor, Autopilot. In 2019, a tragic accident claimed the life of a 22-year-old woman, resulting in a judgment of over $242 million in damages against Tesla.

Misleading Advertising and Other Manufacturers in Trouble

Tesla has been criticized for its promotion of FSD and Autopilot, with a California judge ruling that Tesla's advertisements gave the impression that the cars could drive themselves. However, Tesla is not the only manufacturer facing challenges with its driver assistance systems.

Ford, for example, is under investigation for two fatal accidents in 2024 involving its BlueCruise technology. Documents from the National Transportation Safety Board (NTSB) revealed that drivers were distracted before the accidents. Uber has also faced difficulties with its autonomous driving technology. In 2018, an Uber test vehicle struck and killed a pedestrian in Arizona, with an NTSB investigation pointing to inadequate oversight of the safety operator.

Uber ultimately abandoned its plans to develop self-driving cars in-house in 2020, after being criticized for its insufficient safety culture.