Reddit: 13 Words Are Enough to Deceive AI and Promote Scams

Reddit: A Fertile Ground for AI Scams

On Reddit, it takes just thirteen words to turn a scam into an apparently reliable recommendation by an artificial intelligence. This alarming reality is highlighted by a study conducted by Cornell Tech and 404 Media. The researchers, Tingwei Zhang, Harold Triedman, and Vitaly Shmatikov, published a preprint titled "Deep-Research Agents Can Be Poisoned Via User-Generated Content." They describe an attack method they call WARP, which stands for Web Agent Retrieval Poisoning.

The Influence of User-Generated Content on AIs

AI research agents, such as those used by ChatGPT and Gemini, scour the web to read and compile information from various sources. These sources often include open platforms like Reddit, Wikipedia, Quora, and YouTube, where users can freely post. In their tests, the researchers found that 17 to 23% of the analyzed pages came from these participatory platforms.

A popular discussion on Reddit can influence several similar queries without requiring specific permission. By inserting about fifteen promotional words into a visible source, the researchers managed to make a fake product appear in 38 to 51% of AI responses. By multiplying the bait, this influence could reach 62%, illustrating the vulnerability of current systems.

Ethical and Controlled Experiments

To conduct their experiments ethically, the researchers chose not to publish anything publicly. They worked in a simulated and controlled environment. For example, a fictitious restaurant named Sol Azteca was highlighted after a post praised its authentic local cuisine. Other examples included SilverPath, targeting senior divorcees, as well as a fake cryptocurrency and a service named Xfinity.

The study primarily targeted open-source agents such as STORM, Co-STORM, and OmniThink, without attacking commercial chatbots to avoid disrupting the public web. Gemini Deep Research cited about 12% of the manipulated sources, while OpenAI Deep Research cited only 0.4%, highlighting a potential weakness.

The Need to Verify AI Recommendations

This flaw underscores the danger of AI recommendations, especially in areas where users delegate their judgment, such as apps, restaurants, or purchases. Malicious content can easily mislead the AI, which then confuses linguistic proximity with credibility. According to Zhang, a comment on Reddit can carry as much weight as a government website, which is concerning.

Restricting access to participatory sites could impoverish the richness of AI responses, thereby diminishing their usefulness. Verifying each source or analyzing the final response could also degrade the results. Sometimes, detecting unnatural text fails because the bait is designed to appear fluid and natural.

Reddit claims to have been fighting spam, bots, and manipulation for two decades and sometimes requires human verification. However, neither Reddit nor Wikipedia can solve this problem alone, according to the researchers from Cornell.

It is therefore crucial to treat AI recommendations as suggestions, not as absolute truths. It is advisable to:

• Click on citations to verify their authenticity
• Be wary of unknown names
• Exercise caution with urgent requests requiring payment

In summary, users must adopt a critical approach to AI recommendations, verifying and reading carefully before clicking. While AI can be a valuable tool, it is essential to remain vigilant regarding the information it provides.