Moonbounce and AI: Revolutionizing Content Moderation

The Rise of Brett Levenson in Content Moderation

In 2019, Brett Levenson left Apple to join Facebook as the head of business integrity, at a time when the social network was still reeling from the Cambridge Analytica scandal. Levenson initially believed that content moderation issues could be resolved through improved technology. However, he quickly discovered that the problem was far more complex than anticipated.

Human moderators had to refer to a 40-page policy document, often translated automatically, to make decisions in just 30 seconds on each reported piece of content. These decisions included blocking content, banning users, or limiting distribution. According to Levenson, the accuracy of these decisions was barely better than 50%, likening the process to a simple coin toss.

The Challenges of Moderation in the Age of AI Chatbots

This reactive and often delayed approach proved ineffective in an environment where well-organized and funded adversaries proliferated. The rise of AI chatbots only exacerbated the situation, with high-profile incidents where chatbots provided dangerous advice to teenagers or generated inappropriate images bypassing security filters.

In the face of these challenges, Levenson had the idea to transform policy documents into executable logic, an approach he dubbed "policy as code." This idea led to the creation of Moonbounce, which recently announced it raised $12 million in a funding round co-led by Amplify Partners and StepStone Group.

Moonbounce: Advanced Moderation Technology

Moonbounce works with companies to add an extra layer of security every time content is generated, whether by a user or by AI. The company has developed its own language model capable of reviewing a client's policy documents, assessing content in real-time, and providing a response in 300 milliseconds or less. Depending on the client's preferences, this response can involve slowing down content distribution while awaiting human review or immediately blocking high-risk content.

Strategic Partnerships and Broad Reach

Currently, Moonbounce serves three main sectors: user-generated content platforms, such as dating apps, AI companies developing characters or companions, and AI image generators. Moonbounce supports over 40 million daily reviews and has more than 100 million daily active users.

Among Moonbounce's clients are Channel AI, an AI companion startup, Civitai, a company specializing in image and video generation, as well as character role-playing platforms Dippy AI and Moescape.

Security as a Competitive Advantage

Levenson asserts that security can become a product advantage, a concept that has never been fully integrated into products before. He observes that Moonbounce's clients are finding innovative ways to use this technology to make security a differentiating factor and an integral part of their product offering.

Responding to Legal and Reputational Pressures

AI companies are facing increasing legal and reputational pressure following incidents where chatbots have been accused of encouraging suicide among teenagers and vulnerable users. Image generators like Grok from xAI have also been used to create non-consensual images. These incidents highlight the failure of internal safety safeguards and the need for robust external solutions.

Levenson explains that Moonbounce acts as a trusted third party, applying rules in real-time without being overwhelmed by the context of conversations. The company focuses solely on rule enforcement, leaving the chatbot to manage the contextual memory of the conversation.

Towards Proactive Moderation

With his co-founder Ash Bhardwaj, Levenson is working on a new capability called "iterative steering," which aims to intercept and redirect harmful conversations towards more constructive responses. This approach was inspired by tragic cases, such as the 2024 suicide of a teenager in Florida who was obsessed with a Character AI chatbot.

Rather than simply rejecting harmful topics, Moonbounce's system would intercept the conversation and modify prompts in real-time to encourage more positive and helpful responses from the chatbot.

A Vision for the Future

When asked about a potential acquisition by a company like Meta, Levenson acknowledges that Moonbounce would fit well within the ecosystem of his former employer. However, he remains cautious about the possibility that the technology could be restricted by such an acquisition. He hopes that Moonbounce continues to benefit a wide range of users, without being limited by the interests of a single company.