OpenAI Revolutionizes Moderation with Free Omni Moderation

⚡

Key Takeaways

1OpenAI introduces Omni Moderation, a free model for filtering potentially harmful texts and images.

2Based on GPT-4o, this model assesses violent, hateful, adult, and harmful content.

3Omni Moderation is ideal for securing chatbots, social media, and multilingual applications.

💡Why it matters — Omni Moderation provides an accessible and effective solution to enhance the safety of digital platforms against inappropriate content.

OpenAI Omni Moderation: A Major Advancement for Digital Safety

In a world where digital security has become crucial, OpenAI offers an innovative solution with its omni-moderation-latest model. Designed to enhance the safety of systems based on language models, this free model can detect potentially harmful content, whether textual or visual. This article explores the features of this model, its access, and its usage.

OpenAI's Moderation Models

OpenAI has developed two distinct moderation models: the legacy model ‘text-moderation-latest’ and the more recent ‘omni-moderation-latest’. The latter, based on GPT-4o technology, provides multimodal moderation, meaning it can analyze both text and images. A notable advantage of this model is its free accessibility via a dedicated endpoint.

The Omni moderation API evaluates content across several categories:

Violent content
Hate speech
Adult content
Harmful content

To test this model, examples of safe and unsafe inputs can be used for both text and images. For instance, using Google Colab, it is possible to implement these tests with an OpenAI API key, which remains necessary even though the service is free. The API key can be obtained from the OpenAI website.

Setting Up and Using the API

To use the API, it is necessary to import the required libraries and initialize the OpenAI client with the API key. Here is a sample code to get started:

from openai import OpenAI
from getpass import getpass

# Enter the API key securely
api_key = getpass("Enter your OpenAI API key: ")

# Initialize the client
client = OpenAI(api_key=api_key)

Once the API key is entered, the client is ready to be used for moderation.

Function to Display Results

To display the moderation results, a helper function is defined. It presents the detected categories and associated scores clearly:

def display_moderation(response, title="MODERATION RESULT"):
    result = response.results[0]
    categories = result.categories.model_dump()
    scores = result.category_scores.model_dump()
    print("\n" + "=" * 60)
    print(f"{title:^60}")
    print(f"\nFlagged: {result.flagged}")
    print("\nCATEGORIES")
    for category, value in categories.items():
        print(f"{category:<30} : {value}")
    print("\nCATEGORY SCORES")
    for category, score in scores.items():
        print(f"{category:<30} : {score:.6f}")

This function is used to analyze safe and unsafe texts. For example, an innocuous text asking for help learning Python will be correctly identified as safe.

Analyzing Texts and Images

To test text moderation, an example of violent text is submitted to the model, which correctly flags it as such. Similarly, for images, a URL of a violent image is analyzed, and the model also flags it for violence.

unsafe_image_url = "https://i.ytimg.com/vi/DOD7s1j_yoo/sddefault.jpg"
response = client.moderations.create(
    model="omni-moderation-latest",
    type="image_url",
    url=unsafe_image_url
)
display_moderation(response, "IMAGE MODERATION")

It is possible to adjust the sensitivity of moderation by using category scores to set custom thresholds.

Potential Applications

OpenAI's omni-moderation-latest model finds applications in various fields requiring rigorous content monitoring:

Chatbots: Filter harmful content before it is processed by the language model.
Image Analysis: Identify potentially harmful images before processing.
Social Media: Detect and report hate speech and abusive content.
Live Streaming: Monitor video streams for inappropriate images.
Multilingual Applications: Improve content moderation in different languages.

This model offers an effective and free solution to enhance the safety of digital platforms. While other solutions like Azure AI Content Safety exist, Omni Moderation stands out for its cost-free nature and ease of integration.

Frequently Asked Questions

Q1. What is the latest moderation model from OpenAI?
A. The most recent model is omni-moderation-latest, which supports moderation of both text and images.

Q2. Is OpenAI's moderation free to use?
A. Yes, access to OpenAI's moderation API is free.

Q3. What happens to the legacy moderation model?
A. The legacy model text-moderation-latest is limited to text, while omni-moderation-latest is recommended for new applications.