OpenAI Expands Its Decision API Under Rival Pressure

OpenAI's decision API is now accessible to all clients, featuring voice and visual enhancements with a focus on latency. Trials with cited partners indicate significant time savings, while public rankings highlight other players, notably Liquid and Base Lab.

Rankings Favor Other Models Over OpenAI

Two public benchmarks, the AIM System One Index and the Artificial Analysis Decision Index, spotlight Liquid's D1 Ultra, released the previous week, as well as Base Lab's Elephant-1. TypeSafe's latest Jev model remains well-positioned, holding seventh and third places according to the index. OpenAI's decision model appears lower, in eighth and fourteenth places. Additionally, TypeSafe's offerings are currently the most consulted on OpenRouter. Another test, JevMan, has decision models playing PacMan, where the latest Jev excels; its creator, Felix Wunderlich of Opper AI, explains that the game requires a continuous stream of quick decisions, with each intersection demanding a choice within a few hundred milliseconds.

General Availability and Added Options by OpenAI

The decision API was announced during the developer day on September 29 and then made available to all clients on a Thursday. The public version adds voice and visual modalities. OpenAI has also worked on geographically distributing its inference resources to reduce latency based on regions.

Technical Choices: Reconfigured Luna and Latency Gains

Rather than designing a specific calibrated model, OpenAI has reconfigured ChatGPT Luna to accelerate inference and lower costs. According to Nikunj Handa, this has reduced processing times from a range of one to three seconds to 100, 120, and 180 milliseconds. He notes that targeted training could follow, but Luna is the current engine. Handa also highlights a potential advantage in multi-modality: for image classification, Luna is described as powerful and better calibrated, while clients are requesting more flexibility to produce outputs in the form of images.

Initial Use Cases: Replacing LLMs and Metrics at Speak

In initial trials with clients like Box, Datadog, Ramp, and Speak, teams have used the API to replace OpenAI's LLMs while increasing throughput. According to Andrew Hsu, CTO of Speak, the delay between two actions has dropped from 1.4 seconds to 0.13 seconds. This product has also been adopted by OpenAI to enhance the responsiveness of its voice assistant Dots.

A Defined Competitive Landscape but Fluid Criteria

OpenAI is making its decision API widely available in a market where TypeSafe, founded by a former researcher from a leading lab, has reached a valuation of $7.6 billion. The Jev model has popularized the approach of super classifiers, capable of choosing among options and indicating confidence, and developers report that they are replacing slower and more expensive LLMs for certain workflows. At OpenAI, Nikunj Handa mentions a multitude of possible use cases. However, the criteria for a "good" decision model remain variable depending on the tasks, including intelligence, cost, calibration, and speed. Diogo Almeida, CEO of TypeSafe, is cautious regarding benchmarks and recommends contextual testing, although TypeSafe published its own rankings the previous week with a new Jev incorporating support for images and intelligent statements. Handa still aims for latency gains down to 60 milliseconds and anticipates the emergence of a wide range of use cases as these models proliferate.