OpenAI: Jalapeño Faster and More Efficient for Inference

OpenAI releases test results for Jalapeño, its in-house inference chip. The company claims efficiency gains per watt and reduced latency across several open-weight models and announces an internal deployment by the end of the year, while continuing to use third-party accelerators and already preparing for the next generations.

Announced Deployment and Multi-Generation Roadmap

OpenAI plans to launch the deployment of Jalapeño on its computing infrastructure before the end of the year. The company presents Jalapeño as the initial step in a roadmap covering multiple generations, with generation 2 already in advanced development and generation 3 being defined. To prepare for this launch, OpenAI is continuing production validation, software improvements, large-scale operational setup, and performance evaluation across a broader range of models. OpenAI believes that meeting the growing demand for AI will require more computing power from all sources and plans to continue widely deploying NVIDIA accelerators and other partners for training and inference.

Reported Metrics: Efficiency per Watt and Improved Latency

OpenAI evaluated Jalapeño using InferenceX, a public benchmark from SemiAnalysis that covers the complete processing of an AI query, comparing it to commercial systems across regimes ranging from high-throughput to low-latency interactive. The company highlights a metric focused on the amount of useful work done per unit of energy while adhering to a latency constraint, which is particularly important for agents. Across three public models tested, OpenAI indicates that Jalapeño delivers between 1.5 and 1.9 times more work per watt at maximum throughput and shows between 1.7 and 3.6 times lower end-to-end latency compared to the systems tested, with performance 2.1 to 4.1 times higher in highly interactive scenarios. OpenAI also reports an increased number of tokens per user and a higher throughput per kilowatt, placing Jalapeño on the Pareto frontier of the tested range. Comparisons were normalized by the rated power of the accelerators; Jalapeño is rated at 700 watts, with sustained power measured at 550 watts or less under the tested loads. On Kimi K2.5 1T, OpenAI reports approximately 1.5 times more peak performance per watt and about 3.4 times less latency than the comparison system. OpenAI states that its internal tests on its own cutting-edge models further expand this advantage and that the architecture would gain value with larger and more demanding workloads.

Agent-Oriented Architecture: Local Data and Integrated Network

Jalapeño has been designed to meet the needs of current and future language models, particularly for interactive agents. OpenAI emphasizes a rack-scale co-design that encompasses the chip, memory, network, software, and system, based on real-world workloads. The pre-filling phase is described as highly compute-intensive, decoding is limited by memory bandwidth, and communication between cores and chips is identified as a potential source of latency. Jalapeño aims to reduce data movement and communication delays by allowing the model state, including the KV cache, to be placed and retained locally, and by enabling the appropriate combination of compute, memory, and network according to the inference phase. The network is integrated and sized to contain the load within a connected system. OpenAI presents the accelerator as balanced and interchangeable, capable of supporting changing model architectures, performing well in pre-filling and decoding, and adapting to the evolving balance between these stages.

AI-Accelerated Design and Programming

OpenAI attributes a direct role to its models in the development of Jalapeño: earlier generations are said to have assisted in design and setup, while recent models optimize usage and programming. The company reports a transition from initial design to manufacturing in nine months, leveraging AI to explore implementations, shorten design, measurement, and verification loops, and optimize arithmetic circuits. Jalapeño is presented as a clear and predictable programming target, with work described via local tensors, explicit communication, and predictable synchronization, leaving the AI to optimize mapping, placement, scheduling, and coordination. However, OpenAI notes that each new family of models requires dedicated kernels and optimizations. By using Codex with GPT-Astra, the team claims to have brought three open-weight models, not originally planned, to a high level of performance in two months, which is presented as evidence of flexibility and programming speed. On certain attention blocks and expert mixture components of GPT-OSS, OpenAI reports AI-generated implementations that are 1.5 to 1.8 times faster than those written by human experts, noting that these gains pertain to selected blocks and not the entire model.

Expected Implications for Cost and Availability

OpenAI asserts that generating more useful work with constant power and hardware can help meet demand and reduce the cost of producing an outcome. The company believes this can increase operational leverage by growing useful work and revenue faster than service costs, while facilitating shorter iteration cycles and the emergence of new uses. OpenAI presents Jalapeño as extending inference regimes: an ultra-fast mode with efficiencies previously limited to the fast mode, a fast mode with efficiencies previously unique to batch mode, and increased efficiency in batch mode. OpenAI projects faster responses and more responsive agents for customers, as well as more reliable access when demand increases. The company describes a multi-level advantage and characterizes Jalapeño as first-party silicon with measured results, while aiming for faster, more powerful, and more efficient products, in line with its stated mission around general AI.