OpenAI has pulled back the curtain on Jalapeño, its in-house AI inference chip, and the results are turning heads across the semiconductor industry. Unveiled publicly at the Hot Chips 2026 conference at Stanford University and benchmarked against Nvidia's current-generation GB200 and GB300 systems, OpenAI's custom accelerator claims to deliver dramatically better performance per watt and end-to-end latency — the two metrics that matter most for the economics of running AI at scale.
The announcement signals a pivotal moment in the AI hardware race. For years, Nvidia has been the uncontested king of AI compute. Jalapeño is the most credible public challenge to that dominance yet — and it comes from the very company that helped make Nvidia's data center business worth hundreds of billions of dollars.
What Is Jalapeño, and Who Built It?
Jalapeño is a custom Application-Specific Integrated Circuit (ASIC) designed specifically for large language model inference — the process of running AI models to generate responses to user queries. OpenAI co-developed the chip with Broadcom, leveraging Broadcom's long experience in custom silicon and its ASIC design capabilities.
The chip features a NUMA-style (Non-Uniform Memory Access) architecture built around 64 memory and compute slices. Its disclosed specifications are substantial: 13.4 petaFLOP/s of MXFP4 matrix compute performance, 15.4 TB/s of HBM4 memory bandwidth, 216 GiB of high-bandwidth memory, and a 700-watt thermal design power. That power envelope is notably modest compared to Nvidia's Blackwell-era NVL72 rack systems, which consume far more energy per unit of compute.
The chip is designed to scale aggressively. OpenAI says a single rack can hold 128 Jalapeño accelerators, and a full pod can reach 2,048 chips — delivering up to 27 exaFLOP/s of MXFP4 compute, 432 terabytes of HBM4, and 32 petabytes per second of aggregate memory bandwidth at the cluster level.
The Benchmark Numbers
OpenAI's headline claims are striking. Against Nvidia's GB200 and GB300 rack systems — the current apex of Nvidia's data center lineup — Jalapeño reportedly delivers:
- 1.5x to 1.9x higher peak throughput per watt (tokens processed per joule of energy)
- 1.7x to 3.6x lower end-to-end latency (time from request submission to final token delivered)
These aren't lab-cooked numbers with no oversight. SemiAnalysis engineers were present at OpenAI's facilities and verified the benchmark runs in person, lending credibility to the results. The independent firm confirmed the efficiency lead, though it also noted an important caveat: Nvidia's GB300 still leads Jalapeño on absolute raw throughput per package by approximately 20 to 25 percent. Jalapeño's advantage is in doing more useful inference work per unit of energy consumed, and doing it faster from a user experience standpoint.
OpenAI framed its benchmarking methodology around two core metrics: time to last token (a proxy for user experience in chat and API applications) and tokens per joule (a proxy for the cost of inference at scale). By focusing on the full Pareto frontier of request latency versus energy per token — rather than chasing raw chip-level throughput figures — OpenAI is arguing that Jalapeño is optimized for what actually matters when running ChatGPT and its API at billions of queries per day.
AI Designed This Chip
One of the most remarkable aspects of the Jalapeño story is how it was built. OpenAI says AI played a central role in an unusually fast nine-month RTL-to-tapeout development cycle — the process of going from register-transfer level design (the functional specification of the chip) to a finished layout ready for fabrication.
The company used Codex — its own AI coding model — to help accelerate hardware design verification and optimization tasks. AI assistance enabled engineers to improve the chip's performance, power consumption, and silicon area utilization in ways that would have taken significantly longer through traditional methods.
The timeline bears this out: architecture concepts were developed in late 2024, the RTL was frozen in 2025, tapeout occurred in late 2025, Codex was running on the chip in early 2026, and ChatGPT began running on Jalapeño infrastructure not long after. That pace — from architectural concept to production deployment in roughly 18 months — is exceptional for custom silicon.
Why This Matters for the AI Industry
The economics of AI inference are becoming the defining business problem for companies in this space. Training large models is expensive but episodic; inference is continuous, scaling with every user query, every API call, every automated agent workflow. As AI becomes embedded in more products and services, the cost and speed of inference increasingly determines which companies can profitably run AI at scale.
Nvidia has dominated because its GPUs, while general-purpose, were good enough at inference and available in the volumes required. Custom ASICs from companies like Google (TPUs) and Amazon (Trainium, Inferentia) have long promised better efficiency for specific workloads, but Nvidia's software ecosystem — CUDA, cuDNN, and a decade of optimization — has been a powerful moat.
Jalapeño's emergence changes the calculus, at least for OpenAI's own workloads. If the chip can deliver the efficiency gains claimed in production at scale, OpenAI stands to dramatically reduce the cost of running its models — costs that have been a persistent drag on its path to profitability. And by demonstrating that a custom chip can be designed in nine months with AI assistance, OpenAI is implicitly signaling that this advantage could compound over time: each generation of chip designed faster and better than the last.
Nvidia Isn't Standing Still
The benchmarks do not include Nvidia's newest Vera Rubin architecture, which began shipping to select customers recently and represents Nvidia's next leap in performance. Vera Rubin is expected to extend Nvidia's throughput leadership substantially, meaning Jalapeño's efficiency advantages may face stiffer competition once Vera Rubin reaches broad deployment.
Nvidia also continues to benefit from its installed base, software ecosystem, and the breadth of its product portfolio — from edge inference to hyperscale data center. A single efficient chip, however impressive, doesn't replicate that ecosystem overnight.
What Comes Next
OpenAI says Jalapeño will begin deploying in limited volumes at the end of 2026, with more significant rollout planned through 2027. Initially, the chip will power inference for OpenAI's own products — ChatGPT and its API — rather than being sold or licensed to external parties.
Whether OpenAI eventually offers Jalapeño-based inference as a commercial service — effectively competing with Nvidia not just as a chip designer but as an infrastructure provider — remains to be seen. But the message from Hot Chips 2026 is clear: the era of uncontested Nvidia supremacy in AI inference is over. The race is on, and OpenAI is now a serious hardware competitor.
