GPT-OSS-120B Benchmark: Key Insights and Best Practices

GPT-OSS-120B is OpenAI's larger open-weight reasoning model, released in August 2025 under the Apache 2.0 license. Because anyone can download and serve it, "the GPT-OSS-120B benchmark" means several things: the scores OpenAI published in its model card, the speed and accuracy independent testers measure across API providers, the hardware throughput results in MLPerf Inference, and the numbers you produce in your own environment.

This guide keeps those layers separate. It summarizes the published results available as of October 30, 2026, explains how to read them, and lays out a reproducible protocol for benchmarking quality, latency, throughput and cost yourself. Every figure comes from a cited source; examples describe methods, not results, and nothing here reports tests run by Cognativ.

Key Takeaways

• Efficiency and strengths: GPT-OSS-120B is a sparse Mixture-of-Experts (MoE) model with about 117 billion total parameters but only about 5.1 billion active per token, which keeps compute per token low.
• Published results: at high reasoning effort, OpenAI reports 92.5% on AIME 2025 without tools, 80.1% on GPQA Diamond, 90.0% on MMLU and 62.4% on SWE-bench Verified.
• Reasoning effort matters most: the same model scores very differently at low, medium and high effort, so a result without its effort level is hard to interpret.
• Methodology and practices: the same weights can score differently on different endpoints, so a fair comparison pins prompts, sampling, reasoning effort, context handling and hardware.

Introduction to GPT-OSS-120B

GPT-OSS is a family of two open-weight models, GPT-OSS-120B and GPT-OSS-20B, built for reasoning, tool use and structured outputs. According to the gpt-oss-120b and gpt-oss-20b model card that OpenAI published on August 8, 2025, both use a mixture-of-experts transformer and support three reasoning effort levels. The weights are on Hugging Face under Apache 2.0.

Open weights change what a benchmark means. With a closed API, a score describes one vendor's product. With GPT-OSS-120B, the same weights can run on your own GPUs, a managed cloud endpoint or a specialized inference provider, and each serving stack can change both speed and answer quality. A credible evaluation holds prompts, decoding and resource budgets constant; only then do observed differences reflect capability rather than environment.

Architecture Primer: Why 120B Looks "Bigger Than It Runs"

The model card lists 36 layers, 116.8 billion total parameters and 5.1 billion active parameters per token. Each MoE layer holds 128 experts, and a router selects the top 4 for every token, so the model has the capacity of a large network while each forward pass costs roughly what a much smaller dense model would.

The attention stack uses grouped-query attention (64 query heads, 8 key-value heads) and alternates banded-window layers of 128 tokens with fully dense layers. Rotary position embeddings, extended with YaRN, give a context length of 131,072 tokens. Grouped-query attention shrinks the KV cache, which directly affects how many concurrent requests a GPU can hold in a throughput benchmark.

MXFP4 Quantization and the Single-GPU Claim

OpenAI released the MoE weights in MXFP4, at about 4.25 bits per parameter. Because MoE weights make up more than 90% of the parameters, the checkpoint is 60.8 GiB and the model card states that GPT-OSS-120B fits on a single 80 GB GPU. The Hugging Face card notes that all published evaluations used this MXFP4 quantization, so a benchmark of a different quantization is not directly comparable. For memory sizing and GPU choices, see our guide to GPT-OSS-120B hardware requirements.

Reasoning Effort and the Harmony Format

Reasoning effort is set in the system prompt, for example "Reasoning: high." The model card describes "smooth test-time scaling": accuracy rises roughly log-linearly with reasoning length on most tasks, at the cost of more tokens, latency and spend. That makes effort the most important setting to pin in any benchmark.

The model was trained on OpenAI's harmony response format, and the official gpt-oss repository warns that it will not work correctly without it. Serving frameworks usually apply harmony through the chat template, but a custom harness must apply it explicitly. A broken template is a common reason a local benchmark underperforms the published numbers.

GPT-OSS-120B Benchmark Results from OpenAI's Model Card

The most complete set of quality results is still OpenAI's model card. It reports pass@1 accuracy at each reasoning level with the default system prompt, plus "with tools" variants where the model could use tools. The table reproduces selected figures for GPT-OSS-120B.

Source: OpenAI, gpt-oss-120b and gpt-oss-20b model card, arXiv:2508.10925 (August 2025). Values are accuracy percentages except Codeforces, which is an Elo rating.

Reasoning and Math

Math and science show the largest gains from reasoning effort. AIME 2025 without tools climbs from 50.4% at low effort to 92.5% at high effort, and GPQA Diamond rises from 67.1% to 80.1%. Humanity's Last Exam stays hard at every setting, as expected for a benchmark built to sit at the frontier of expert knowledge.

Coding and Agentic Workflows

SWE-bench Verified measures resolving real GitHub issues, Tau-Bench measures multi-turn tool use in simulated customer-service domains, and Codeforces measures competitive programming. These results are sensitive to agent scaffolding, tool schemas and retry policy, so your own agentic results will reflect your framework as much as the model. The model card also reports HealthBench close to OpenAI o3 and a multilingual MMMLU average of 81.3% at high effort, with lower scores in lower-resource languages.

How GPT-OSS-120B Compares with o3-mini and o4-mini

OpenAI's summary is that GPT-OSS-120B "surpasses OpenAI o3-mini and approaches OpenAI o4-mini accuracy" on its core evaluations. Treat that as the vendor's characterization under its own harness. Comparisons with other open source models, or with closed systems released since August 2025, should come from independent evaluations with identical prompts and settings.

Independent GPT-OSS-120B Benchmarks and Evaluations

Because the weights are open, several independent groups benchmark GPT-OSS-120B. They answer different questions, and mixing their numbers is a common source of confusion.

Artificial Analysis: Provider Speed, Latency and Price

Artificial Analysis continuously benchmarks GPT-OSS-120B across API providers. On October 30, 2026, its provider page listed 16 providers for the high-effort variant, used a default workload of 10,000 input tokens and reported median figures over the past 72 hours. It showed Cerebras as the fastest provider at about 1,663 output tokens per second, followed by SambaNova at about 707 and Groq at about 471, with blended prices from about $0.04 to $0.39 per million tokens. These figures change constantly, so record the date whenever you cite them.

For reasoning models, the page's time to first answer token includes thinking time, so it is not comparable with a non-reasoning model's time to first token. Artificial Analysis also revises its workloads and Intelligence Index over time, so scores from different index versions are not comparable.

Provider Accuracy Is Not Guaranteed

In August 2025, Artificial Analysis ran AIME 2025 32 times per provider at high effort and found results from 93.3% on several endpoints down to 80.0% on others, with one heavily compressed deployment far lower. As Simon Willison summarized on August 15, 2025, one cloud's low score traced to an older vLLM build that ignored the reasoning-effort setting and silently ran at medium effort; it was later fixed.

OpenAI then published a guide to verifying gpt-oss implementations, with a function-calling compatibility suite and instructions for running AIME and GPQA through the gpt_oss.evals module. If your scores are close to the published ones and the tool tests pass, the guide says you likely have a correct implementation. The lesson for any benchmark: verify the endpoint before you measure it.

MLPerf Inference: Hardware Throughput Under Fixed Rules

MLCommons added GPT-OSS-120B to MLPerf Inference v6.0, with results published in April 2026. It is the first MLPerf Inference benchmark with separate performance and accuracy datasets. Throughput is measured at low effort on PubMed summarization (mean input 5,000 tokens, mean output 1,250), while accuracy is checked at high effort on AIME 2024, GPQA Diamond and LiveCodeBench v6. The server scenario requires a 99th-percentile time to first token of 3.0 seconds or less and time per output token of 80 ms or less; the interactive scenario tightens those to 2.0 seconds and 15 ms.

In MLPerf Inference v6.1 (September 2026), GPT-OSS-120B was the most popular datacenter benchmark, with 54 systems, and the interactive scenario now allows speculative decoding. MLPerf shows what a full system sustains under audited rules, not how one GPU will behave with your prompts. Other trackers, such as SemiAnalysis InferenceX for accelerator cost-efficiency and Princeton's Holistic Agent Leaderboard for agent tasks, use their own harnesses, so compare numbers only within the same source.

How to Read GPT-OSS-120B Benchmark Numbers

A score is only meaningful alongside its settings. Before you use a published figure in a decision, check the objectives and threats below.

Evaluation Objectives

Most evaluations have three objectives: quantify quality, latency and cost under realistic prompts; compare API providers and self-hosted runs without confounds; and identify where the model is wrong, with a failure taxonomy that guides mitigation. A leaderboard helps with the first; only your own workload answers the other two.

Threats to Validity

• Reasoning effort not stated: low and high effort differ by more than 40 points on AIME 2025.
• Tools versus no tools: tool-enabled scores are not comparable with closed-book scores.
• Prompt drift: different system preambles, templates or schemas change results.
• Decoding variance: temperature, top-p and maximum tokens must be fixed and reported.
• Hardware and serving variance: accelerators, kernels, framework versions and quantization change speed and accuracy.
• Data contamination and selective reporting: leaked test items and omitted failures inflate scores.

Mitigation is procedural: preregister the protocol, check every setting into a run card, and publish seeds, logs and errors with the headline numbers.

Benchmark Suite Design

Combine a few public benchmarks for calibration with a private set that reflects your tasks. Public benchmarks show whether your setup reproduces published numbers; private tasks show whether the model is good enough for your product.

Task Coverage

• Reasoning: AIME-style math and GPQA Diamond, the sets OpenAI and MLPerf use.
• Coding: synthesis and repair scored with pass@k, plus real issues from your repositories.
• Knowledge and reading: multi-hop question answering with citation checks.
• Agentic workflows: retrieval, tool calls and function chains with your real schemas.

Dataset Hygiene, Ground Truth and Scoring

Freeze splits, document sources and record every change in a dated changelog. Use exact or normalized match for short answers, rubric grading for long-form explanations and unit tests for code. For structured outputs, enforce a JSON Schema and record the schema-pass rate, because a correct answer in the wrong format is a production failure.

Reproducible Inference Configuration

Consistent results depend on a standardized inference configuration. If you are setting up a local stack for the first time, our guide to setting up a local LLM for optimal performance covers the basics.

Decoding Defaults

The official gpt-oss repository recommends temperature 1.0 and top-p 1.0. Low-temperature settings that suit other models are not the documented default here, so report any deviation and expect results to differ from published figures. Set generous output limits for reasoning tasks: MLPerf allows up to 32,768 output tokens in its accuracy runs, and a low cap truncates chains of thought. Fix stop sequences and system prompts per task family.

Reasoning Effort, Context and Tool Use

Run the primary evaluation at the effort level you plan to deploy, then ablate the other two to map the quality-latency frontier, and confirm that output length changes between levels so you know the endpoint honors the setting. For long contexts, report input lengths, KV-cache settings and truncation rules. For agentic trials, standardize tool schemas, count tool calls and tool latency separately from model time, and run OpenAI's compatibility tests first.

Metrics: Quality, Latency, Cost

Measure three dimensions together, because improving one usually costs another.

Quality

Report exact match or F1 for short answers, rubric scores for explanations, pass@k for code and citation precision and recall for retrieval. Repeat runs: OpenAI's harness uses multiple attempts per AIME and GPQA problem, and MLPerf uses 3 to 8 repeats per accuracy dataset. Report 95% confidence intervals and per-category breakdowns that show where the model is consistently wrong.

Latency, Speed and Throughput

Track time to first token (TTFT), time per output token (TPOT), output tokens per second and end-to-end latency including tool overhead, at p50, p95 and p99. For a reasoning model, separate time to first token from time to first answer token. For throughput, follow MLPerf's pattern: fix a latency target and report the highest load that still meets it. Single-request speed and loaded throughput are different numbers, and both belong in a report.

Pricing

For hosted APIs, compute cost per million tokens and, more importantly, cost per successful task, because high reasoning effort multiplies output tokens. For self-hosted deployments, include amortized hardware, power and operations, and state every assumption.

Step-by-Step Experimental Protocol

This protocol keeps friction low while producing results others can reproduce.

1. Start: freeze benchmarks, prompts, schemas and seeds.
2. Environment: pin the weights hash, framework, kernels and tokenizer.
3. Verify: run the compatibility tests and a quick GPQA or AIME eval, and compare with published figures.
4. Warm up: do one excluded dry run per task.
5. Primary runs and ablations: repeat each task at deployment effort, then vary effort, context and concurrency one at a time.
6. Validation and accounting: combine automatic graders with human spot checks, and log tokens, latency, tool calls, errors and retries.
7. Report: compute means, confidence intervals, failure patterns and cost per correct answer, and record what changed since the last run.

Measuring Quality with gpt_oss.evals

OpenAI's verification guide shows how to point gpt_oss.evals at any OpenAI-compatible endpoint. The vLLM GPT-OSS recipe lists reference results for GPT-OSS-120B without tools of 79.4 on GPQA and 93.0 on AIME 2025 at high effort, and notes that scores vary slightly between runs.

python -m gpt_oss.evals --base-url http://localhost:8000/v1 \
  --eval aime25 --sampler chat_completions \
  --model openai/gpt-oss-120b --reasoning-effort high

Measuring Latency and Throughput with vLLM

The vllm bench serve command reports TTFT, TPOT, inter-token latency, end-to-end latency and throughput at chosen percentiles. The vLLM recipe recommends matching maximum concurrency to the server's maximum sequence count and sending at least five times that many prompts. Prefer prompts that match your real input and output lengths over random data.

vllm bench serve --model openai/gpt-oss-120b \
  --dataset-name random --random-input-len 1024 --random-output-len 1024 \
  --max-concurrency 64 --num-prompts 320 \
  --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,95,99

The recipe also explains the main trade-off: at a fixed batch size, more tensor parallelism usually lowers latency but reduces throughput per GPU, while larger batches raise throughput until KV-cache memory runs out. Benchmark the configuration you intend to run.

Minimal Evaluation Pseudocode

If you build your own harness, keep the loop simple and log what the report needs:

for t in tasks:
    for seed in t.seeds:
        r = client.generate(system=f"Reasoning: {effort}\n{t.system}",
                            user=t.user, temperature=1.0, top_p=1.0,
                            max_tokens=t.max_out, seed=seed)
        log.append({"task": t.id, "seed": seed, "ttft_ms": r.ttft_ms,
                    "truncated": r.finish_reason == "length",
                    "schema_ok": validate(r.text, t.schema),
                    "score": grade(r.text, t.gold),
                    "tokens": r.usage, "tool_calls": r.tool_calls})

Reporting Template: What to Publish

A benchmark is only useful if someone else can understand and repeat it. Publish these fields with every run:

• Model and decoding: weights hash, quantization, reasoning effort, temperature, top-p, maximum tokens and stop sequences.
• Context and environment: window, truncation, retrieval depth, accelerator, framework version, or provider, region and date.
• Quality: scores with confidence intervals, schema-pass rate and refusal rate.
• Latency, throughput and cost: TTFT, TPOT and tokens per second at stated concurrency, plus cost per correct answer.
• Failure modes and replicability: frequent "wrong because" patterns, seeds, logs and an example notebook.

Keep these fields on a one-page run card and maintain a dated run ledger, because framework and kernel updates can shift results as much as a new model version.

Pitfalls and How to Avoid Them

Most misleading GPT-OSS-120B benchmarks fail in predictable ways:

• Template errors: a malformed harmony template degrades output → use the framework's chat template and verify with the compatibility tests.
• Ignored reasoning effort: the endpoint silently runs another level → confirm that output length changes across levels.
• Schema drift: small format shifts create silent penalties → lock schemas and write a validator.
• Token starvation: a low token limit clips reasoning → set generous caps and track truncation.
• Cherry-picking: reporting only the best attempt → disclose sampling and retry policy.
• Apples-to-oranges comparisons: different kernels, quantization or batch policies → normalize them or use separate tables.

Comparative Interpretation and Cost Planning

When you compare GPT-OSS-120B with closed systems such as Gemini, Claude or proprietary GPT models, use identical prompts, tools and output limits, and state that the numbers apply to your setting. Check for drift across providers, including rate limits and safety filters, and avoid turning one benchmark suite into a claim about overall intelligence. If you need help turning evaluation results into a model and architecture decision, our LLM solutions team works through these trade-offs.

Cost and Capacity Planning

Throughput at your latency target, multiplied by serving replicas, determines how many concurrent sessions you can support. Use the lowest reasoning effort that meets your accuracy bar on interactive paths, publish cost bands at p50 and p95 including retries, and re-measure after every framework or weights update. For self-hosted sizing, our second hardware guide covers GPT-OSS hardware requirements for both model sizes, and our AI infrastructure practice helps size and operate private deployments.

Safety, Harms and Limitations

Track refusal rate, over-refusal on benign prompts and policy adherence. The model card notes that OpenAI did not apply direct optimization pressure to the chain of thought, so raw reasoning can contain hallucinated or policy-violating content and should not be shown to users without filtering or summarization. Finally, benchmarks cannot capture full production complexity, such as intent drift and tool latency variance, so treat scores as directional and confirm them on your own data.

Frequently Asked Questions

Conclusion

The published GPT-OSS-120B benchmark results are strong for an open-weight model that fits on a single 80 GB GPU, but every number depends on reasoning effort, tools, serving stack and harness. The model card shows what the model can do under reference conditions, independent trackers show how specific providers and systems perform, and only your own evaluation shows whether it fits your workload.

Verify the endpoint first, pin the reasoning level, log full latency and pricing data, publish failure patterns and re-measure after every update. Teams that do this can make sound quality-versus-cost decisions across providers and deployments. If you want help designing a GPT-OSS-120B evaluation or moving from benchmark to production, contact Cognativ to discuss your AI project.