What Is Temperature in LLMs and Its Impact on Output Quality

Temperature is a sampling parameter in large language models (LLMs) that governs how conservative or adventurous the model is when it chooses the next token. In practical terms, temperature reshapes the probability distribution over candidate next words: lower temperature settings make the distribution sharper and the output more predictable, while higher temperature values flatten it and make the output more diverse and less predictable.

For years, tuning LLM temperature was one of the fastest levers for controlling tone, creativity and risk in LLM output. That is still true for many open-weight models and older API models. In 2026, though, the picture has changed: several of the newest reasoning models from OpenAI, Anthropic and Google either reject custom temperature values or ignore them, and their vendors now recommend steering behavior with prompts and reasoning effort instead.

This guide explains how temperature works mathematically, how it interacts with top-p, top-k and min-p, how current APIs and local runtimes handle sampling parameters as of October 2026, and how to choose, test and document settings by task. Vendor behavior is cited to official documentation, and anything that varies by model is flagged so you can check your own model's page.

What Is Temperature in LLMs?

An LLM generates text one token at a time. At each step the network produces a raw score, called a logit, for every token in its vocabulary. Those scores are converted into probabilities, and the sampler picks the next token from that probability distribution. Temperature is the number that controls how peaked or how flat that distribution is before the pick happens.

OpenAI's API reference describes it plainly: temperature takes a value between 0 and 2, and "higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic" (OpenAI Responses API reference). Google's Gemini prompting guide uses similar language, noting that lower temperatures suit prompts that need a more deterministic or less open-ended response, while higher temperatures can lead to more diverse or creative results (Gemini prompt design strategies).

Two clarifications prevent most misunderstandings. First, temperature does not change what the model knows; it changes how the sampler chooses among options the model already considers plausible. Second, "creativity" is a loose label. Higher temperature increases variety, but variety is not the same as quality, originality or usefulness, and past a certain point it mostly adds noise.

How Temperature Works

Modern LLMs turn logits into a probability distribution with the softmax function. Temperature, written T, divides every logit before the softmax is applied:

p_i = exp(z_i / T) / Σ_j exp(z_j / T)

Here z_i is the logit for token i and p_i is the resulting probability. Because every logit is divided by the same number, the ranking of tokens never changes. What changes is the size of the gaps between them:

• Low temperature (T below 1): dividing by a small number stretches the gaps between logits, so the softmax distribution becomes sharper. Most of the probability mass lands on the top few tokens, and the model behaves more deterministically.
• T equal to 1: the logits pass through unchanged, so the model samples from the distribution it learned.
• High temperature (T above 1): dividing by a larger number shrinks the gaps, so the distribution flattens. Less probable tokens get a real chance of being sampled, which produces more varied and less predictable text.

A convenient intuition is that temperature is a knob for how bold the sampler is. The same base logits can yield very different generated text depending on the numerical value of T.

Interpreting Temperature at the Token Level

Mathematically, dividing logits by T makes the softmax distribution sharper as T falls and flatter as T rises. At small temperature values the sampler assigns high probabilities to the top candidates, so the model keeps selecting high-likelihood tokens and the response stays close to its most typical phrasing. As temperature increases, probability mass diffuses into the tail of the distribution, and rarer word choices start to appear. This mechanism explains why low T tends to produce consistent responses across runs, while high T produces more diverse outputs.

Because generation is sequential, each sampled token becomes part of the context for the next one. A single unusual choice early in a response can push the rest of the continuation in a new direction, which is why high-temperature outputs can drift further from the brief the longer they get.

What Temperature 0 Really Means

As T approaches 0, the distribution collapses onto the single most likely token, which is the same as greedy decoding. Many runtimes treat 0 as a special case; vLLM's sampling parameters, for example, state that zero means greedy sampling (vLLM SamplingParams).

Greedy does not guarantee identical output on hosted APIs, however. Anthropic's migration guide notes that using temperature 0 for determinism "never guaranteed identical outputs" on its earlier models (Claude Opus 5.5 migration guide), and OpenAI documents its seed parameter as a best-effort beta feature where determinism is not guaranteed (OpenAI Chat Completions reference). If you need reproducibility, log inputs and outputs rather than relying on a sampling setting.

Temperature, Training Data, and Input Context

Even with identical temperature settings, output quality is bounded by training data and input context. Temperature only redistributes probability among tokens the model already ranks; it cannot add knowledge the model never learned or facts the prompt never supplied. When a model has been trained on clean, domain-matched data, the top of its distribution tends to reflect sensible priors, and a lower temperature simply keeps it there.

Conversely, if the prompt omits key constraints, model performance degrades at any temperature. The model's response will mirror uncertainty in the context more than your sampling choice. In short, temperature modulates variability around what the network learned, while data coverage and conditioning still dominate the generated output. Clear instructions, examples and retrieved evidence usually move quality further than any sampling tweak.

Effects of Low, Medium and High Temperature

The exact behavior at a given value depends on the model, so the bands below describe directions rather than universal thresholds. A value of 0.7 on one model is not guaranteed to behave like 0.7 on another, because the raw logits differ between model families.

Effects of Low Temperature

With a low temperature, the sampler concentrates probability mass on the top candidates and produces near-deterministic output, reducing to greedy decoding at 0.

• Strengths: stable, concise wording that is easy to compare across runs; a good fit for tasks requiring precision, such as structured extraction, classification, SQL or code generation and policy-constrained answers.
• Weaknesses: lower diversity and a higher risk of repetitive phrasing. Some model vendors also warn that greedy or very low temperature can cause loops on reasoning-heavy models, as covered later in this guide.
• When to use: schema-bound outputs, exact paraphrases, unit-test generation, or any flow where you need consistent expected output across runs.

Medium Temperature and Its Applications

A moderate temperature aims for a balance: responses remain coherent and grounded in the prompt but still vary in wording and examples, so they do not sound robotic.

• A common choice for conversational agents, support assistants and content suggestions that must remain useful without sounding scripted.
• Useful for summaries, email drafts and educational content where tone matters but correctness remains important.

When a Moderate Temperature Works Best

A moderate setting is often the pragmatic default for assistants. The model generates slightly more variety than greedy decoding while staying close to its most likely continuation, which keeps text engaging without sacrificing predictable structure. Many APIs already default to a temperature of 1, so for chat-style products the most defensible starting point is often the model's own default, adjusted only after you have measured a reason to change it.

Effects of High Temperature

With a high temperature, sampling explores the tail of the distribution, selecting more unusual continuations and yielding more diverse responses.

• Strengths: more surprising ideas and varied style; helpful for brainstorming, naming, fiction and ad-copy ideation.
• Weaknesses: increased risk of drift, contradictions and reduced factual accuracy. Coherence tends to drop as values climb; OpenAI's API, for example, caps temperature at 2.
• When to use: ideation, alternate phrasings, character voices, brand tone exploration and speculative drafts that a human will filter.

Comparing Outputs (Qualitative Patterns)

Consider a prompt such as "Suggest three strategies to reduce cart abandonment." At a low temperature, a model typically returns the most conventional answers, phrased much the same way on every run. At a moderate temperature, the same ideas tend to appear with more specific wording and occasional variations. At a high temperature, less common suggestions show up more often, alongside a greater chance that some of them miss the brief. The only reliable way to find the sweet spot for your model and task is to compare outputs side by side on your own prompts and rate them for usefulness, novelty and correctness.

Temperature vs. Top-p, Top-k and Min-p

Some SDKs expose a "creativity" control that maps to temperature; others expose temperature directly alongside several truncation controls. Temperature reshapes the whole distribution, while truncation samplers decide which tokens are allowed into the candidate pool at all. A useful shorthand is that temperature sets the mood and top-p, top-k and min-p set the boundaries.

Top-k Sampling

Top-k keeps only the k most probable tokens and discards the rest before sampling. It was used for open-ended generation in Fan et al.'s 2018 paper on story generation (Hierarchical Neural Story Generation). Google's documentation describes a top-k of 1 as greedy decoding and a top-k of 3 as choosing among the three most probable tokens using temperature. The weakness of top-k is that the pool size is fixed: k tokens may be too many when the model is confident and too few when many continuations are reasonable.

Top-p (Nucleus Sampling)

Top-p, also called nucleus sampling, keeps the smallest set of tokens whose cumulative probability reaches p. It was introduced by Holtzman et al. to reduce the bland, repetitive text that pure maximization produces and the incoherence of unrestricted sampling (The Curious Case of Neural Text Degeneration). OpenAI's reference explains that a top_p of 0.1 means only the tokens comprising the top 10% of probability mass are considered. Because the pool expands and contracts with the model's confidence, top-p adapts better than top-k.

Min-p Sampling

Min-p sets a threshold relative to the most likely token. Ollama's documentation gives a worked example: with min_p of 0.05 and a top token at probability 0.9, tokens below 0.045 are filtered out (Ollama Modelfile reference). When the model is confident, the threshold rises and the pool shrinks; when it is uncertain, the threshold falls and more options survive. The method was formalized by Nguyen et al., who argue it preserves coherence at higher temperatures better than top-p (Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs). Min-p is mainly available in open-weight runtimes such as Ollama and vLLM rather than in hosted frontier APIs.

Penalties, Length Limits and Stop Sequences

Frequency and presence penalties adjust logits based on what has already been generated. In OpenAI's Chat Completions API both range from -2.0 to 2.0; positive frequency penalties reduce verbatim repetition, and positive presence penalties increase the likelihood of moving to new topics. Local runtimes offer similar controls, such as Ollama's repeat_penalty. Maximum output length and stop sequences do not change token probabilities but do constrain rambling and enforce clean termination.

How the Controls Combine

Temperature and truncation samplers are coupled. A high temperature with a very tight top-p may still look conservative, because only a few tokens survive the cut, while a moderate temperature with no truncation can be unusually adventurous. Gemini's guide describes one common order: tokens are first limited by top-k, then filtered by top-p, and the final token is selected with temperature sampling. Other runtimes order their samplers differently, so check your stack's documentation. OpenAI also advises altering temperature or top_p but not both, which keeps experiments interpretable.

Next Word Prediction and Decoding Strategies

LLMs generate one next token at a time. A typical sampling loop transforms logits by temperature, applies the softmax, optionally prunes candidates with top-k, top-p or min-p, then draws the next token and appends it to the context. This iterative choice is what shapes both the style and the content of the final output.

Common decoding strategies include:

• Greedy decoding: always pick the most probable token. It is equivalent to temperature approaching 0 and is fully determined by the logits.
• Beam search: keeps several high-probability candidate sequences in parallel and returns the best-scoring one. It suits tasks with a narrow set of correct answers, such as some translation work, but tends to reduce variety in open-ended text.
• Temperature with truncation: temperature combined with top-p, top-k or min-p is the standard approach for controllable free-form generation.
• Speculative decoding: a smaller draft model proposes tokens that the main model verifies. It speeds up generation and is not a diversity control, so it does not replace temperature.

If you run models yourself, these choices live in your inference server rather than in a vendor API. Our guide to setting up a local LLM for optimal performance covers the runtime side of that setup.

How Reasoning Models and Current APIs Handle Temperature (October 2026)

The biggest change since this article was first published is that sampling parameters are no longer universally adjustable. Reasoning models generate internal thinking before their final answer, and vendors have concluded that hand-tuned sampling interferes with that process more often than it helps. The rules below come from each vendor's documentation as of October 30, 2026; they change frequently, so verify against your specific model's page before you ship.

OpenAI

The Responses API still documents temperature (0 to 2, default 1) and top_p (0 to 1), with a recommendation to alter one but not both. For reasoning models the constraint is stricter. OpenAI's GPT-6 migration guide says that when reasoning effort is anything other than none, you must remove temperature, top_p and top_logprobs from requests (Using GPT-6). The same guide notes that GPT-6 Astra and GPT-6.1 Sol do not support none at all, while GPT-6.1 Terra, GPT-6 Sol and GPT-6 Luna do. In practice, temperature is only adjustable on those models when reasoning is switched off. To control how much the model thinks, use reasoning.effort, whose supported values are model-dependent (OpenAI reasoning guide).

Anthropic

Anthropic's thinking documentation lists a group of current models, including Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5, Claude Opus 4.7 and Claude Opus 4.8, on which non-default temperature, top_p or top_k values return a 400 error on every request, whether or not thinking is used. On those models the top_p default is 0.99, so even sending a top_p of 1, or sending temperature and top_p together at their defaults, is rejected (Claude thinking documentation).

Anthropic's API version 2026-10-21 goes further: a request that includes temperature, top_p or top_k at any value returns a 400 error on every model, and requests without them sample with the defaults, including a temperature of 1.0. On older models such as Claude Sonnet 4.6, the guide warns that a request that lowered temperature for more consistent output loses that effect and that "no other setting replaces it"; teams that depend on a non-default value can keep that request on the older 2023-06-01 version (Migrating to API version 2026-10-21). For current models Anthropic recommends prompting to guide behavior and the effort parameter to control thinking depth.

That shift makes instruction quality more important than ever. If you used to rely on a low temperature to keep answers terse or on-format, you now need explicit instructions, examples and structured outputs; our developer guide to prompt engineering covers those techniques in depth.

Google Gemini

Google's Gemini 3 guide strongly recommends keeping temperature at its default of 1.0 for all Gemini 3 models, warning that setting it below 1.0 may lead to looping or degraded performance, particularly on complex math and reasoning tasks (Gemini 3 developer guide). Newer models go further: Google's text-generation and prompting pages state that Gemini 4 and Gemini 3.6 and later models ignore custom values for temperature, top_p and top_k, return deprecation warning headers, and will return 400 errors in a future release (Gemini text generation). Thinking depth is controlled with thinking_level instead.

Open-Weight Models

Open-weight models still expose the full sampling toolkit, but their publishers increasingly publish recommended values rather than leaving them to guesswork. OpenAI's gpt-oss repository recommends sampling with temperature 1.0 and top_p 1.0 (gpt-oss README); see our overview of GPT-OSS 20B capabilities for how that model behaves in practice. Qwen's Qwen3 model card recommends temperature 0.6, top-p 0.95, top-k 20 and min-p 0 for thinking mode, and temperature 0.7, top-p 0.8, top-k 20 and min-p 0 for non-thinking mode, and it explicitly warns against greedy decoding because it can cause performance degradation and endless repetition (Qwen3-8B model card). The common thread across vendors is that reasoning models are tuned for sampling near their defaults, not for temperature 0.

Local Runtimes: Ollama and vLLM

In Ollama, sampling parameters are set in a Modelfile or per request. The documented defaults are temperature 0.8, top_k 40, top_p 0.9, min_p 0.0 and repeat_penalty 1.0 (disabled), though an individual model's Modelfile can override them. vLLM's SamplingParams default to temperature 1.0, top_p 1.0, top_k 0 (all tokens) and min_p 0.0, but by default the server loads the model's own generation config from its repository, so a model's recommended values can apply even when your request does not set them (vLLM model configuration). Setting the generation config to vllm switches back to vLLM's built-in defaults. Knowing which layer sets the effective value is the first step in debugging unexpected output.

Choosing Temperature by Use Case

The right temperature depends on the model, the data and the desired outcome. Before choosing a value, confirm the model accepts one at all; on many 2026 reasoning models the correct setting is simply to omit the parameter. Where temperature is adjustable, the vendor guidance cited above points to a few consistent patterns:

• Extraction, classification and code: lower temperatures for more focused, repeatable output, combined with structured outputs or schemas for format. On reasoning models that fix sampling, rely on schemas and explicit instructions instead.
• Retrieval-grounded answers: keep sampling conservative and let retrieved evidence, citations and instructions carry factual accuracy. Temperature cannot fix missing context.
• General chat and drafting: start from the model's default, which is 1 on OpenAI's and Anthropic's APIs and the recommended setting for Gemini 3, and change it only if evaluation shows a benefit.
• Brainstorming and creative writing: higher temperatures for more variety, paired with top-p or min-p truncation to keep coherence, and a human or automated filter afterward.
• Open-weight reasoning models: use the publisher's recommended values from the model card, such as Qwen3's thinking-mode settings or gpt-oss's temperature 1.0, and avoid greedy decoding where the card warns against it.

Treat these as starting points, not answers. The same nominal value behaves differently across model families, so a short evaluation on your own prompts is worth more than any generic table. When sampling decisions sit inside a broader model selection, routing or deployment question, our LLM solutions team can help design the evaluation and configuration around your use case.

Optimizing LLM Output: A Tuning Playbook

A disciplined process keeps temperature decisions evidence-based rather than a matter of taste:

1. Confirm what the model accepts. Check the model page for supported sampling parameters, defaults and reasoning-effort options before changing anything.
2. Define your metric. Decide whether you will judge outputs by human ratings, rubric scores, exact match, format compliance or novelty, and align evaluation with goals.
3. Change one knob at a time. Sweep temperature across a few values while holding top-p constant, or vice versa, in line with OpenAI's advice to alter one rather than both.
4. Add truncation if needed. If outputs are too wild, tighten top-p or add min-p where available; if they are too bland, loosen truncation or raise temperature slightly.
5. Constrain format separately. Use system instructions, explicit schemas, structured outputs or tool calling for predictable structure, rather than relying on low temperature to enforce format.
6. Compare outputs on a validation set. Run each candidate setting multiple times, because a single sample hides variance.
7. Document defaults. Record temperature, top-p, penalties, effort level and API version for each product surface, along with the rationale, so settings do not drift silently across releases.

That last step matters more in 2026 than before, because a model upgrade or an API version bump can turn a working parameter into a 400 error or silently ignore it. A per-model configuration registry, checked in CI against vendor changelogs, prevents surprises. If you are wiring several models into one product, our AI integration services cover that kind of configuration and change management.

Measuring Quality Across Temperature Settings

To avoid purely subjective choices, combine several types of measurement:

• Content metrics: task rubrics, exact match, factuality checks, and citation precision and recall.
• Behavioral metrics: length adherence, format compliance and refusal accuracy.
• Human ratings: usefulness, correctness, tone and coherence.
• Stability: variance across repeated runs at a fixed setting.
• Cost and latency: more diverse outputs can increase downstream filtering or re-ranking work.

Interactions and Edge Cases

Short prompts magnify randomness, because the model relies more on its priors when context is thin; adding examples often helps more than lowering temperature. Long prompts that approach the context limit can degrade coherence regardless of sampling. When factuality is paramount, pair any temperature with retrieval and verification rather than trusting a low setting alone. Finally, remember that some services override sampling in certain modes, so if changing temperature does not change behavior, check whether the provider is ignoring the value.

Troubleshooting: Symptoms and Fixes

• 400 error mentioning temperature or top_p: the model or API version does not accept sampling parameters. Remove them and steer with prompts and the vendor's effort or thinking setting.
• Changing temperature has no visible effect: the provider may be ignoring the value, as Gemini's newest models do, or a server-side generation config may be overriding it. Check response headers, warnings and runtime configuration.
• Outputs too bland or repetitive: raise temperature slightly, loosen top-p, or apply a small presence or repetition penalty where supported.
• Outputs incoherent or off-brief: lower temperature, tighten top-p or add min-p, and strengthen the system prompt with examples.
• Loops or endless repetition on a reasoning model: move back to the vendor's default sampling; Google and Qwen both warn that low temperature or greedy decoding can cause this.
• Hallucinated facts: add retrieval grounding, constrain answers to tool outputs, and verify; temperature alone will not make output true.
• Inconsistent formatting: use structured outputs or JSON schemas, then allow more variety only inside free-text fields where it is safe.

Frequently Asked Questions

Conclusion: Temperature Is Still Central, but No Longer Always Yours to Set

Temperature remains a central concept for understanding how large language models trade coherence for diversity. By dividing logits before the softmax, it sharpens or flattens the distribution the sampler draws from, and combined with top-p, top-k, min-p and penalties it gives you fine control over the variety of generated text.

What changed by October 2026 is who controls it. On many frontier reasoning models from OpenAI, Anthropic and Google, the vendor now fixes or ignores sampling parameters and asks you to steer with prompts, structured outputs and reasoning effort. On open-weight models and older API models, temperature is still yours to tune, ideally starting from the publisher's recommended values and validating changes on your own prompts. Either way, there is no universally best temperature: treat sampling as a documented, tested configuration, not a guess.

If you need help choosing models, sampling settings and evaluation methods for a production LLM application, talk to Cognativ about your AI project.