Skip to content
Request a free consultation
What Is LLM Inference? How Models Generate Text and What It Costs

What Is LLM Inference? How Models Generate Text and What It Costs

A plain guide to how trained language models answer prompts: the prefill and decode phases, KV cache memory, TTFT metrics, serving choices and cost levers.

LLM inference is the work a trained large language model does every time someone sends it a prompt: it reads the input, then writes a reply one token at a time. Training happens once per model version; inference happens on every request, so it is where users feel speed and where an AI product spends money month after month.

This guide explains how inference works inside the model, which metrics describe it, how its two phases map to latency and to the bill, where it can run and which optimizations change the numbers.

What Is LLM Inference and How It Differs From Training

Inference is the runtime use of a finished model. Its weights stay fixed, and the only job left is to turn a given prompt and context into output tokens as quickly and cheaply as the system allows.

During training, a model sees huge volumes of text and adjusts billions of parameters until its predictions of the next token improve. That stage needs large clusters for weeks, and it ends with a checkpoint: a fixed set of weights. Inference loads that checkpoint and runs it forward only, with no learning. The model does not remember a conversation unless the application sends the earlier turns back as part of the next prompt.

The two stages also fail differently. A training run that is slow costs the team time. An inference system that is slow costs every user a wait and every request a share of hardware time. That is why serving engineers talk less about accuracy on a benchmark and more about tail latency, utilization and cost per request, while still checking that the answers stay correct.

How LLM Inference Works: Prefill, Decode and the KV Cache

A single request moves through tokenization, a prefill pass over the whole prompt and a decode loop that emits one token per step. Each stage stresses the hardware in a different way.

A top-to-bottom flow of four steps: tokenize the prompt, prefill pass, decode loop and detokenize the output.
Every request is read in one parallel prefill pass, then written one token at a time.

Tokenization and the prefill pass

The text is first split into tokens, the sub-word units the model was trained on, and mapped to numeric IDs. In the prefill pass the model processes all prompt tokens at once. Because every prompt token is known in advance, the GPU can compute them in parallel, so prefill is limited mainly by arithmetic capacity. Its output is the first generated token plus a stored record of the prompt's attention state.

The decode loop

Decode is autoregressive: the model produces one token, appends it to the sequence and runs again to produce the next, until it reaches a stop token or a length limit. Each step does relatively little arithmetic but must read the model's weights from GPU memory again, so decode is usually limited by memory bandwidth rather than compute. A long answer is slow mainly because it needs many sequential steps, and those steps cannot be run in parallel for the same request.

The KV cache

Attention lets each new token look back at every earlier token. Recomputing that history at every step would be wasteful, so servers keep the key and value tensors of past tokens in a key-value (KV) cache and only add entries for the new token. The cache saves compute but costs memory, and it grows with sequence length and with the number of concurrent requests. On a busy server, KV cache space, not model size alone, often decides how many users one GPU can handle at the same time.

How the Model Picks Each Next Token

At every decode step the model outputs a probability for each token in its vocabulary. The decoding strategy turns those probabilities into one choice, trading predictability against variety.

Greedy decoding always takes the most probable token. It is fast and repeatable, which suits extraction and classification, but it can loop or sound flat in longer text. Beam search keeps several candidate sequences alive and picks the best-scoring one at the end; it can improve coherence for short, structured outputs at the price of extra compute per request. Sampling draws from the distribution instead, shaped by temperature, top-k and top-p settings, so the same prompt can produce different wording.

These settings are part of inference, not training, which makes them one of the cheapest levers available. A model that seems unreliable may simply be sampling too loosely for the task. For factual or regulated outputs, conservative settings plus a validation step on the result are a sensible starting point; creative drafting can tolerate more randomness.

What Sets LLM Inference Latency, Throughput and Cost

Inference performance is described with a handful of metrics, and each one traces back to either the prefill or the decode phase. Knowing which phase drives a number tells you which lever can move it.

A two-by-two grid of four measures: time to first token, inter-token latency, system throughput and cost per request.
Measure first-token wait and per-token speed separately; each points to a different phase.

NVIDIA's LLM benchmarking metrics documentation defines time to first token (TTFT) as the wait from sending a query to receiving the first token, and inter-token latency (ITL), also called time per output token, as the average gap between later tokens. End-to-end latency is TTFT plus generation time. Tokens per second and requests per second measure throughput across all users, while per-user tokens per second measures what one person experiences.

The mapping is direct. Long prompts, large documents and big system instructions raise TTFT because they lengthen prefill. Long answers raise end-to-end latency because they add decode steps. High concurrency raises throughput but can lengthen each user's ITL, because requests share the same GPU memory bandwidth. Cost follows the same split: hosted APIs price input and output tokens separately, and self-hosted systems pay for the GPU hours that both phases consume.

PhaseWhat users noticeWhat drives itLevers a team controls
PrefillWait before the first wordPrompt and context lengthShorter prompts, prefix caching, retrieval that sends only relevant passages
DecodeSpeed of the streaming replyOutput length, memory bandwidthOutput limits, smaller or quantized models, speculative decoding
KV cacheSlowdowns under loadSequence length times concurrencyPaged memory management, context limits, request scheduling

A practical rule follows from the table: measure TTFT and ITL separately on your own prompt and answer lengths. A single average latency figure hides which phase is the problem and which fix will help.

Where LLM Inference Runs: API, Managed Endpoint or Self-Hosted

The same model can be served three broad ways. The choice decides who tunes the engine, how costs scale with traffic and how much control you keep over data, model versions and latency.

A hosted model API is the fastest start: the provider runs the hardware and the serving engine, and you pay per input and output token. You give up control over model versions, batching and hardware placement, and prompts leave your environment under the provider's terms.

A managed endpoint on a cloud platform runs a model you select, often an open-weight one, on dedicated capacity. You typically pay for instance time whether or not it is busy, so utilization matters, but you choose the model, the region and the scaling rules.

Self-hosting means running an inference engine such as vLLM, SGLang or TensorRT-LLM on your own GPUs or reserved cloud machines. It offers the most control over data and tuning and the most operational work: drivers, upgrades, monitoring and capacity planning. Teams sizing that hardware can use our guide to AI inference infrastructure considerations, which covers the planning questions this article leaves aside.

Options can be combined: an API for low-volume or frontier-model tasks, and a self-hosted open model for steady, high-volume or sensitive workloads. The deciding inputs are traffic shape, data rules and how much engineering time the team can commit.

LLM Inference Optimization Techniques That Change the Numbers

Optimization works at four levels: the model weights, the attention kernels, the serving engine's scheduler and the application's prompts. Each level targets a different bottleneck.

Four stacked layers: application prompts, serving scheduler, attention kernels and model weights.
Optimization works at four levels, and the biggest gains combine more than one.

Batching and paged KV memory

GPUs are efficient only when busy, so servers group requests. Continuous batching, also called in-flight batching, adds and removes requests at each decode step instead of waiting for a whole batch to finish. Its limit is KV cache memory, which naive allocation wastes through fragmentation. The PagedAttention paper behind vLLM (SOSP 2023) manages that memory in fixed-size blocks, like operating-system paging, and reports roughly 2 to 4 times higher throughput than earlier systems at the same latency.

Prefix caching

When many requests start with the same system prompt, documents or tool definitions, the engine can reuse the stored attention state for that shared prefix and skip part of prefill. Hosted providers expose this as discounted cached input tokens, which works only when the reused prefix matches exactly. Prompt order therefore affects both TTFT and cost.

Smaller numbers and smaller models

Quantization stores weights in fewer bits, for example 8-bit or 4-bit formats instead of 16-bit, which cuts memory use and the bandwidth each decode step needs. Accuracy can drop on some tasks, so it should be tested on your own data; our explainer on LLM quantization techniques covers the methods in detail. Knowledge distillation goes further by training a smaller student model to imitate a larger one, and pruning removes weights or attention heads the model barely uses.

Faster kernels, speculative decoding and parallelism

FlashAttention computes exact attention in tiles that fit in fast on-chip memory, reducing slow reads and writes to GPU high-bandwidth memory. Speculative decoding lets a small draft model propose several tokens that the large model verifies in one pass; its original 2022 paper reported 2 to 3 times faster sampling on T5-XXL with identical outputs. When a model is too large for one GPU, tensor parallelism splits each layer's matrices across devices and pipeline parallelism splits layers into stages, both at the cost of communication between GPUs.

LLM Inference Mistakes That Raise Cost or Hurt Quality

Most inference problems in production are not exotic. They come from measuring the wrong thing, ignoring output length or changing the model without checking what the change did to answers.

Trusting someone else's benchmark. Published throughput figures depend on prompt length, output length, batch size and whether caching was on. Results that do not match your traffic shape say little about your latency or cost.

Optimizing only the prompt side. Teams trim prompts but let the model write long, padded answers. Because decode is sequential, output length is often the larger driver of wait time; an explicit length limit or a structured output format can help more than prompt edits.

Breaking the cache by accident. Inserting a timestamp, user name or random ID at the start of every prompt changes the prefix and defeats prefix caching. Put variable fields after the stable instructions.

Quantizing or distilling without an evaluation set. A smaller or lower-precision model may be fine on average and wrong on the cases that matter. Keep a fixed set of real tasks and compare outputs before and after each change.

Adding slow guardrails late. Output filters and validation models also run at inference time. A safety layer slower than the generator adds a latency step users notice. Building the generation path, guardrails and evaluation together is the core of LLM solution development, rather than an afterthought.

LLM Inference in 2026: MoE Models, Multi-Node Serving and Cheaper Repeats

The core mechanics above are stable, but the workloads running on them keep changing. Industry benchmarks and provider documentation in 2026 show where inference effort is now being spent.

MLCommons' MLPerf Inference v6.1 round, summarized by its working group chairs on September 17, 2026, found that gpt-oss-120b, a mixture-of-experts (MoE) model, overtook Llama2-70b as the most popular datacenter benchmark for the first time. The round added end-to-end retrieval and agentic edge tests, recorded 16 multi-node submissions, and updated the gpt-oss-120b interactive, latency-bound scenario to allow speculative decoding. MoE models activate only part of their weights per token, which lowers compute per step but keeps the full model in memory, so memory capacity stays central to serving cost.

Serving engines track the same shift. vLLM's current documentation, checked in November 2026, lists continuous batching, chunked prefill, prefix caching, several speculative decoding methods and low-precision formats including FP8 and 4-bit FP4 variants. Providers, meanwhile, price repeated context more cheaply: Anthropic bills cache reads at 0.1 times the base input price, and less on some newer models, with a default 5-minute cache lifetime, and OpenAI says cached input can be discounted by up to 95%. OpenAI's Batch API also offers a 50% discount for jobs that can wait up to 24 hours. For agents that resend long context on every step, those cache and batch rules can matter more to the bill than the headline price per token. Matching them to the right hardware and hosting mix is part of AI infrastructure planning.

FAQs About LLM Inference

People researching LLM inference often ask the same practical questions about hardware, pricing, prompt caching and serving, so the short answers below collect them in one place.

Does LLM inference require a GPU?

Not always. Small and quantized models can run on CPUs or laptops with local tools, though more slowly. Serving many users or large models at low latency generally needs GPUs or other accelerators with high memory bandwidth.

How long does a prompt cache last?

It depends on the provider and model. Anthropic also sells a 1-hour cache, with writes billed at 2 times the base input price. OpenAI keeps cached prefixes for at least 30 minutes by default on GPT-5.6 and later, and offers 12-hour retention in its Responses API at a higher write rate.

How is LLM inference cost calculated?

Hosted APIs usually charge per million input tokens and per million output tokens, with lower rates for cached input or for batch jobs on some platforms. Self-hosted cost is GPU time divided by the tokens or requests that time actually serves.

What is chunked prefill?

Chunked prefill splits a long prompt's prefill into smaller pieces that the scheduler can mix with other users' decode steps. One very long prompt then delays everyone else less. vLLM lists it among its batching features.

Is inference the same as serving?

Inference is the model computation itself. Serving is the system around it: request routing, batching, scaling, authentication, monitoring and the API that applications call.

Treat Inference as a Running Cost You Can Measure

Inference is the part of an AI system that runs forever, so small, measured changes to prompts, output length, caching and hosting add up long after the model choice is made.

Start with your own traffic: log prompt and output lengths per request, and keep an evaluation set before changing models or precision. Then pick the serving option and the two or three optimizations that hit your actual bottleneck. If you want help working out where your inference time and spend go, talk to our team about your LLM inference setup.

Never miss a post

Practical Cognativ articles on AI, software delivery, data, ecommerce, and RAPID transformation.

  • Free weekly tech news
  • Nearly 1,000 readers
  • Unsubscribe in one click
  • Never sold or shared

As featured inForbesUSA TodayCEOWorld Magazine