Inference is using a trained model: you send input, weights stay frozen, output comes back. Training is the one-time, months-long process that creates the model; inference is every single use afterward — each chat reply, autocomplete, generated image. The economics split the same way: training is a fixed cost paid once, inference is a marginal cost paid on every request, which is why at scale the API bill — priced per token — is an inference bill.
What happens during LLM inference
Two phases with different bottlenecks. Prefill processes your entire prompt in parallel — compute-bound, and the reason long prompts cost money even before any output. Decode then generates the reply one token at a time, each step feeding on all previous ones — memory-bandwidth-bound, and the reason text streams word by word. The metrics you’ll see quoted map straight onto these: time to first token measures prefill, tokens per second measures decode. A context window stuffed to 500K tokens makes both phases slower and pricier — one more reason not to fill it just because you can.
Why inference got cheap — and an industry
Per-token prices have fallen orders of magnitude in a few years, driven by better hardware and a stack of software tricks: quantization (weights in fewer bits), batching (many users share one GPU pass), KV-caching (don’t recompute the prompt for every token), speculative decoding (a small model drafts, the big one verifies) and mixture-of-experts architectures that activate only a fraction of their weights per token. Cheap inference also spawned a market: inference providers (Groq, Together, Fireworks, DeepInfra and others) compete to serve open-weight models faster and cheaper than anyone else — same model, very different speeds and prices depending on who runs it.
Why it matters to you
When you pick a model you’re really picking an inference deal: quality per dollar at your latency tolerance. And the calculus changed in the reasoning era — “thinking” models spend thousands of extra tokens deliberating before answering, deliberately trading more inference compute for better answers. Fast, cheap inference is what makes that trade affordable.