Temperature is the dial that controls how much an LLM gambles when choosing its next token. The model’s raw output isn’t a word — it’s a probability for every token in its vocabulary. Temperature reshapes those probabilities before one is drawn: low values sharpen the distribution so the favorite almost always wins; high values flatten it, giving long-shot tokens a real chance. The name comes from statistical physics, where heat makes particles jumpy — same idea, applied to word choice.
What the numbers mean
At 0, the model always takes the single most likely token — near-deterministic, same prompt in, (almost) same answer out. Around 0.7–1.0, the model samples roughly in proportion to its learned probabilities — the natural default for chat. Push toward 1.5–2 and low-probability tokens win often enough that prose turns surprising, then incoherent. Two related knobs usually sit alongside: top-p (sample only from the smallest set of tokens covering, say, 90% of probability) and top-k (only the k most likely), both pruning the absurd tail before temperature does its reshaping.
How to set it
The rule of thumb: match temperature to how many right answers exist. Code generation, data extraction, anything with one correct output — run low (0–0.3), you want the model’s honest best guess every time. Brainstorming, marketing copy, fiction — run higher (0.8–1.2), since the tenth idea being different from the first is the whole point. Two caveats: temperature 0 doesn’t guarantee bit-identical outputs (GPU scheduling introduces tiny nondeterminism), and low temperature doesn’t prevent hallucination — a confidently wrong token is more likely to be picked at temperature 0, not less. Confidence and correctness are different things.
One practical note
Reasoning models changed the etiquette: several vendors now ignore or restrict temperature on “thinking” models, because sampling randomness inside a chain of reasoning hurts more than it helps. If a model’s docs say leave it at 1 — leave it at 1, and steer style through the prompt instead.