Temperature and LLMs: The Secret Behind Word Choices

Large Language Models

Statistical physics provides a framework for understanding how systems transition from deterministic behaviors to generative behaviors, which is essential in the context of large language models (LLMs). The central question of this discipline is to determine the probability that a system occupies a given state among all possible states. While one might be tempted to consider the energies of states as probabilities, this would be incorrect. Indeed, energies can take values such as 5, -3, or 127, but this gives no indication of their probability. Probabilities, unlike energies, must be positive and sum to 1.

To solve this problem, we must stop thinking in terms of probabilities and instead convert the energy of each state into a weight. This weight is calculated using the exponential:

• weight = e^(-E_i/kT)

This formula does not generate probabilities, but rather a relative importance. States with low energy are assigned large weights, while those with high energy receive exponentially smaller weights. At this stage, these weights are not yet normalized; they simply indicate the strength with which each state competes with the others.

To transform these weights into probabilities, one simply divides the weight of each state by the sum of all weights:

• P(state) = e^(-E_i/kT) / Σ(e^(-E_j/kT))

This formula is known as the Boltzmann distribution. Although the equation may seem complex, it boils down to two simple operations: the exponential translates energy into a relative preference, and the denominator rescales these preferences to form a valid probability distribution.

The Role of Temperature in the Boltzmann Distribution

Temperature plays a crucial role in the Boltzmann distribution, but it does not alter the energies themselves. Instead, it changes the significance of the energy differences between possible states. Take, for example, three states with energies of 0, 1, and 2. At a very low temperature, such as T=0.1, even small energy differences become significant in the exponential. The state with the lowest energy captures almost all the probability, making the others nearly impossible. In contrast, at a higher temperature, such as T=10, these energy differences are heavily compressed, producing almost identical weights and a distribution close to uniform.

Here are two tables illustrating this concept (assuming k=1):

| Energy E | Weight e^(-E/T) if T=0.1 | Probability (%) | |-----------|---------------------------|------------------| | 0 | 199.99 | 99.99 | | 1 | 0.000045 | 0.0045 | | 2 | ~0 | 0 |

| Energy E | Weight e^(-E/T) if T=10 | Probability (%) | |-----------|--------------------------|------------------| | 0 | 136.7 | 0.9053 | | 1 | 3.2 | 0.8193 | | 2 | 0.1 | 0.1 |

The states themselves have not changed; their energies remain 0, 1, and 2. What changes is the relative importance of these energy differences.

The LLM Equivalence: Vocabulary as States

To understand how this applies to large language models, we simply need to rename our variables. In the context of physics, a system could be a gas, and the "states" are the different energy levels that its molecules can occupy. In an LLM, the system is the precise moment when the model must predict the next word. The "states" are the entire vocabulary of the model, which consists of tens of thousands of words or tokens competing to be chosen next.

When an LLM processes a prompt, it does not immediately generate probabilities. Its final neural layer produces raw, unnormalized numbers for each word in its vocabulary. In machine learning, these raw scores are called logits. One word may have a logit of 12.5, another -3.2, and another 0.8.

The crucial conceptual bridge is that logits represent the model's energy, but inverted.

Softmax is Boltzmann in Disguise

To transform these raw, unbounded logits into a clean probability distribution that sums to 1, machine learning engineers use the Softmax function. If we explicitly include the temperature parameter (T), the Softmax equation is as follows:

• P(word_i) = e^(z_i/T) / Σ(e^(z_j/T))

Does this sound familiar? It’s exactly the same mathematical principle. We replace the negative energy -E with our positive logits +z. We exponentiate to transform the scores into relative weights, and then we divide by the sum of all weights to normalize them. Softmax is nothing more than the Boltzmann distribution under a cloak of computing.

How Temperature Controls the "Heat" of Text

Given that the mathematics are identical, the temperature parameter behaves the same way as in thermodynamics. It does not alter the underlying beliefs of the model, as the raw logits remain unchanged. Instead, it dictates the rigor with which the model applies these beliefs.

Let’s examine how adjusting the thermostat modifies the behavior of an LLM:

• Absolute Zero (T=0): The greedy deterministic state: As the temperature approaches zero, even microscopic differences in logits are amplified to infinity. If "apple" has a logit of 5.01 and "banana" has 5.00, dividing by a tiny temperature exaggerates this difference of 0.01. "Apple" will capture 99.99% of the probability mass. The model becomes a deterministic machine, always choosing the word with the highest score. This is ideal for tasks requiring strict logic, such as writing Python code, solving mathematical equations, or extracting JSON data. There is no room for randomness.

• Room Temperature (T=1): The baseline: At T=1, the temperature effectively disappears from the equation. The model produces the exact probability distribution it learned during training. The text flows naturally, balancing expected grammar with the natural variance of human language.

• High Heat (T > 1): Creative chaos: When we increase the temperature, we compress the differences between logits. The exponential function flattens out. The word with the highest score loses its absolute dominance, and lower-scoring words begin to capture significant probability mass.

Think of statistical physics: at high temperatures, gas molecules have so much thermal energy that they can easily access higher and more difficult energy states. In an LLM, a high temperature gives the algorithm the energy needed to access mathematically improbable words. The model takes risks. It becomes creative, poetic, and unpredictable.

If you push the temperature too high (for example, T=2), the distribution becomes almost completely uniform. The model loses its grip on context and begins to generate incoherent text, almost random, closer to noise than language.

A Small but Important Detail: T = 0 is Not Really Zero

There is a subtlety to know if you have ever set temperature=0 in an API call. Mathematically, T=0 should be undefined, as you would be dividing each logit by zero, which the formula simply cannot handle. So what happens when you request it?

In practice, LLM inference libraries do not literally compute e^(z/0). Instead, T=0 is treated as a special instruction: skip the softmax formula entirely and directly choose the logit with the highest score. This is called greedy decoding, always producing argmax(z), the token with the highest raw score, without exponentials, without normalization, without a probability distribution.

The result resembles an extremely cold Boltzmann distribution, with almost all probability concentrated on a single state. But there is a difference in how this is achieved. Mathematically, you would reach this result gradually, cooling T towards zero and observing the distribution sharpen step by step. In code, there is no gradual cooling, just a shortcut that jumps straight to the answer.

Observation: Real Logits from GPT-2

Everything above has been demonstrated with theoretical numbers, three abstract states, and some invented logits for "apple" and "banana." Let’s replace this with something real. GPT-2, a small and freely available language model, will provide us with real logits for a real sentence, and we can observe how softmax transforms them before our eyes, at four different temperatures.

The setup is simple: provide the model with the prompt "I am tired, I will take a __," and ask it what comes next. Internally, it produces a raw logit for each word in its vocabulary, tens of thousands of numbers. For visualization, we will keep only the five highest candidate logits and renormalize them with softmax at T = 0.1, 0.5, 1.0, and 2.0.