I am (finally) taking a course on statistical physics, and have found that the field connects a multitude of my research interests in an oddly1 elegant way. In this post, I want to share some overlapping ideas between statistical mechanics, (softmax) self-attention, regression, and neuroscientific theories of memory. My hope is that, after reading this, one appreciates the lack of walls between these fields in the same way that I do.
I. Hedging Your Tokens
Consider a physical system with \(N\) discrete states, where the \(j\)-th state carries an energy \(E_j\). Once the system settles into thermal equilibrium with a heat bath of temperature \(T\), the probability of finding it in state \(j\) is given by the Boltzmann distribution: \[ p_j = \frac{e^{-\beta E_j}}{Z},\quad \text{where } Z = \sum_{k=1}^N e^{-\beta E_k} \text{ and } \beta = \frac{1}{k_B T}. \] The inverse temperature \(\beta\) determines how strongly energy differences matter, while the partition function \(Z\) turns those relative preferences into a probability distribution. It also stores the thermodynamic behavior of the system in a compact form. For example, the average energy follows from \[ \langle E \rangle = -\frac{\partial \log Z}{\partial \beta}. \]
The quantity we will keep returning to is the Helmholtz free energy, \[ F = -\frac{1}{\beta} \log Z = \langle E \rangle - TS, \] where \(S\) is the entropy. Energy pulls probability toward a small set of favorable states, while entropy spreads it back across the available possibilities; free energy records the compromise the equilibrium distribution has struck. A macroscopic observable \(O\) is then a thermal average over those microscopic states: \[ \langle O\rangle = \sum_{j=1}^N p_j O_j = \frac{1}{Z} \sum_{j=1}^N O_j e^{-\beta E_j}. \] With these objects, we can describe (almost)2 everything in equilibrium statistical mechanics. At high temperature, energy differences are washed out and the measure spreads toward uniformity. As the temperature falls, the same landscape becomes legible: probability gathers around its lowest valleys and progressively freezes out the rest.
Now move from a heat bath to a transformer processing a token sequence. A query arrives with \(N\) context tokens available to it, each carrying a key and a value, but it can return only one vector. The mechanism must therefore allocate a unit budget of attention across the context: it should favor tokens whose keys align with the query, while retaining enough spread to avoid turning every uncertain comparison into a brittle winner-take-all decision.
We can make that tension precise by assigning the \(j\)-th token an energy relative to the query, \[ E_j = -q^\top k_j. \] High query-key alignment becomes low energy. If \(p\) is any distribution over the context, its free energy is \[ F_\beta[p] = -\sum_j p_j q^\top k_j + \frac{1}{\beta}\sum_j p_j\log p_j. \] The first term rewards compatibility. The second is negative Shannon entropy, so it charges the query for concentrating too abruptly. This is the same competition we met in the physical system, except that the energy landscape now lives over context tokens.
Let \(Z_q=\sum_\ell\exp(\beta q^\top k_\ell)\), and let \[ p_j^\star = \frac{\exp(\beta q^\top k_j)} {Z_q} \] denote the corresponding softmax distribution. The free-energy functional can be rearranged into \[ F_\beta[p] = -\frac{1}{\beta}\log Z_q + \frac{1}{\beta}D_{\mathrm{KL}}\!\left(p\,\|\,p^\star\right). \] The second term is nonnegative and vanishes only at \(p=p^\star\), so softmax is the unique minimizer. The exponential form is forced by this particular pairing of linear compatibility and Shannon entropy; a different regularizer would produce a different allocation rule. Within those assumptions, though, the transformer and the thermal system have chosen the same finite Gibbs distribution.3
Derivation: minimizing the free energy
Enforce \(\sum_j p_j=1\) with a Lagrange multiplier: \[ \mathcal L(p,\lambda) = -\sum_j p_j q^\top k_j + \frac{1}{\beta}\sum_j p_j\log p_j + \lambda\left(\sum_jp_j-1\right). \] At an interior optimum, \[ \frac{\partial\mathcal L}{\partial p_j} = -q^\top k_j+\frac{1}{\beta}(1+\log p_j)+\lambda=0. \] Thus \(p_j\propto\exp(\beta q^\top k_j)\), and normalization supplies the partition function. Substituting this expression back into \(F_\beta[p]\) gives the KL decomposition above.
Each token also carries a value vector \(v_j\). Once the free-energy problem has chosen its distribution, retrieval is simply the thermal expectation of those values. Writing the keys and values as columns of \(K\) and \(V\), \[ \langle v\rangle = \sum_jp_j^\star v_j = \frac{\sum_jv_j\exp(\beta q^\top k_j)} {\sum_\ell\exp(\beta q^\top k_\ell)} = V\,\operatorname{softmax}\!\left(\beta K^\top q\right). \] Scaled dot-product attention folds an effective inverse temperature \(1/\sqrt{d_k}\) into its scores, while query-key normalization and learned scales expose that control more directly. In either case, temperature determines how far the query hedges: low \(\beta\) produces a broad average, while high \(\beta\) sends the distribution toward the most compatible token.
II. Do You Remember?
Softmax has now appeared as an equilibrium distribution over context positions, but the weighted average it produces has another, older interpretation. An associative memory stores patterns by content rather than by address: instead of naming where a memory lives, we present a partial or corrupted cue and let the dynamics decide what the cue resembles. The keys provide addresses, the query supplies the cue, and the values carry what can be recovered from those addresses. In that language, the attention output \(\sum_j\alpha_jv_j\)4 is a retrieved memory.
The canonical physical model of associative retrieval is the Hopfield network. Let \(s_i\in\{-1,1\}\) denote the state of neuron \(i\), and let \(\xi^\mu\in\{-1,1\}^N\), for \(\mu=1,\ldots,M\), be the stored patterns. The standard Hebbian5 couplings are \[ J_{ij} = \frac{1}{N}\sum_{\mu=1}^M\xi_i^\mu\xi_j^\mu \quad (i\ne j), \qquad J_{ii}=0, \] with energy \[ E(s)=-\frac12\sum_{i,j}J_{ij}s_is_j. \] Starting from a noisy cue, the network updates one spin at a time according to its local field: \[ s_i\leftarrow \operatorname{sign}\!\left(\sum_{j\ne i}J_{ij}s_j\right). \] Symmetry of \(J\) makes the energy a Lyapunov function, so these asynchronous updates descend until the network reaches a fixed point.
Derivation: why asynchronous updates lower the energy
Suppose only spin \(i\) changes, from \(s_i\) to \(s_i'\), and define \(h_i=\sum_{j\ne i}J_{ij}s_j\). Because \(J\) is symmetric and \(J_{ii}=0\), every term unaffected by \(s_i\) cancels: \[ \Delta E = E(s')-E(s) = -(s_i'-s_i)h_i. \] The update chooses \(s_i'=\operatorname{sign}(h_i)\). If the spin stays put, \(\Delta E=0\); if it flips, the new spin agrees with the field and \(\Delta E<0\). This monotonic argument relies on sequential updates and does not transfer unchanged to arbitrary synchronous ones.
Retrieval becomes easier to discuss once we measure it. The overlap between the current state and pattern \(\mu\) is \[ m_\mu(s)=\frac{1}{N}(\xi^\mu)^\top s. \] An overlap near one means that the state has recovered that pattern, while an overlap near zero means that little of it remains. Near a target memory, its contribution to the local field supplies the retrieval signal; the other \(M-1\) patterns contribute crosstalk. As the load grows, that accumulated noise eventually overwhelms the basin that was meant to correct the cue.
This is the setting behind the familiar critical load \(\alpha_c\approx0.138\), where \(\alpha=M/N\). The number describes the zero-temperature retrieval phase of the standard Hebbian model with independent, balanced random patterns in the thermodynamic limit. It is not a universal line separating perfect memory from total failure: the boundary moves with the pattern distribution, update rule, retrieval criterion, and tolerated error, and spurious minima can exist below it. What the calculation identifies is the point at which an extensive-overlap retrieval state loses stability under those assumptions.
Modern Hopfield networks change the interaction rather than accepting that crosstalk as fixed. Krotov and Hopfield replaced the quadratic interaction with higher-order terms, and Demircigil et al. studied the exponential limit, proving exponential capacity for random patterns under corresponding separation conditions. Ramsauer et al. then introduced a continuous-state energy whose update can be written in the language of attention.
Let \(X=[x_1,\ldots,x_M]\in\mathbb R^{d\times M}\) contain the stored patterns and let \(\xi\in\mathbb R^d\) be the current query state. Up to an additive constant, the energy is \[ E(\xi) = \frac12\|\xi\|^2 - \frac{1}{\beta}\log\sum_{\mu=1}^M \exp\!\left(\beta x_\mu^\top\xi\right). \] Its gradient is \[ \nabla E(\xi) = \xi-X\,\operatorname{softmax}\!\left(\beta X^\top\xi\right), \] so a stationary point satisfies the fixed-point equation \[ \xi^\star = X\,\operatorname{softmax}\!\left(\beta X^\top\xi^\star\right). \] The quadratic term is essential: without it, the energy is generally unbounded below as \(\|\xi\|\) grows, and the gradient no longer yields this fixed point.
Derivation: the modern Hopfield update
Differentiating log-sum-exp cancels the prefactor \(1/\beta\): \[ \nabla_\xi \left[ \frac{1}{\beta}\log\sum_\mu e^{\beta x_\mu^\top\xi} \right] = \frac{\sum_\mu x_\mu e^{\beta x_\mu^\top\xi}} {\sum_\nu e^{\beta x_\nu^\top\xi}} = X\,\operatorname{softmax}(\beta X^\top\xi). \] Writing the energy as a convex quadratic minus a convex log-sum-exp gives the concave-convex update \[ \xi_{t+1} =X\,\operatorname{softmax}(\beta X^\top\xi_t). \] Under the conditions in Ramsauer et al.'s convergence result, this iteration lowers the energy and approaches its stationary set. It is a fixed-point update, not an ordinary gradient step with the quadratic term discarded.
The update is attention with the stored patterns acting as both keys and retrieved values. Ordinary transformer attention keeps the same weighting algebra but permits a separate value matrix, \[ \operatorname{Attn}(q,K,V) = V\,\operatorname{softmax}(\beta K^\top q). \] Once \(V\) differs arbitrarily from \(K\), the output need not descend the same scalar energy. The equivalence remains exact at the level of the association weights, while the literal Lyapunov interpretation belongs to the tied-pattern construction.
The resulting landscape can support three qualitatively different fixed-point structures. If no pattern separates cleanly from the rest, the update can settle near a global average. A cluster of similar patterns can instead form a metastable basin that averages over the subset, while a sufficiently isolated pattern supports a fixed point close to that pattern alone. Increasing \(\beta\) sharpens the weights, but temperature does not choose the regime by itself; pattern norms, separations, and memory count determine which basins exist. Ramsauer et al. found more global averaging in early transformer layers and more partial averaging in higher layers, which supports the memory interpretation without putting every attention head into one universal metastable phase.
The modern Hopfield route begins with an energy and discovers attention in its fixed point. Kanerva's sparse distributed memory reaches a similar retrieval rule from the opposite direction, by counting how neighborhoods overlap in the Hamming cube \(\{0,1\}^n\). A memory has an address \(p_a^\mu\) and a pointer \(p_p^\mu\), while fixed hard locations store superpositions of the pointers written near their addresses. Writing activates every hard location within Hamming radius \(d\) of a pattern; reading activates those within the same radius of the query \(\xi\).
A pattern closer to the query has written to more of the locations the query will read. Its effective weight is therefore an intersection count: \[ \mathcal I(t;d,n) = \left|O_n(p_a,d)\cap O_n(\xi,d)\right|, \qquad t=d_H(p_a,\xi). \] With one such weight for each stored pattern, the read rule becomes \[ \xi^{\mathrm{new}} = g\!\left( \frac{\sum_{p\in P}\mathcal I(d_H(p_a,\xi);d,n)\,p_p} {\sum_{p\in P}\mathcal I(d_H(p_a,\xi);d,n)} \right). \] The threshold \(g\) converts the averaged pointer back to a binary pattern. The smooth-looking weight has emerged from discrete read and write decisions. For a more tactile account of that mechanism, I still highly recommend Trenton Bricken's Mormon-missionary analogy from his doctoral defense.6
Derivation: counting the SDM intersection
Suppose the two ball centers differ in \(t\) coordinates. A candidate hard location can differ from the first center in \(a\) of those coordinates, and it can flip \(b\) of the \(n-t\) coordinates where the centers agree. Its distances to the centers are \(a+b\) and \(t-a+b\), so \[ \mathcal I(t;d,n) = \sum_{a=0}^{t}\sum_{b=0}^{n-t} {t\choose a}{n-t\choose b} \mathbf 1\!\left\{ a+b\le d,\; t-a+b\le d \right\}. \] No point can lie in both balls when \(t>2d\), which makes the intersection exactly zero beyond that boundary.
Bricken and Pehlevan showed that, over the close-pattern region carrying most of the read weight, the nonzero intersection counts are approximately exponential in center distance: \[ \mathcal I(t;d,n)\approx c_1e^{-c_2t}. \] This is a local fit rather than an identity over the entire cube. The count bends away from the fit as the balls separate, then drops to zero at \(t>2d\). What matters for the connection to attention is that binary addresses also admit an exact bipolar embedding, \[ \hat u=\frac{2u-\mathbf 1}{\sqrt n}, \qquad d_H(u,v)=\frac n2\left(1-\hat u^\top\hat v\right). \] Substituting this relation into the exponential approximation yields \[ \mathcal I \approx c_3\exp\!\left(\beta\hat p_a^\top\hat\xi\right), \] and the common factor \(c_3\) disappears when the weights are normalized. The continuous read rule then takes the familiar form \[ \tilde\xi^{\mathrm{new}} \approx P_p\,\operatorname{softmax}\!\left(\beta P_a^\top\tilde\xi\right). \] Binary SDM and arbitrary learned vectors do not share an exact geometry, so extending this result requires normalized addresses and a fitted \(\beta\). The common object is the normalized exponential kernel, not a claim that SDM secretly thermalizes.
Keeler's comparison with the classical Hopfield network sharpens the distinction. A Hopfield network can be recovered as a special SDM construction by setting the write radius to \(d_{\mathrm{write}}=0\), making hard locations coincide with stored patterns so that \(r=m\), and replacing the thresholded read with unthresholded bipolar similarity. The read radius is not also set to zero. Hopfield retrieval weights every pattern by its signed similarity to the cue, while canonical SDM assigns no weight once the relevant neighborhoods stop intersecting. Two memory systems can therefore arrive at closely related retrieval operators while retaining very different geometries underneath them.
III. Kernels of Truth
We have reached normalized exponential weighting through equilibrium and through associative memory. The third route begins with a quieter problem: estimating a function near a point where we have observations but no parametric model. Yu Sun and collaborators made the connection to attention explicit, and the derivation here follows a recent vignette by Dhruv Pai at Tilde Research.
Suppose we observe input-output pairs \((x_i,y_i)\) and want to predict the response at a new point \(x_0\). A constant model ignores the inputs and minimizes \[ \sum_{i=1}^n\|y_i-c\|^2, \] which returns the global mean. It is a perfectly sensible answer to the wrong question, since it forgets where \(x_0\) sits relative to the data. The smallest useful change is to remain constant only locally, weighting each observation by a kernel \(K_h(x_i,x_0)\) whose bandwidth \(h\) determines what counts as nearby: \[ \hat c(x_0) = \arg\min_c\sum_{i=1}^n K_h(x_i,x_0)\|y_i-c\|^2. \] The solution is the Nadaraya-Watson estimator, \[ \hat f(x_0) = \frac{\sum_iK_h(x_i,x_0)y_i} {\sum_jK_h(x_j,x_0)}. \] Instead of fitting one global function and evaluating it everywhere, the estimator rebuilds a weighted average around each query. At this point the memory language is already peeking through: observations are stored, similarity determines relevance, and prediction retrieves an average of their values.
Derivation: from local regression to attention
Write \(w_i=K_h(x_i,x_0)\). Differentiating the weighted loss gives \[ \nabla_c\sum_iw_i\|y_i-c\|^2 = -2\sum_iw_i(y_i-c). \] Setting this gradient to zero yields \[ \hat c(x_0)=\frac{\sum_iw_iy_i}{\sum_iw_i}, \] with the scalar derivation applying coordinatewise to vector-valued responses.
Now relabel \(x_i\) as a key \(k_i\), \(y_i\) as a value \(v_i\), and \(x_0\) as a query \(q\).7 Choose the Gaussian kernel \[ K_h(k_i,q) = \exp\!\left(-\frac{\|q-k_i\|^2}{2h^2}\right). \] For unit-normalized queries and keys, \[ \|q-k_i\|^2=2(1-q^\top k_i). \] Letting \(\tau=h^2\), the kernel becomes \[ K_h(k_i,q) = e^{-1/\tau}\exp(q^\top k_i/\tau). \] The first factor is common to every key and cancels when the weights are normalized.
Collecting the keys and values as columns of \(K\) and \(V\), substitution now gives \[ \hat f(q) = \frac{\sum_i\exp(q^\top k_i/\tau)v_i} {\sum_j\exp(q^\top k_j/\tau)} = V\,\operatorname{softmax}\!\left(\frac{K^\top q}{\tau}\right). \] Softmax attention is Gaussian Nadaraya-Watson regression when queries and keys are unit normalized. The bandwidth is \(h\), while \(\tau=h^2\) is its square and plays the role of temperature, so \(\beta=1/\tau\). Query-key normalization makes this relationship explicit by normalizing the vectors and learning the remaining logit scale.8
Without unit normalization, exponential dot-product attention still acts as a normalized similarity smoother, but it is no longer the same isotropic Gaussian kernel in Euclidean distance. That distinction matters because the learned query and key projections decide which points count as neighbors before the averaging begins. Nadaraya-Watson describes the aggregation rule once that geometry has been chosen; it does not explain how the transformer learns the geometry.
The regression picture contributes a failure mode that the thermodynamic and memory pictures leave implicit. A small bandwidth follows the nearest values closely and becomes sensitive to sparse or noisy context. A large bandwidth reduces that sensitivity by averaging more examples, but it can blur distinctions the prediction needs. Temperature therefore controls a statistical bias-variance tradeoff as well as an energy-entropy one. The formula has stayed still, while the question we know to ask of it has changed.
The three routes meet at one operator: \[ \frac{1}{\beta}\log\sum_j e^{\beta s_j} = \max_{p\in\Delta_N} \left[ \sum_jp_js_j+\frac{1}{\beta}H(p) \right]. \] Statistical mechanics reads the maximizing distribution as equilibrium occupancy. A modern Hopfield network places the same log-sum-exp inside an energy whose fixed point retrieves stored patterns. Sparse distributed memory obtains approximately exponential weights by counting shared Hamming neighborhoods, and kernel regression reaches the normalized average after choosing a Gaussian notion of locality. The common skeleton is exact in some steps and approximate in others, which is more interesting than pretending that the mechanisms are identical.
This also narrows the strongest claim we can make about softmax. It is the unique solution to a linear allocation problem regularized by Shannon entropy, and Gaussian Nadaraya-Watson regression reproduces it under the normalization assumptions above. It is not the singular solution to inference in general. Other entropies and other kernels produce different weighting rules, some sparse, some compactly supported, and some better suited to geometries that dot products describe poorly.
Studying for this post has been both rewarding and mildly destabilizing. I began by asking whether the universe computes, expecting the answer to depend on where we draw the boundary between a physical dynamic and an algorithm. I am less convinced that this is the useful boundary now. Across these examples, the same mathematics returns whenever a system scores alternatives, resists committing too quickly, and must compress many possibilities into one response. The universe need not be running a transformer for softmax to keep appearing; it is enough that equilibrium, memory, and estimation repeatedly confront that shape of problem.
- "Oddly" is a massive understatement. Some of the correspondences between statistical mechanics, neuroscience, and learning theory have genuinely changed how I think about what the universe is doing. The boundary between a physical process and a computation is much thinner than I was taught to believe, and I'm not sure it exists at all.
- Our specific setup is known as the canonical ensemble, where the system can exchange energy with a heat bath but has a fixed number of particles and volume. In situations where the particles themselves can move in and out of the system, we would use the grand canonical ensemble, which introduces a chemical potential \(\mu\) to account for particle exchange. For the purposes of the connections I wish to draw here, the canonical ensemble suffices.
- Here I use "Gibbs distribution" and "Boltzmann distribution" interchangeably for this finite canonical ensemble. Gibbs measures also exist for finite systems; in the infinite-volume setting, they require a local consistency construction because a single global partition function may not exist.
- Here, \(\alpha_j\) represents the attention weight for the \(j\)-th key-value pair.
- Hebbian learning is a principle that states that the synaptic strength between two neurons increases if they are activated together. In the context of the Hopfield network, the Hebbian rule encodes the stored patterns into the network's connectivity, allowing for associative retrieval based on partial or noisy cues. Famously, "neurons that fire together, wire together."
- Bricken is also a leading researcher in Anthropic's interpretability effort. His work and research taste were instrumental in my decision to pursue a research career in mechanistic interpretability. Thanks, Trenton!
- It is understandable to wonder if we are forcing a connection with this relabeling. Note that we are preserving the functional roles and relationships of every variable. NW regression takes no interest in the names of variables; it only requires a similarity space for conditioning with \(x_i\) and \(x_0\) and a target object to be averaged with \(y_i\). Attention follows the same structure, allowing for a relabeling.
- QKNorm normalizes query and key vectors before taking their dot product, then replaces the usual \(1/\sqrt{d_k}\) factor with a learned scale. Henry et al. introduced the method in 2020. Bricken and Pehlevan later arrived at the same normalization requirement through their SDM analysis, so the convergence is conceptual rather than chronological.
References
- Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K. and Bengio, Y. (2014). - Attention is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. and Polosukhin, I. (2017). - Neural networks and physical systems with emergent collective computational abilities
Hopfield, J. J. (1982). - Spin-glass models of neural networks
Amit, D. J., Gutfreund, H. and Sompolinsky, H. (1985). - Dense Associative Memory for Pattern Recognition
Krotov, D. and Hopfield, J. J. (2016). - On a Model of Associative Memory with Huge Storage Capacity
Demircigil, M., Heusel, J., Löwe, M., Upgang, S. and Vermet, F. (2017). - Hopfield Networks is All You Need
Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M. and others (2021). - Sparse Distributed Memory
Kanerva, P. (1988). - Attention Approximates Sparse Distributed Memory
Bricken, T. and Pehlevan, C. (2021). - Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T. and others (2023). - Comparison between Kanerva's SDM and Hopfield-type neural networks
Keeler, J. D. (1988). - On Estimating Regression
Nadaraya, E. A. (1964). - Smooth Regression Analysis
Watson, G. S. (1964). - Learning to (Learn at Test Time): RNNs with Expressive Hidden States
Sun, Y., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G. and others (2024). - Regression is All You Need
Pai, D. (2025). - Query-Key Normalization for Transformers
Henry, A., Dachapally, P. R., Pawar, S. and Chen, Y. (2020).