It did not take long for me to sunset my Substack. I'd prefer to keep my thoughts, technical and not, in one place. Since writing my first post, I've read a lot more about consciousness, emergence, reductionism, and the philosophy of mind. I wanted to revisit the argument with those readings in mind.
My undergraduate education has been a mosaic. I joined Illinois with an interest in operator algebras and quantum Shannon theory. I realized QIT could quickly leave me hyper-specialized and unemployed outside of academia (although I doubt my passion for the field will ever truly fade; it is incredibly elegant). The majority of my four years have been spent at the LPNA researching tensor computations and contractions. I fell in love with applied mathematics research, spending days developing theory and weeks designing algorithms and testing whether they worked.
Recently, neuroscience has piqued my interest. I have wondered whether this is my excuse to participate in machine learning research without “leaving” physics, but the answer hasn't bothered me enough yet. Working in theoretical neuroscience, and more broadly the physics of learning, has made me less comfortable with how I understand reductionism. In trying to explain natural and artificial intelligence through physical theories, I keep encountering behavior that is best described at the level of the whole system. I find the usual formulations of both reductionism and holism unsatisfactory here.
I want to understand how emergent behavior can be real while remaining grounded in microscopic physics. The following is my current attempt to work through that question.
I. The Nonpartisan
Reductionism, in its textbook form, is the conviction that all macroscopic physical behavior can be explained and derived from governing microscopic laws. The ultimate reductive base, on this view, is the Standard Model and particle physics. Holism holds that a system develops behavior that its parts cannot fully describe.
Spiking neural networks often use models that are direct descendants of standard physical models. Integrate-and-fire neurons, the simplest possible model, are mathematically reduced to an RC circuit. When many of these components are connected, their synchronized firing gives rise to brain rhythms. This collective property cannot be located in any single neuron. A Hopfield network similarly settles into collective states that represent memories. The memory is the network's emergent state, which is ontologically distinct from the state of any individual neuron.
Theoretical neuroscientists use reduction to define the building blocks and collective dynamics to understand how the whole functions. For a while, I was content to say that progress required both approaches. I now find that answer unsatisfactory because it leaves the ontological question untouched. Reductionism and holism also make claims about the structure of reality. Is emergence a feature of our descriptions, or of reality itself?
II. Jaynes, Landauer, and the Thin Wall
In the Jaynesian reinterpretation of statistical mechanics, the Boltzmann distribution follows from an inference problem. Given a set of microstates and an observed macroscopic average, such as mean energy, we choose the maximum-entropy distribution because it introduces the least additional assumption. On this reading, the partition function has an epistemological role. Statistical mechanics tells us what a rational agent should believe about a system given limited information, without treating the Boltzmann distribution as an independent physical postulate.
This helps me understand why partition functions, free energies, and variational principles appear across physics, machine learning, and Bayesian inference. If statistical mechanics is a calculus of belief under constraint, then I would expect the same mathematics in learning problems with the same constraints. Distributing probability over discrete options by trading off expected compatibility against distributional uncertainty leads to the Gibbs measure. I have arrived at this independently in softmax attention, associative memory, and kernel regression. I think the convergence reflects a shared problem statement.
I find a purely epistemological reading harder to maintain when I consider Landauer's principle. On that reading, the partition function is a bookkeeping device, and thermodynamic quantities summarize our ignorance. Yet Landauer's principle implies that, in the standard erasure setting, resetting an initially unbiased bit to a fixed value releases at least \(k_B T \ln 2\) of heat into a thermal reservoir at temperature \(T\), on average. The cost follows from the second law. Whatever role information plays in our descriptions, physically processing it is subject to thermodynamic constraints.
Jaynes helps me think about statistical mechanics as inference, while Landauer makes it difficult to separate information processing from physical reality. I do not think either account is sufficient on its own. Any account of emergence I find satisfactory has to accommodate both.
III. Liquidity, and Other Real Things
I find it useful to begin with the distinction between weak and strong emergence.
In weak (or epistemic) emergence, macroscopic behavior may be unexpected or computationally intractable to derive, while remaining fully determined by the microscopic laws. Strong (or ontological) emergence attributes irreducible causal powers to the macroscopic system. Even a complete specification of the lower level and its laws would be insufficient to predict or explain the higher-level phenomenon.
For a while, I thought critical thresholds suggested something ontologically new. Brain rhythms emerge only above a certain number of IF neurons, and spontaneous magnetization appears in the Ising model only below a critical temperature. Does the existence of these thresholds mean that new kinds of things exist at the macroscopic scale?
An individual H\(_2\)O molecule is not liquid, however exhaustively we describe it. Liquidity is a property of water at the collective level, and it has causal consequences. It allows us to swim and makes it possible to drown. I take this to be something ontologically real that is absent from the description of an individual molecule. Statistical mechanics accounts for it through intermolecular forces and the thermodynamic limit, without introducing new fundamental laws. The same Hamiltonian describes a system in which a new collective property exists.
I distinguish novel ontology, the appearance of new entities or properties, from novel nomology, the need for new laws to govern them. My claim is that composition can produce the first without the second. Brain rhythms are real collective properties that the physics of coupled oscillators can account for using the equations governing individual neurons. Memories in a Hopfield network are also real, and the statistical mechanics of its spin interactions generates them without supplementary axioms.
I call this effective reductionism because effective field theory provides a model for the relationship I have in mind. An effective field theory captures dynamics at a particular energy scale by integrating out degrees of freedom at higher energies. It has coupling constants and organizing principles appropriate to its scale, while remaining derivable in principle from the more fundamental theory. The renormalization group provides the formal machinery for relating these descriptions. This allows us to describe how a theory can be explanatorily autonomous at its own scale while remaining grounded in microscopic physics.
I want to apply this relationship to emergence more generally. Higher-level patterns have explanatory and causal relevance beyond making calculations manageable for someone who cannot track \(10^{23}\) molecules. I take them to be real patterns in the lower-level substrate, borrowing Dennett's phrase. Their reality is compatible with a fundamental ontology that remains unchanged.
IV. Twin Truths
If the mathematics doesn't change, and the microscopic specification and fundamental laws suffice to derive every macroscopic phenomenon, why insist that higher-level entities are ontologically real rather than descriptively convenient? A strict reductionist such as Jaegwon Kim would argue that what I am calling ontological novelty is descriptive novelty. On this account, I have identified a useful, causally relevant pattern at a particular scale. Calling it a new constituent of reality would confuse a change in description with a change in what exists.
I initially thought this depended on the question being asked. A practical agent encounters liquidity as causally relevant, while a physicist studies the molecular dynamics underlying it. Both descriptions serve a purpose. Making what counts as real depend on who is asking and why conflicts with my realist commitments. As a physicist, I believe quantum tunneling is actually happening, regardless of whether someone finds it relevant. I want an account of emergence consistent with that commitment.
I find it more convincing to treat the molecular and macroscopic descriptions as two truths about the same reality, related by grounding. Liquidity is a real pattern in molecular dynamics, just as brain rhythms are real patterns in neural activity. Both are grounded in and fully derivable from the physics at the lower level. This gives the descriptions an ordering. The microscopic account is more fundamental because it grounds the macroscopic one, independently of which description is more useful for a particular question.
I am therefore a realist about higher-level patterns and a reductionist about their grounding. I take emergence to be a feature of reality, even when computational limitations make it difficult to derive. It remains weak in the nomological sense because it requires no additional laws or irreducible causal powers.
V. The Hard Test
Phase transitions, liquidity, brain rhythms, and Hopfield memories are relatively well-understood cases of reduction. I am less sure how this framework applies to consciousness.
Consciousness makes me return to the question of what distinguishes a real pattern from a convenient shorthand. If higher-level descriptions are derivative, what makes them explanatory rather than only predictively useful? I find this easier to answer when the explanandum is magnetization than when it is the experience of seeing red.
I am inclined to place consciousness in the same framework. Given my belief that everything is physical, consciousness should be a real pattern in neural dynamics governed by the same laws as any other physical phenomenon. The difficulty seems to come partly from the brain's scale and recurrent connectivity, which exceed those of any physical system we have successfully reduced. I am also interested in the problem of studying consciousness through our own conscious activity. My consciousness is discussing its own existence, so the activity I use to investigate it is itself part of what I am trying to explain.
VI. Mind the Gap
David Chalmers argues that consciousness poses a different problem from other cases of emergence. Suppose we had a complete neuronal simulation, resolving every spike, synapse, and neuromodulator at arbitrary resolution. We could, in principle, derive its functional properties, including its input-output relations and behavioral dispositions. Chalmers argues that this would still leave open why there is something it is like to be that system. Why should the neural activity involved in seeing red be accompanied by an experience of red? This is the “hard problem”. Complexity alone does not explain the difficulty. A weather system is enormously complex, but nobody thinks there is a hard problem of hurricanes.
I suspect the gap between neural description and felt experience is epistemic. Before statistical mechanics, physicists could experience, measure, and use heat without an account of how temperature arose from molecular motion. Kinetic theory connected the two descriptions. I expect that some future theory could make a similar connection between neural activity and experience, although I cannot yet explain how.
Temperature already had a functional characterization before its reduction. It equilibrates between bodies in contact, relates to pressure and volume through the gas laws, and causes mercury to expand. Boltzmann showed that mean molecular kinetic energy satisfies these roles. The reduction worked because both the macroscopic concept and its microscopic candidate could be characterized in third-personal, publicly observable terms.
Consciousness is harder to characterize in these terms because the explanandum includes the experience of seeing red. A description of how the brain processes wavelengths may leave that experience unexplained. If no functional characterization captures this first-personal quality, it is unclear what a future theory would identify with the neural description. Temperature did not present this difficulty.
Dennett and the Churchlands approach this by treating first-personal qualities as functional properties. On this account, “what it is like” can ultimately be characterized through dispositions to report, discriminate, and react, all accessible to third-personal description. Consciousness would then be reducible in the same way as temperature. Another possibility is that we lack the bridging concepts needed to connect neural processes with experience. A future theory might make that connection as intelligible as the relation between kinetic energy and temperature. Our current position would then resemble physics before entropy was formulated.
I lean toward the second possibility because physics has often progressed by developing concepts that made an apparent explanatory gap tractable. Before the renormalization group, phase transitions were well documented empirically, but the appearance of long-range order in systems with short-range interactions remained theoretically mysterious. Universality classes, fixed points, and scaling dimensions provided concepts for understanding it that were absent from the earlier theory. I suspect consciousness may require a comparable conceptual development. The historical examples give me a reason to pursue this possibility, but they cannot establish that it will succeed.
VII. Ouroboros
In the other cases of scientific reduction I have discussed, the system being studied and the system doing the studying are distinct. The physicist is not made of the gas whose temperature she measures. A neuroscientist studying brain rhythms can, in principle, stand outside the system and observe it without that observation being constituted by the process under investigation. I find this separation harder to maintain for consciousness because the neural activity that constitutes my experience of thinking about consciousness is the phenomenon I am trying to explain.
I wonder whether self-reference could impose principled limits on what a conscious system can explain about itself. Gödel's incompleteness theorems and results concerning computation provide examples of limits in formal systems. A sufficiently powerful formal system cannot prove its own consistency, and a universal Turing machine cannot, in general, decide whether an arbitrary program will halt. I am unsure how far these results apply to empirical neuroscience. Brains are not formal systems in the logician's sense, but there may still be principled limitations on how fully a conscious system can model the processes that constitute its consciousness.
A complete reductive model of consciousness would need to capture the neural dynamics that generate phenomenal experience. Constructing and verifying the model is itself a conscious process, which means the model must, in some sense, account for its own construction. The explanandum therefore includes the process of explaining it. I do not know whether this loop creates a genuine impossibility or a very difficult engineering problem.
Even if a conscious system cannot fully model itself, I would still regard consciousness as grounded in physics. The limitation could belong to what the system can explain from within, while its physical grounding remains intact. Certain explanatory projects might therefore remain incomplete for the system attempting them.
Coda
I entered theoretical neuroscience expecting a straightforward reductionist program. Working in the field made me think more carefully about why explanations at several scales remain useful, and what connects them.
I take liquidity, brain rhythms, and perhaps consciousness to be real features of the world. They arise through composition and scaling and are grounded in and derivable from the lower-level physics. This allows new kinds of things to exist without additional fundamental laws. Descriptions at different scales can therefore concern the same reality while differing in how fundamental they are.
Effective field theory is the model I have in mind. Brain rhythms are best described through coupled oscillators, while the oscillator description is grounded in and derivable from ion-channel physics. The higher-level description has its own vocabulary without becoming independent of the lower-level account. I call this effective reductionism because it lets me work at the scale relevant to a problem while retaining a commitment to the physical grounding of that scale.
I do not yet know whether consciousness fits this account. I suspect the explanatory gap reflects something missing from our theory, and I want to understand what a theory capable of closing it would require.
- Traditionally credited for founding atomism with his student Democritus, Leucippus' theories were fundamental to the later school of reductionism.
- A single neuron in a Hopfield network has a state, +1 or −1. A memory is a global fixed point of the energy landscape, a collective property of \(N\) coupled spins. Decomposing it into a sum of individual contributions loses the collective property being described.
- In his 1957 papers, “Information Theory and Statistical Mechanics”, Jaynes develops maximum entropy as a method of inference. It produces the least biased distribution consistent with known constraints. The Boltzmann distribution then follows as a theorem, without being introduced as a separate physical postulate.
- I develop these convergences in detail in a separate post, "Attention Seeking," which derives softmax attention independently from free-energy minimization, modern Hopfield networks, sparse distributed memory, and Nadaraya-Watson regression.
- “Effective” refers to effective field theory in physics. An effective theory captures the relevant dynamics at a given energy scale by integrating out degrees of freedom at higher energies. It has its own vocabulary while remaining derivable from a more fundamental theory. I believe this mathematical relationship between levels also provides a way to understand emergence across physics, neuroscience, and mind.
- P.W. Anderson's “More is Different” (1972) argues that each level of complexity requires its own organizing principles. I am interested in whether this is a methodological claim about how to study systems or an ontological claim about what exists. The framework in this essay is partly my attempt to work through those readings.
- I want to understand why these derivations converge. They may share an inference problem under constraints, as the Jaynesian reading suggests. The convergence could also reflect a computational structure in nature, as a Landauer-inspired reading might suggest, or a mathematical necessity about exponential families that is prior to both physics and computation. This remains an open question in my research.
- Landauer's bound relates the average heat released into a thermal reservoir to the decrease in a memory's entropy. It assumes an initially thermal reservoir at temperature \(T > 0\), initially uncorrelated with the memory, and microscopically reversible dynamics of the combined memory and reservoir. For an ideal two-state memory, resetting an initially unbiased bit to a fixed value reduces its entropy by \(k_B \ln 2\), so the average heat released is at least \(k_B T \ln 2\), where \(k_B\) is Boltzmann's constant. For biased bits or imperfect erasure, the bound depends on the actual entropy reduction.
- In The Devil in the Details, Batterman gives an anti-reductionist interpretation of universality and the renormalization group. He argues that the insensitivity of macroscopic behavior to microscopic details gives the macroscopic account explanatory autonomy that the microscopic description alone cannot capture. Butterfield offers a reductionist response, which I favor. Batterman's objection still challenges the relationship between levels that I am proposing.
- Dennett's “Real Patterns” (1991) argues that a pattern is real if it allows for predictions that would be lost if the pattern were eliminated from our descriptions. On this criterion, liquidity is real because “the liquid will flow downhill” is a prediction you lose if you restrict yourself to molecular-level vocabulary. I think this deflationary criterion is sufficient for treating higher-level patterns as ontologically real.
- Kim's causal exclusion argument challenges non-reductive physicalism. If every mental event has a sufficient physical cause, the mental cause is either identical to the physical cause (reductive identity) or causally redundant (epiphenomenalism). I accept the identity claim at the level of grounding while maintaining that the higher-level description identifies a real pattern. I am unsure whether this gives me a third option or amounts to Kim's reductionism.
- I treat methodological naturalism as a well-justified working axiom. The history of science has consistently rewarded the assumption that phenomena have natural explanations. That inductive record gives me a reason to keep making the assumption, but it cannot prove that every phenomenon will admit a naturalistic explanation.
- I am inclined to treat the hard problem as partly a pseudo-problem generated by the circularity of our phenomenal vocabulary. We draw the language used to specify consciousness from the domain we are trying to explain. The word “feel” in “why does this feel like something?” already presupposes the phenomenal character that demands explanation. I suspect this makes part of the problem linguistic, while leaving the remaining explanatory difficulty unresolved.
- Possible sources of bridging concepts include Global Workspace Theory (Baars, Dehaene), Integrated Information Theory (Tononi), predictive processing accounts (Seth, Friston), higher-order theories (Rosenthal), and attention schema theory (Graziano). Each proposes a different approach, and none has achieved the consensus that statistical mechanics enjoys.
- Gödel's results concern formal systems of sufficient expressive power and say nothing directly about physical systems. The brain is a dynamical system subject to noise, bounded resources, and thermodynamic constraints. Related limits in computability theory, including Rice's theorem and the halting problem, provide somewhat closer analogies for physical computational systems. Their relevance to consciousness still requires an argument.
- I am still working through computational complexity theory results regarding self-reference and self-simulation. I would like to write about Turing machines, the halting problem, Rice's theorem, Kleene's recursion theorem, and Löb's theorem, alongside a thought experiment about simulation on a hypothetical computer with infinite hardware capacity and unlimited compute.
- These questions also motivate my work on statistical mechanics, machine learning theory, and the philosophy of mind. My technical work studies tensor-theoretic interpretations of transformers and statistical-mechanical accounts of attention. I believe these fields converge on the same mathematical objects because they share underlying inference problems, and I want to understand that connection more precisely.
References
- Information Theory and Statistical Mechanics
Jaynes, E. T. (1957). - Information Theory and Statistical Mechanics. II
Jaynes, E. T. (1957). - Irreversibility and Heat Generation in the Computing Process
Landauer, R. (1961). - Neural networks and physical systems with emergent collective computational abilities
Hopfield, J. J. (1982). - More Is Different
Anderson, P. W. (1972). - Real Patterns
Dennett, D. C. (1991). - Multiple Realization and the Metaphysics of Reduction
Kim, J. (1992). - What Is It Like to Be a Bat?
Nagel, T. (1974). - Facing Up to the Problem of Consciousness
Chalmers, D. J. (1995). - Renormalization Group and Critical Phenomena. I. Renormalization Group and the Kadanoff Scaling Picture
Wilson, K. G. (1971). - Neurophilosophy: Toward a Unified Science of the Mind-Brain
Churchland, P. S. (1986). MIT Press. - Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I
Gödel, K. (1931). - The Devil in the Details: Asymptotic Reasoning in Explanation, Reduction, and Emergence
Batterman, R. W. (2002). Oxford University Press. - Less is Different: Emergence and Reduction Reconciled
Butterfield, J. (2011).