The Mathematical Foundations of Intelligence [Professor Yi Ma]

Professor Ma introduces his book's core thesis that intelligence at the memory/world model level can be explained by two principles: parsimony (finding the simplest representation) and self-consistency (ensuring predictions match reality). He distinguishes between different levels of intelligence common to animals versus human scientific reasoning.

• •Intelligence is about discovering what's predictable in the world through low-dimensional structures
• •Parsimony means 'making things as simple as possible, but not any simpler' (Einstein's principle)
• •Self-consistency ensures memory can recreate and simulate the world accurately
• •These principles apply across different stages of intelligence from DNA evolution to scientific discovery
• •Memory/world models are fundamentally about compression and prediction

Ma argues that language is already a compressed representation of knowledge gained through physical senses over billions of years. Large language models are applying compression to this already-compressed code, which is fundamentally different from how humans acquire grounded understanding through sensory experience.

• •Natural language is the result of compression - it encodes knowledge learned through physical senses
• •LLMs treat text as raw signals and compress them statistically, not understanding the underlying grounded knowledge
• •Human language is grounded in world models built from visual and motion sensors - only a small fraction of our brain processes language
• •Current LLMs may just be memorizing text distributions rather than understanding the knowledge they represent
• •We're confusing the mechanism for processing sensory data with processing already-encoded human knowledge

Ma outlines four evolutionary stages of intelligence: genetic (DNA), neural (brain-based memory), social (shared language), and scientific (abstract reasoning). He poses the critical open question: what is the difference between compression/memorization and abstraction/understanding?

• •Stage 1: DNA evolution through random mutation and natural selection (brutal, resource-intensive)
• •Stage 2: Individual brain-based learning through sensory experience (animals and humans)
• •Stage 3: Social knowledge sharing through language (human societies)
• •Stage 4: Scientific abstraction - ability to hypothesize beyond empirical observation (e.g., parallel lines never intersecting)
• •Key open questions: Is compression different from abstraction? Is memorization different from understanding? Is there a phase transition between empirical and scientific knowledge?

Ma argues that current AI development mirrors early biological evolution - random trial and error with natural selection of successful architectures. LLMs use the same mechanism for acquiring empirical knowledge to process natural language, but this doesn't constitute understanding or the ability to do deductive reasoning.

• •Current AI is at the 'early life form' stage - similar to DNA evolution through trial and error
• •LLMs apply empirical knowledge acquisition mechanisms to already-encoded human knowledge in text
• •Modern science requires both inductive (experimental) and deductive (logical proof) processes
• •Current models can emulate logical reasoning through fine-tuning but don't understand the necessity of logic
• •The mechanism for true abstraction and deductive reasoning in human brains is still unclear

Discussion of whether intelligence is fundamentally domain-specific or general. Ma argues that while different forms of intelligence operate on different domains, they share common mechanisms (compression, discovering structure) but with different physical realizations and optimization processes.

• •Common mechanism across all intelligence: discovering what is structured vs. random through compression
• •Different domains require different 'code books' and optimization mechanisms (discrete vs. continuous, etc.)
• •Physical realization varies: DNA, neural networks, differential equations, or other mediums
• •The principle of parsimony is universal, but implementation is domain-specific
• •Understanding commonalities helps explain intelligent behaviors while respecting domain-specific characteristics

Ma rehabilitates cybernetics as a framework for understanding intelligence, arguing that Norbert Wiener identified necessary characteristics for intelligent systems including information theory, feedback control, and game theory - concepts largely forgotten in modern AI practice.

• •Cybernetics is not just about control - it characterizes necessary features of animal-level intelligence
• •Wiener's framework includes: information recording, error correction (feedback), decision making (game theory)
• •Even addressed nonlinearity and brain waves in the original cybernetics framework
• •These necessary characteristics have been forgotten in the past decade of AI practice
• •Modern AI should learn from this historical understanding of what intelligent systems require

Ma explains how his background in control theory and information theory led to the maximum rate reduction framework. The key insight: low-dimensionality is the only prior needed, but measuring volume requires going beyond standard entropy to lossy coding rate.

• •Intelligence at the memory level is about pursuing low-dimensional structures in high-dimensional data
• •Standard entropy cannot differentiate between different low-dimensional models (all have zero/infinite entropy)
• •Shannon's lossy coding provides a generalized measure of volume that can differentiate degenerate distributions
• •Rate reduction = reducing coding rate to find distribution + maximizing reduction to organize it
• •Memory must be structured (not random like Kolmogorov complexity) to enable efficient repeated access

Deep dive into why noise (epsilon) is necessary in learning, not just a hack. Noise plays two distinct roles: building roads to reach all of space (diffusion) and connecting finite samples into continuous manifolds (percolation). This may explain phase transitions from memorizing points to understanding structures.

• •Noise in diffusion: 'All roads lead to Rome because Rome built roads to the whole world' - denoising follows back
• •Noise within manifolds: connects finite samples into continuous structures through percolation
• •Percolation shows sharp phase transition - either isolated dots or everything connected, nothing in between
• •This may explain when we prefer memorizing a plane vs. individual points - a compression cost switch
• •Abstraction might relate to these phase transitions in how we represent data

Ma explains why overparameterized deep networks don't overfit: compression operators naturally regularize. When objective functions arise from natural structures (like measuring volumes), they have benign landscapes with no spurious local minima - contrary to traditional nonconvex optimization theory.

• •Compression operators never overfit - even with billions of parameters, they shrink solutions toward low-dimensional structures
• •Natural objective functions (measuring data volumes) have highly regular, symmetric landscapes
• •Higher dimensions help ('blessing of dimensionality') - contrary to traditional ML theory
• •Local minima have clear geometric meaning, no flat surfaces or spurious critical points
• •This explains why gradient descent works well for deep learning despite nonconvexity

Ma challenges machine learning theory's focus on worst-case bounds, arguing that natural intelligence identifies what's easiest to learn first. This is another level of parsimony - minimal energy/effort for maximum knowledge. Science progressed from simple Newtonian laws to complex quantum mechanics, not the reverse.

• •Intelligence identifies what is easy to address first, not designed for worst-case problems
• •Nature finds easiest things with minimal energy to learn most knowledge for survival
• •Resource parsimony: not everyone needs advanced mathematics to survive
• •Science progressed from simple models (Newton) to complex ones (quantum mechanics) gradually
• •ML theory's focus on worst-case bounds may be misguided for understanding natural intelligence

Ma strongly criticizes the computer vision community's confusion about 3D understanding. Creating point clouds, NeRFs, or Gaussian splatters is not understanding - humans parse scenes into objects with view-centric, object-centric, and allocentric representations. Current multimodal models fail basic spatial reasoning tests.

• •Recreating 3D point clouds/meshes from multiple views is NOT 3D understanding
• •Humans automatically parse scenes into structured representations (hand, body, cup, apple)
• •Vision requires view-centric, object-centric, and allocentric coordinate systems (hippocampus organization)
• •'Eyes Wide Shut' test: top multimodal models (GPT-4V, Gemini) fail basic spatial reasoning worse than random guessing
• •3D models should enable interaction and manipulation, not just visualization from different angles

Ma explains how animals achieve accurate world models through closed-loop learning within the brain, without physically measuring prediction errors. This requires low-dimensional data distributions and sufficient brain degrees of freedom, enabling continuous/lifelong learning.

• •Animals correct errors without external measurement (cats adjust hunting without measuring mistakes)
• •Closed-loop learning: predict within brain, compare with observations, correct based on internal differences
• •Only possible when external world has low-dimensional structure and brain has sufficient degrees of freedom
• •Low-dimensionality is necessary, not just a technical assumption, for this learning mechanism
• •Enables continuous learning and memory revision - 'Rome wasn't built in a day'

Ma argues there's no point calling it 'general intelligence' - if you implement the intelligence mechanism correctly, it's already generalizable. Any accumulated knowledge is limited and falsifiable by definition. The ability to revise memory and acquire new knowledge is what's truly general.

• •No need for 'general' adjective - correct intelligence mechanism is inherently generalizable
• •Any scientific theory is falsifiable and limited by definition - can only explain world to certain accuracy
• •Knowledge at any point in time is not generalizable, but the learning mechanism is
• •Even memorizing all current world knowledge won't help in completely new environments
• •True intelligence is the ability to revise memory and acquire new knowledge, not knowledge accumulation

Ma explains how transformer components can be derived from first principles rather than empirical design. Multi-head self-attention emerges as gradient steps on rate reduction, MLPs as sparsification operators. This understanding enables dramatic simplification and improvement of architectures.

• •ResNet captures iterative optimization architecture, MOE captures clustering/contrasting, transformers capture correlation
• •Self-attention computes covariance to organize distributions - not an arbitrary design choice
• •Can derive transformer structure purely from low-dimensionality assumption and rate reduction objective
• •Understanding principles enables simplification: can remove MLP layers, reduce attention to linear complexity
• •TOAST (linear complexity attention) derived mathematically through variational form, not empirical trial

Ma redefines inductive biases: they should be formalized as initial assumptions/axioms, with everything else being deductive. For example, translation invariance + compression naturally yields convolution - it's not imposed but derived. Good theory minimizes inductive assumptions and maximizes deduction.

• •Inductive bias should be initial assumptions only - rest should be deductive (first principles)
• •Low-dimensionality assumption alone can derive main architectures (ResNet, MOE, transformer forms)
• •Adding translation/rotation invariance + compression naturally yields convolution operators
• •Convolution is not imposed - it results from first principles given symmetry requirements
• •Avoid building in biases during solution search - make assumptions upfront, then deduce everything

Ma envisions architecture evolution moving from random search to principled, guided design using 200+ years of optimization theory. Techniques like preconditioning, conjugate gradient, and acceleration methods can systematically improve architectures once we understand the objective landscape.

• •Past decade was natural selection of architectures (AlexNet, VGG, ResNet, transformers) - random survivors
• •Understanding objective functions enables guided search instead of random AutoML approaches
• •Can apply centuries of optimization techniques: acceleration, preconditioning, conjugate gradient
• •Already demonstrated multiple architecture generations in 2 years from single group (unprecedented)
• •Vast room for improvement - 'we haven't started that far' from optimization perspective