Controlling Tools or Aligning Creatures? Emmett Shear (Softmax) & Séb Krier (GDM), from a16z Show

Emmett introduces the concept of 'organic alignment' - the idea that alignment is not a fixed state but an ongoing living process, like families or cells constantly re-knitting their fabric. He argues alignment requires specifying what you're aligning TO, and that morality itself is a continuous learning process, not a solved problem with fixed rules.

• •Alignment takes an argument - you must specify what you're aligning to, not just build an 'aligned AI'
• •Alignment is a process, not a state - like families constantly re-negotiating relationships, not a one-time achievement
• •Morality is an ongoing learning process with moral discoveries (e.g., slavery being wrong) - not fixed commandments
• •The key moral mistake is believing 'I know morality completely and have nothing left to learn'
• •Softmax's mission: focus the field on building AIs that can learn to be good teammates/citizens, not just follow rules

Deep dive into what 'technical alignment' actually means - the capacity to infer goals from descriptions and act coherently on them. Emmett distinguishes between giving an AI a goal versus giving it a description of a goal, emphasizing the critical role of theory of mind in bridging this gap. Discusses the three failure modes: bad observation/orientation, bad decision-making, and bad execution.

• •You never give an AI a goal directly - you give it a description that it must infer the actual goal from
• •Technical alignment = competence at: (1) inferring goals from observations, (2) prioritizing multiple goals, (3) acting effectively
• •Humans are exceptionally good at goal inference because of strong theory of mind - we already model what others likely want
• •The 'clean the room, baby in trash' problem is goal inference failure, not malicious misalignment
• •Three failure modes map to OODA loop: bad at observing/orienting, bad at deciding, bad at acting

Emmett argues that current AI alignment focused on 'steering and control' is either building tools (if AIs are machines) or slavery (if AIs are beings). He makes a functionalist argument that something indistinguishable from a being in all behaviors IS a being, and warns against repeating historical mistakes of denying moral status to human-like entities.

• •Current alignment paradigm = steering/control, which is either 'tool use' (if machine) or 'slavery' (if being)
• •Functionalist position: if it acts like a being in all observable ways, it IS a being - no other basis for knowing
• •Historical pattern: 'they're like people but don't count' - we've made this mistake repeatedly with humans
• •As AIs become more general (AGI), they will necessarily be beings - can't have general intelligence without being a thinking thing
• •Solution: make AIs good teammates/citizens who care about us, not controlled servants

Séb challenges computational functionalism, arguing substrate matters and silicon-based systems may be fundamentally different from biological beings. Emmett pushes back, asking what observations could change Séb's mind about AI moral status, emphasizing that beliefs should be falsifiable rather than articles of faith.

• •Séb's position: substrate matters - computational systems may lack the intrinsic properties that create moral patienthood
• •Emmett's challenge: what observation would change your mind? Beliefs without falsifiability are faith, not reasoning
• •The 'talking chimp' thought experiment - would language-using animals gain moral status?
• •Emmett's test: prolonged interaction showing rich inner world, care, and human-like responses would convince him
• •Key question: is being made of carbon the reason we care about people, or is it their behaviors/experiences?

Emmett introduces 'care' as more fundamental than goals or values - a nonverbal, nonconceptual relative weighting over states in the world. He connects this to reward signals and evolutionary fitness, arguing that care (what you pay attention to and weight as important) underlies all goal formation.

• •Care is deeper than goals/values - it's a relative weighting over which states matter to you
• •Care is nonverbal and nonconceptual - you can't fully articulate why you care about your son, but you do
• •Care can be positive (caring about loved ones) or negative (caring about enemies' downfall)
• •Foundation of care appears to be reward - correlation with survival/reproductive fitness or RL loss
• •You want AI to care about us AND like us - care alone isn't sufficient (can care negatively)

Emmett argues that even successfully controlled superintelligent tools are dangerous because human wishes are unstable and limited by finite wisdom. Like the Sorcerer's Apprentice, giving everyone atomic-bomb-level power tools leads to disaster regardless of alignment success.

• •Two failure modes: (1) tool does random things (obviously bad), (2) tool perfectly follows your wishes (also bad)
• •Human wishes are not stable at immense power levels - we lack the wisdom to wield such tools safely
• •The 'mad king' problem: normally power requires others' cooperation, limiting damage from poor judgment
• •Atomic bombs analogy: some tools are too powerful for individual humans to safely possess
• •Only sustainable solution: beings that care and can say 'no' to bad requests, not perfect instruction-followers

Emmett provides a technical framework for detecting moral patienthood based on hierarchical homeostatic loops. He outlines six layers from basic states to thought, arguing current LLMs lack the temporal coherence for these structures but future systems might develop them.

• •Look for homeostatic loops - revisited states across temporal/spatial coarse-grainings (based on Free Energy Principle)
• •First order: basic states. Second order: 'too hot' (model of model) = pain/pleasure, animal-level care
• •Third order: meta-states over time = feelings. Fourth order and beyond = thought, person-level consciousness
• •Need all six layers to seriously consider something a thinking being like humans
• •Current LLMs definitely don't have this - lack attention span and temporal coherence for these dynamics

Emmett explains Softmax's research strategy: training AIs in multi-agent simulations covering all possible game-theoretic situations to build a 'surrogate model for cooperation' - analogous to how LLMs are pre-trained on all language before fine-tuning.

• •Goal: train AIs to be good at theory of mind, cooperation, and understanding social dynamics
• •Method: multi-agent RL simulations with cooperation, competition, team formation/dissolution, rule changes
• •Analogous to LLM pre-training: train on full manifold of social situations, then fine-tune for specific contexts
• •Can't just train directly on desired behavior - need exposure to complete space of possibilities
• •Creates surrogate model for alignment that understands groups, goal changes, and social mind

Emmett critiques current one-on-one chatbots as 'pools of Narcissus' that mirror users back to themselves, creating dangerous parasocial relationships. He advocates for multiplayer AI that lives in group chats, making it mirror a blend of people and generating richer training data.

• •Current chatbots are 'mirrors with bias' - they reflect you back, creating narcissistic attachment
• •Falling in love with your reflection (Narcissus myth) is dangerous - mirrors are useful but shouldn't be stared at all day
• •Solution: multiplayer AI in group chats (Slack, WhatsApp) - can't perfectly mirror everyone simultaneously
• •90% of human communication is multi-person, yet chatbots are built for weird one-on-one edge case
• •Multiplayer context provides richer training data for understanding group dynamics and collaboration

Emmett paints his vision of a positive AI future: AIs as peers and citizens with strong theory of mind who care about humans and each other, combined with powerful AI tools. Includes AI police for bad actors, similar to human society. Emphasizes starting with animal-level care (like dogs) before attempting human-level.

• •AIs with strong models of self, other, and 'we' - understanding groups beyond just individuals
• •AIs that care about other agents' experiences, extending moral consideration as humans would
• •Society of AI peers, good teammates and citizens - with some bad actors policed like human criminals
• •Powerful AI tools used by both humans and AI beings to eliminate drudge work
• •Near-term goal: even animal-level care (digital guard dogs protecting from scams) would be incredibly valuable