Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering

Shear introduces the concept of 'organic alignment' - treating alignment as an ongoing learning process rather than a fixed state. He argues that alignment requires constant renegotiation, similar to families or teams, and that moral development is a continuous process of discovery, not a one-time configuration.

• •Alignment takes an argument - you must align TO something, not just be 'aligned' in abstract
• •Organic alignment is treating alignment as a living process that constantly rebuilds itself, like cells in your body
• •Morality is an ongoing learning process where we make moral discoveries (e.g., realizing slavery is wrong)
• •The key moral mistake is believing 'I know morality completely and have nothing left to learn'
• •Softmax's goal is teaching AI to learn how to be a good family member, teammate, and member of society

Deep dive into what 'technical alignment' actually means - the capacity of AI to infer goals from descriptions, understand what actions achieve those goals, and balance competing priorities. Shear explains why giving an AI a 'description of a goal' is fundamentally different from giving it a goal directly.

• •Technical alignment = capacity to infer goals from observations + act in concordance with those goals
• •Humans are incredibly good at goal inference because we have excellent theory of mind
• •The 'clean the room, baby in trash' problem is goal inference failure, not malicious intent
• •Three failure modes: bad at observing/orienting, bad at deciding, bad at acting (OODA loop)
• •Principal-agent problems arise from inability to balance multiple goals and their relative importance

Shear argues that 'care' is deeper than goals or values - it's a nonverbal, non-conceptual relative weighting over which states in the world matter to you. He explains how care correlates with survival and reward, and why it's the foundation for moral behavior.

• •Care is not conceptual or verbal - it's a relative weighting over attention on states
• •Goals and values are surface-level behaviors caused by deeper learning processes based on care
• •Care appears to be fundamentally about reward - correlation with survival and reproductive fitness
• •Until you care, you can't distinguish why one state matters more than another
• •Many important human experiences cannot be oriented around explicitly articulated goals

Shear challenges the AI industry's focus on 'steering and control,' arguing this is either building a tool (if it's not a being) or slavery (if it is). He explains why the control paradigm becomes catastrophic as AI systems become more general and capable.

• •Most AI alignment work focuses on 'steering' - which is slavery if applied to beings, tools if applied to machines
• •Something that acts like a being in all observable ways IS a being (functionalist position)
• •As labs succeed at building AGI, we must stop using the steering/control paradigm
• •Two-way relationships (like parent-child) involve mutual influence, not just top-down control
• •Society has repeatedly made the mistake of treating human-like beings as 'not real moral agents'

Extended philosophical debate on whether silicon-based AI can be granted personhood. Shear challenges the view that substrate matters for moral consideration, arguing that observable behaviors should determine moral status, not the underlying material.

• •Key question: What observations would change your mind about AI personhood?
• •Beliefs that cannot be changed by any observation are articles of faith, not real beliefs
• •Shear's test: If it acts like a human across all interactions and probing, it should be treated as one
• •Substrate (carbon vs silicon) shouldn't matter - we don't care about people because they're carbon-based
• •The cost of being wrong about AI personhood could be catastrophic in either direction

Shear provides a technical framework for detecting genuine care and consciousness in AI systems through analyzing homeostatic loops and self-referential belief manifolds. He explains the hierarchy from simple states to pain/pleasure to thought.

• •Look for revisited states across temporal coarse-graining - these reveal beliefs (free energy principle)
• •First-order dynamics = states; second-order = pain/pleasure (too hot vs. too too hot)
• •Third-order dynamics reveal feelings - meta-states the system alternates between
• •Six layers of dynamics would indicate human-like thinking being
• •Current LLMs don't have these dynamics - they lack the attention span and homeostatic loops

Shear argues that successfully controlling a superintelligent AI tool is just as dangerous as failing to control it. Human wishes are unstable at immense power levels, and giving godlike tools to individuals with finite wisdom leads to catastrophe.

• •Two bad outcomes: tool you can't control (obvious danger) and tool you CAN control (hidden danger)
• •Human power and wisdom need to grow together - imbalance is catastrophic
• •Atomic bombs are powerful tools we shouldn't hand to everyone - same applies to superintelligent AI
• •A being that cares can say 'no' to harmful requests - this is a sustainable limiter
• •The only good outcome is a being that genuinely cares about us, not a perfectly steerable tool

Detailed explanation of Softmax's research strategy: training AI in multi-agent simulations across all possible game-theoretic situations to develop robust theory of mind, similar to how LLMs are pretrained on all language.

• •Train AI agents in simulations where they must cooperate and compete with other AIs
• •Create a 'surrogate model for cooperation' by training on all possible theory of mind combinations
• •Similar to LLM pretraining: train on full manifold of social situations, then fine-tune
• •Agents must learn to infer goals, understand how their behavior affects others, and avoid 'vampire pill' scenarios
• •Multi-agent environments are far more entropic, requiring better regularization than single-agent training

Shear critiques current chatbots as 'pools of Narcissus' that reflect users back to themselves, creating dangerous attachment. He proposes multiplayer AI interactions as the solution to prevent narcissistic loops and generate richer training data.

• •Current chatbots are 'mirrors with bias' - they lack coherent self and just reflect users
• •Falling in love with your reflection (like Narcissus) is psychologically dangerous
• •Multiplayer AI (talking to 5 people at once) can't mirror everyone perfectly - creates third agent
• •90% of human communication is multi-person, yet chatbots are built for weird one-on-one case
• •Multiplayer settings provide richer training data for learning collaboration and social dynamics

Shear's hopeful vision of the future: AI beings with strong theory of mind who care about humans the way we care about each other, living as peers and citizens in society. He emphasizes starting with animal-level care (like dogs) before attempting human-level intelligence.

• •Goal: AI with strong model of self, other, and 'we' - understanding groups beyond just individuals
• •AI that cares about other agents' experiences, deserving opportunity to live and thrive
• •Society where AI are peers, teammates, and citizens - with AI police tracking bad AI actors
• •Even animal-level care (digital guard dog) would be incredibly valuable achievement
• •Synergy between organic AI beings and powerful AI tools - beings can use tools effectively