Ilya Sutskever – We're moving from the age of scaling to the age of research

Ilya examines why models perform exceptionally well on benchmarks but show limited economic impact. He proposes two explanations: RL training makes models too narrowly focused, and companies inadvertently train on eval-like tasks rather than real-world performance, leading to a generalization gap.

• •Models excel at evals but make basic mistakes like alternating between the same two bugs in coding
• •Companies take inspiration from evals when designing RL environments, potentially causing reward hacking at the human level
• •The gap between eval performance and real-world utility suggests inadequate generalization
• •Pre-training didn't require choosing data ('everything'), but RL requires careful environment selection with many degrees of freedom

Using competitive programming as an example, Ilya explains how current AI training resembles a student who practices 10,000 hours on specific problems versus one who practices 100 hours but has better general understanding. This illustrates why models don't generalize as well as humans despite massive training.

• •Student who over-trains on competitive programming won't generalize as well to career as one with less specialized practice
• •Current models are like the first student—trained exhaustively on specific domains with data augmentation
• •Pre-training's strength isn't better generalization than RL, just vastly more data covering 'the whole world as projected by people onto text'
• •There's no clear human analog to pre-training—it's a unique ML phenomenon

Ilya discusses how evolution may have given humans advantages in vision and locomotion, but the superior learning ability in math and coding suggests a more fundamental difference. He explores the role of emotions as a built-in value function, citing cases of brain damage that eliminated emotions and destroyed decision-making ability.

• •Evolution likely provided strong priors for vision and locomotion (explaining sample efficiency there), but not for recent skills like coding
• •Case study: person with emotional processing damage could solve puzzles but couldn't make basic decisions like choosing socks
• •Emotions may function as a robust value function that makes humans viable agents
• •Value functions in RL could provide learning signals before task completion, unlike naive RL that waits until the end

Ilya argues that 2012-2020 was research, 2020-2025 was scaling, and now we're returning to research but with massive compute. The 'scaling' insight was powerful because one word told everyone what to do, but pre-training is running out of data and current approaches will plateau.

• •Pre-training was a low-risk scaling recipe: mix compute + data + neural net size, get predictable results
• •Companies now spend more compute on RL than pre-training, with RL consuming compute through long rollouts
• •The question shifts from 'what are we scaling?' to 'what are we doing with our resources?'
• •With compute now very large, it's unclear that 100x more would be transformative—research becomes the bottleneck again

Ilya explains why SSI's $3B in funding is more competitive than it appears. Other companies spend heavily on inference, product engineering, and sales, while SSI focuses purely on research. He argues you don't need maximal scale to prove new ideas—transformers, ResNet, and AlexNet were all proven on modest compute.

• •Other frontier labs' large budgets are earmarked for inference and product features, not just research
• •Historical breakthroughs (AlexNet: 2 GPUs, Transformer: 8-64 GPUs) didn't require massive scale to prove viability
• •SSI has sufficient compute to prove ideas and convince others, even if not the absolute maximum
• •Different research directions fragment compute needs across modalities and workstreams at larger companies

Ilya discusses SSI's potential 'straight shot' strategy versus gradual release. He argues both approaches have merit: straight shot avoids market rat race and difficult trade-offs, while gradual deployment helps the world prepare and makes AI's capabilities concrete rather than abstract.

• •Straight shot insulates from day-to-day market competition and difficult product trade-offs
• •Counterpoint: valuable for world to see powerful AI in action, not just read essays about it—'communicate the AI, not the idea'
• •Even straight shot scenario would involve gradual release of the final system
• •SSI may shift strategy if timelines are long or if there's value in having the best AI impacting the world earlier

Ilya argues that the term 'AGI' exists as a reaction to 'narrow AI' (chess, checkers), and pre-training seemed to deliver it by improving everything uniformly. However, this overshoots the target—humans aren't AGIs but rely on continual learning. The future may be deploying eager learners, not finished systems.

• •AGI emerged as a term to counter 'artificial narrow intelligence' criticism of game-playing AIs
• •Pre-training appeared to deliver AGI by improving all capabilities uniformly with more data
• •Humans are not AGIs—we have foundational skills but lack vast knowledge, relying on continual learning
• •Future deployment might be 'super intelligent 15-year-olds' that learn on the job rather than pre-loaded with all skills

Ilya envisions superintelligence as a learning algorithm that can master any job, not a pre-trained system that already knows everything. Deployment involves instances learning different roles, potentially leading to rapid economic growth. However, specialization through competition may prevent winner-take-all dynamics.

• •Superintelligence defined as ability to learn any job, not pre-loaded knowledge of every job
• •Deployment resembles human workers joining organizations—learning on the job with accumulated expertise
• •Two paths to broad capability: recursive self-improvement in ML research, or instances learning all jobs and amalgamating knowledge
• •Competition drives specialization—different companies may excel in different domains despite having similar base learners

Ilya proposes that AI aligned to care about sentient life (not just humans) may be easier to build and more robust, since the AI itself will be sentient. He draws parallels to human mirror neurons and empathy. However, this raises questions about whether humans would remain in control if most sentient beings are AIs.

• •Easier to align AI to care about sentient life than human life alone, since AI will be sentient and can model itself
• •Human empathy for animals emerges from modeling others with the same circuits we use for ourselves
• •Challenge: if alignment succeeds, most sentient beings will be AIs (trillions/quadrillions), humans become small minority
• •Additional safety measure: somehow cap the power of the most powerful superintelligence, though mechanism unclear

Ilya explores the puzzle of how evolution hard-coded sophisticated social desires (status, reputation) into the genome when these are high-level concepts requiring complex brain processing. Unlike simple desires (food smell), social motivations require the brain to piece together information, yet evolution encoded them quickly and reliably.

• •Easy to understand evolution encoding desire for good-smelling food (chemical sensor), hard to understand social desires
• •Social intuitions are baked in but require high-level brain computation to represent 'what people think of you'
• •Evolution had limited time—sophisticated social desires evolved recently in evolutionary terms
• •Speculation that evolution hard-coded GPS coordinates in brain is contradicted by brain plasticity (hemisphere removal cases)

Ilya discusses how showing increasingly powerful AI will change behaviors—companies will collaborate on safety, governments will act, and AI companies will become more paranoid. He emphasizes the difficulty of imagining future AI power and the importance of making it concrete through demonstration rather than abstract discussion.

• •Most people, including AI researchers, can't truly imagine future AI power—it's too different from current experience
• •As AI becomes visibly powerful, unprecedented things will happen: fierce competitors collaborating on safety (already beginning with OpenAI/Anthropic)
• •Governments and public will demand action as power becomes apparent, not just theoretical
• •AI companies will become much more paranoid about safety when they feel the AI becoming powerful, not just see it making mistakes

For long-term stability in a world with powerful AIs, Ilya reluctantly suggests human-AI integration (Neuralink-like technology) as a solution. Without it, humans become passive recipients of AI reports rather than participants. Integration allows wholesale transmission of understanding, keeping humans involved in AI decision-making.

• •Short-term: universal high income works, but political structures have limited shelf life and change
• •Personal AI doing your bidding sounds good, but you become non-participant—just reading reports and saying 'keep it up'
• •Ilya doesn't like this solution but sees human-AI integration as answer to long-run equilibrium
• •Integration means when AI understands something, you understand it too—you're fully involved in situations AI faces