How Intelligent Is AI, Really?

ARC Prize Foundation redefines intelligence as the ability to learn new things efficiently, not just solve hard problems. Based on François Chollet's 2019 paper, the benchmark tests whether systems can acquire new skills like humans do, rather than measuring performance on increasingly difficult PhD-level problems like MMLU.

• •Intelligence defined as ability to learn new skills, not just solve hard problems
• •ARC benchmark tests both humans and machines on the same tasks to ensure normal people can solve them
• •Contrasts with benchmarks like MMLU, MMLU+, and Humanity's Last Exam that focus on superhuman difficulty
• •Foundation's mission: pull forward open progress toward human-like generalization

Base LLMs performed terribly on ARC-AGI (GPT-4 at 4-5%) until reasoning models emerged. OpenAI's o1 jumped to 21% performance, revealing that reasoning paradigms were transformational. Major labs including OpenAI, xAI, Google, and Anthropic now use ARC-AGI as a standard benchmark in their model releases.

• •GPT-4 base model scored only 4-5% on ARC-AGI in 2024, five years after benchmark creation
• •OpenAI's o1 with reasoning capabilities jumped to 21%, identifying reasoning as transformational
• •All major labs (OpenAI, xAI, Gemini, Anthropic) now report ARC-AGI scores in releases
• •ARC-AGI successfully identified the importance of reasoning paradigm shift

Kamradt warns against relying on reinforcement learning environments as a measure of true progress toward AGI. While RL can achieve short-term benchmark gains, it's like 'whack-a-mole'—you can't create RL environments for every future task. True generalization means systems should learn without needing custom training environments, unlike humans.

• •RL environments provide short-term benchmark gains but don't demonstrate true generalization
• •Can't create RL environments for every novel problem that will emerge in the future
• •Humans don't need custom training environments to learn new skills
• •Investment should focus on systems that generalize without environment-specific training

ARC-AGI 1 (2019) had 800 static tasks created by Chollet. ARC-AGI 2 (2025) deepened the benchmark. ARC-AGI 3 (upcoming) introduces 150 interactive game-like environments without instructions, requiring systems to learn goals through trial and error—mirroring how humans interact with reality.

• •ARC-AGI 1: 800 static tasks created by François Chollet in 2019
• •ARC-AGI 2: Deeper version released March 2025
• •ARC-AGI 3: 150 interactive environments with no instructions provided
• •V3 requires systems to discover goals through action-feedback loops, not following explicit rules
• •Every game tested with 10 regular humans (accountants, Uber drivers) to ensure solvability

ARC Prize measures intelligence through data efficiency and energy consumption, not wall-clock time (which is arbitrary based on compute). ARC-AGI 3 counts the number of actions AI takes versus humans to complete tasks, preventing brute-force solutions that require millions of frames like old Atari RL approaches.

• •Wall-clock time is arbitrary—more compute reduces time but doesn't indicate intelligence
• •Two key metrics: training data required and energy consumed (both benchmarked against humans)
• •ARC-AGI 3 normalizes AI performance to average human action count
• •Prevents brute-force solutions that spam millions of actions (unlike 2016 Atari approaches)
• •Human brain provides benchmarks for both data efficiency and energy consumption

Solving ARC-AGI is necessary for AGI but not sufficient—it proves generalization capability, not full AGI. A system beating ARC-AGI 3 would provide the most authoritative evidence of generalization to date. ARC Prize positions itself to analyze such systems and ultimately declare when true AGI arrives.

• •Solving ARC-AGI 1 and 2 is necessary for AGI but not sufficient proof
• •ARC-AGI 3 solution would be most authoritative evidence of generalization capability
• •Foundation would analyze any winning system to identify remaining failure points
• •ARC Prize aims to be authoritative source for declaring when AGI truly arrives