[State of Evals] LMArena's $100M Vision — Anastasios Angelopoulos, LMArena

Anjney Midha at a16z incubated Arena by forming an entity and providing grants before the team committed to starting a company, allowing them to walk away at any time. The decision to spin out came from realizing that scaling Arena's mission of measuring frontier AI on real-world usage required company resources beyond what academic or nonprofit structures could provide.

• •a16z gave grants and formed entity before Arena committed to becoming a company - highly unusual VC move
• •Team could have walked away with no obligations, but recognized company structure was only path to scale
• •Considered staying academic or going nonprofit but lacked resources to achieve mission under those models
• •Sequoia also provided early grants alongside a16z support

Arena raised $100M primarily to fund inference costs and scaling. The platform processes 250M+ total conversations with tens of millions monthly, making it one of the largest consumer LLM platforms after ChatGPT. They pay standard enterprise rates for inference, not special discounts, and fund all model calls for free user access.

• •Platform processes tens of millions of conversations monthly across 5-6M users
• •Arena funds all inference at standard enterprise discount rates, not special deals
• •25% of users are software developers; half of all users are now logged in
• •Funding provides runway for multiple strategic bets, not necessarily all spent

A major use of funds was migrating from Gradio to React/Next.js. While Gradio scaled Arena to 1M MAU, custom features like loading icons with notifications became difficult to implement. The team also needed a more common tech stack for hiring developers.

• •Gradio successfully scaled Arena to 1M monthly active users but hit development limits
• •Custom UI components (e.g., video loading notifications) were hard to build in Gradio
• •Moved to React/Next.js for better developer hiring pool and tooling ecosystem
• •Grateful to Hugging Face/Gradio but needed more flexibility for product evolution

The "Leaderboard Illusion" paper from Cohere researchers claimed Arena did undisclosed private testing that created leaderboard inequities. Arena's response pointed out factual errors: they claimed 9% open-source sampling when it was actually 60%, and the prerelease testing (like "Nano Banana") was always public and community-loved, not hidden.

• •Paper claimed undisclosed private testing created bias; Arena showed this was factually incorrect
• •Open-source model sampling was 60%, not 9% as paper claimed - major methodological error
• •Prerelease testing with code names like "Nano Banana" was always public and transparent
• •Community loves early access to unreleased models - it's a feature, not a bug

Google's "Nano Banana" image model launch on Arena became a global sensation that changed Google's market share and roadmap. The discussion reveals how multimodal models, especially image generation, are becoming economically critical for marketing and content creation use cases, despite initial skepticism about their strategic importance.

• •Nano Banana was named after Google PM Naina (nickname "Nana") - not randomly generated
• •Launch moment alone shifted billions in Google stock value and triggered OpenAI "code red"
• •Marketing and design are fastest-growing AI adoption segments in enterprise
• •Image models enable instant creation of paper-quality diagrams that previously took PhD students weeks

Arena's public leaderboard is treated as a "charity" and loss leader - models cannot pay to be listed, cannot pay for better scores, and cannot pay to be removed. Every released model gets a statistically sound score based on millions of real user votes, maintaining the platform's role as the industry's North Star.

• •Public leaderboard is a loss leader, not a revenue source - integrity comes first
• •No pay-to-play model: can't pay to get on, can't pay for better scores, can't pay to remove
• •Scores reflect millions of votes from real users globally - purely data-driven
• •Unreleased preview models don't appear on public board, only released models get scored

Arena is expanding into vertical-specific evaluations with Expert Arena showing model performance across medicine, legal, finance, and creative fields. Code Arena evaluates full agent harnesses, not just models. Video arena launching later in 2025, focusing on categories where they have sufficient expert user density.

• •Single-digit percentages of millions of users = substantial expert populations per vertical
• •Expert categories include medicine, legal, business, finance, accounting, creative marketing
• •Code Arena designed to evaluate full agent harnesses like Devin, not just base models
• •Video arena in pipeline for later 2025/early 2026 launch

Arena views every user as earned daily in the competitive consumer market. Persistent chat history after sign-in was a major retention driver. The team focuses on providing continuous value rather than chasing ChatGPT's scale, recognizing that consumer success requires both execution and luck (like viral Nano Banana moments).

• •Persistent chat history for signed-in users was biggest retention mechanism implemented
• •Consumer is hardest market - users are fickle and must be earned every single day
• •Not trying to beat ChatGPT's scale - that's a generational product with luck involved
• •Viral moments like Nano Banana bring users, but retention requires continuous value delivery