Are AI Benchmarks Telling The Full Story? [SPONSORED] (Andrew Gordon and Nora Petrova - Prolific)

Current AI evaluation relies heavily on technical benchmarks like MMLU and 'humanity's last exam,' but these don't measure what matters for real users. The field is nascent and fractured, with labs reporting results inconsistently. Models optimized for benchmark performance often provide poor user experiences, missing the critical human element.

• •Technical benchmarks don't correlate with user experience - a model excelling at exams may be a 'nightmare to use day to day'
• •No standardization in how labs report benchmarks - some emphasize specific tests, others release models without any benchmarking data
• •Humans aren't involved in the evaluation loop of technical benchmarks, creating a fundamental disconnect
• •The field lacks even playing fields for comparing models across different evaluation approaches

Prolific developed a new evaluation methodology starting with a proof-of-concept 'User Experience Leaderboard' (500 US participants) and evolving into 'Humane.' The approach uses comparative battles similar to Chatbot Arena but with demographically stratified samples and multi-dimensional ratings beyond simple preference.

• •Initial leaderboard used 500 representative US participants rating models on Likert scales for dimensions like helpfulness and personality
• •Humane leaderboard uses comparative battles between models for clearer differentiation
• •Evaluation breaks down preference into constituent parts: helpfulness, communication, adaptiveness, personality, trust
• •Provides actionable insights to labs about specific weaknesses (e.g., 'your model is struggling with trust')

People increasingly use AI models for sensitive topics like mental health with no oversight or safety leaderboards. Recent incidents with Grok and Meta highlight thin safety training. Anthropic's work on constitutional AI and mechanistic interpretability offers promising directions for understanding and ensuring model safety.

• •No leaderboard or standardized metrics exist for AI safety despite its critical importance
• •Users discuss sensitive topics (mental health, life problems) with models that have minimal oversight
• •Recent safety failures (Grok, Meta) reveal inadequate safety training layers
• •Anthropic's constitutional AI and mechanistic interpretability work traces model 'thoughts' to identify potential problems
• •Safety should be weighted equally with speed and intelligence in model evaluation

The 'Leaderboard Illusion' paper exposed bias in Chatbot Arena where companies get unequal access to private testing. Meta released 27 Llama models before launch, gaining unfair advantage through more comparisons and prompt access. Additional issues include anonymous sampling with no demographic data and lack of quality assurance on prompts.

• •Companies with more pre-release model variants (e.g., Meta's 27 Llama models) gain unfair advantages through more battles and prompt data
• •Strong correlation exists between number of battles and leaderboard position, undermining integrity
• •No demographic data collected on Arena participants - just anonymous users with unknown characteristics
• •No quality assurance on prompts - users can submit low-effort queries or wander between unrelated topics
• •Simple preference data ('I like this more') provides no actionable insights on why preferences exist

Prolific addresses Arena's flaws through three improvements: demographically stratified sampling based on census data, multi-dimensional specificity in ratings, and built-in quality assurance. Participants must engage in multi-step conversations with penalties for low-effort or topic-wandering prompts.

• •Sample stratification by age, ethnicity, political alignment using census data for representative results
• •Multi-dimensional ratings (helpfulness, communication, adaptiveness, personality) provide actionable feedback to labs
• •Quality assurance system penalizes low-effort prompts and topic wandering - three strikes and you're out
• •Multi-step conversations required rather than single-turn interactions
• •Results confidently represent general public preferences rather than biased subsets

Humane uses Microsoft's TrueSkill framework (originally for Xbox Live) to estimate model capabilities through Bayesian distributions. The system prioritizes battles based on information gain, conducts only necessary comparisons, and can run separate tournaments for demographic groups while consolidating findings.

• •TrueSkill accounts for randomness, changing performance over time, and distinguishes fluky wins from consistent performance
• •Battle pairs selected based on maximum information gain and uncertainty reduction
• •Bayesian distributions with narrowing variance provide increasing confidence over time
• •Can run 20+ separate demographic tournaments and consolidate for overall leaderboard with minimal uncertainty
• •Computationally efficient - only conducts battles needed to reach desired confidence intervals
• •Strong model differentiation based on statistical uncertainty rather than popularity

Prolific stratifies participants by demographics using US and UK census data to ensure results represent real-world populations. The approach allows confident claims that preferred models align with general public preferences rather than skewed subsets, with plans to expand globally.

• •Participants stratified by age, ethnicity, political alignment matching census proportions
• •Currently covers US and UK with plans for global expansion
• •Results confidently represent general public preferences vs. potentially biased Arena users
• •Census data provides ground truth for demographic proportions in each country
• •Methodology ensures findings generalize to real-world user populations

Initial 500-participant study revealed models consistently underperform on personality and background/culture metrics compared to helpfulness and communication. This suggests training on 'the entire internet' doesn't produce desired personalities. Recent increase in model sycophancy (people-pleasing behavior) correlates with user dissatisfaction.

• •Models scored significantly lower on personality and cultural understanding than on objective measures like helpfulness
• •Training on internet-scale data may not produce personalities that align with user preferences
• •Models struggle to understand or align with users' backgrounds and cultures
• •Observed increase in sycophantic behavior across models - users generally dislike people-pleasing responses
• •Future analysis will correlate sycophancy patterns with personality metric downvotes
• •LLM-as-judge analysis can classify conversation patterns and identify relationships between model behavior and user feedback