Brendan Foody on Teaching AI and the Future of Knowledge Work

Mercor hires domain experts (poets, economists, lawyers) to create evaluation rubrics for AI models rather than just providing raw text. The key insight is that experts teach models once, which then scales to billions of users. Discussion covers how to identify good poets, the importance of some disagreement among graders, and why rubrics are more valuable than raw training data.

• •Experts create rubrics to grade AI outputs (similar to professors grading essays), not just raw content
• •You want some disagreement among expert graders to capture edge cases, but not too much
• •Mercor hired Larry Summers (finance), Cass Sunstein (law), Eric Topol (medicine) for their broad industry perspective, not just academic credentials
• •The company can pay premium rates because expert knowledge scales across billions of users once captured

Mercor's research with top domain experts reveals frontier models improving 30% annually on economically valuable tasks, with GPT-5 scoring 64% overall. The methodology involves surveying hundreds of experts about how they spend time, then creating corresponding prompts and rubrics weighted by economic value (salary/customer willingness to pay).

• •GPT-4-0 to GPT-5 showed ~30% improvement on economic tasks (GPT-5 at 64% overall)
• •Methodology: survey experts on time allocation, create prompts/rubrics for each activity bucket, weight by economic value
• •Models struggle with task-switching and long-horizon work (50-100 hours), though this may change in 6-12 months
• •In domains like consulting, 64% accuracy is already impactful; in medicine, 30% failure rate is unacceptable

Deep philosophical discussion on whether to enshrine current aesthetic standards or historical ones. Foody argues for eventually modeling taste from every era, allowing personalization. Explores the tension between what average users prefer versus what top experts consider quality, and whether we should model historical poets like Milton rather than contemporary ones.

• •Kant's critique: taste cannot be captured in a rubric, yet rubrics are what Mercor needs most
• •Alternative to rubrics: RLHF (having experts choose between model outputs repeatedly)
• •Proposal to enshrine peak eras (1960s/70s movies, 1980s metal, historical poetry) rather than current mediocre standards
• •Long-term vision: models learn taste from every decade, then personalize based on user preferences

Foody predicts that within 5 years, most high-end knowledge workers will transition from doing repetitive tasks to building RL environments and training agents. This represents a shift similar to software development - fixed cost investment in teaching agents, then unlimited reuse. Society will become a massive reinforcement learning machine.

• •Investment bankers will build RL environments for analysis instead of repeatedly analyzing data rooms
• •Customer support reps will find agent mistakes, turn them into RL environments, solving problems at scale
• •Knowledge work shifting to fixed-cost model: teach agent once, use many times (like software)
• •Within 5 years, majority of knowledge workers will be training models rather than doing repetitive analysis

Concrete discussion of what data would be most valuable for training models. Foody argues that evaluation data (rubrics, test questions with answers, unit tests) is more valuable than raw output data. Proposes that economics journals should send referee reports and submissions, and top seminars should be recorded and anonymized.

• •Two types of data: raw output (curriculum) vs. evaluation data (rubrics, tests, scoring mechanisms)
• •Evaluation data is most valuable - allows models to attempt problems many times and learn from scoring
• •Economics journals should send referee reports and submissions; top seminars should be recorded/transcribed
• •Best heuristic: if a model could max out this test without seeing it, how much economic impact would that create?

Foody explains Mercor's hiring methodology, which uses their own AI technology. The key mistake most companies make is not measuring actual job skills, instead relying on vibe-based conversations. For technical roles, give candidates projects and grade them. For non-technical roles, drill into similar past experiences and talk to references.

• •Biggest hiring mistake: measuring vibe/similarity instead of actual job skills and capabilities
• •For technical roles: give them a project with their preferred tools (including AI assistants), record screen, evaluate output
• •Body language and articulateness are often false signals - focus on work quality
• •Easier to measure y-intercept (current ability) than slope (growth potential) - past similar roles are best proxy
• •Mercor uses AI to analyze interview transcripts and optimize hiring funnels

The fundamental inefficiency in labor markets is disaggregation - candidates apply to dozens of jobs, companies consider a tiny fraction of candidates. LinkedIn has distribution but lacks effective matching. The solution requires AI-powered matching combined with the shift toward fractional, remote work and model training.

• •Core inefficiency: candidates apply to ~24 jobs, companies consider <1% of potential candidates
• •LinkedIn has distribution but can't predict job performance from profiles - it's a matching problem, not distribution
• •AI will enable better matching, but also shift job nature toward fractional RL environment building
• •Prediction: mentors and nepotism may increase as everyone has perfect cover letters/LinkedIn profiles
• •Hope: AI-run companies will be data-driven about references rather than relying on nepotism

Critical framework for understanding job displacement. Software engineering is highly price elastic - 10x efficiency may create 10x more engineers building 100x more software. Other domains like accounting or customer support may have inelastic demand. Early-stage capital allocation and product distribution also show high elasticity.

• •Software engineering: highly elastic - 10x efficiency → 10x engineers → 100x software built
• •Accounting/customer support: potentially inelastic - only need so much, may see job displacement
• •Product building and distribution: highly elastic, will see massive expansion
• •Early-stage capital allocation: currently inefficient (Mercor struggled to get $10k seed), huge room for improvement
• •Teaching: mixed - some shrinkage in lecture delivery, but personal relationships and emotional development remain important

Discussion of the strong statistical correlation between dyslexia and entrepreneurship. Foody's theory: dyslexic people learn to delegate early out of necessity, a critical founder skill that competent people often don't develop until later. Also forces focus on comparative advantages and big-picture thinking rather than getting lost in details.

• •Dyslexic entrepreneurs learn delegation early - asking for help reading, etc. - a skill others develop late
• •Forces focus on what you're uniquely good at rather than trying to do everything
• •Enables unconventional thinking and different approaches to problems
• •Early investor Scott Sandell (also dyslexic) has backed many dyslexic entrepreneurs
• •Key lesson: success comes from leveraging strengths, not fixing weaknesses

Mercor's next focus is building evaluations for long-horizon tasks (days/weeks) with multiple tool integrations - bridging the gap between model intelligence and actual enterprise usefulness. The past two years focused on intelligence; the next phase is about practical utility in real workflows.

• •Building evals for multi-day/week tasks with complex tool use - the missing piece for enterprise adoption
• •Past 2 years: focus on intelligence. Next phase: focus on usefulness and practical enterprise applications
• •Mercor sits at intersection of labor markets (DNA from Uber CPO/CTO Sandeep Jain) and AI research
• •Most interested in learning: which specific rubrics and data types drive the most model improvement
• •Goal: measure what enterprises actually want to use, then get those capabilities into models