[State of Post-Training] From GPT-4.1 to 5.1: RLVR, Agent & Token Efficiency — Josh McGrath, OpenAI

Josh explains his transition from pre-training data curation to post-training, motivated by the opportunity to change model behavior by 40% rather than achieving 3% compute efficiency wins. He details the infrastructure complexity of RL runs compared to pre-training, with significantly more moving parts including diverse task grading setups and external/internal partnerships.

• •Post-training offers higher impact on model behavior (40% changes) vs pre-training efficiency gains (3%)
• •RL runs have order-of-magnitude more moving parts than pre-training: multiple tasks, different grading setups, more infrastructure
• •Understanding unfamiliar code (internal or external) is critical for debugging runs at midnight
• •Codex has transformed workflow: 30-40 minutes of design doc writing enables 15 minutes of AI work that replaces hours of manual coding

Discussion of the shopping model released during Black Friday/Cyber Monday, featuring new interruptibility where users can interrupt the model's chain of thought and provide corrections mid-stream. Josh explains it's a separate model for now to experiment with deep-research-style capabilities for shopping, though models eventually converge in capabilities.

• •New interruptibility feature allows users to press escape and correct the model mid-task (similar to Codex)
• •Shopping model shows chain of thought with products being evaluated, enabling real-time user feedback
• •Separate model enables faster experimentation with deep-research-style capabilities for specific domains
• •Deep Research and GPT-5 thinking on high reasoning are roughly on par according to published evals

Josh argues that the key insight in post-training is that RLHF and RLVR are both policy gradient methods—the real difference is input data quality and signal trust, not optimization techniques. He highlights GRPO from DeepSeek Math as underappreciated because it introduced verifiable reward signals (math answers you can verify) versus human preferences you can't.

• •RLHF and RLVR are both policy gradient methods; the innovation is in input data quality, not the math
• •Current papers read like optimization papers but miss the real innovation: the spectrum of signal quality and trust
• •GRPO from DeepSeek Math is underappreciated—the key innovation was verifiable reward signals (math answers) vs unverifiable human preferences
• •The industry should focus more on 'how clean is this signal and how much do I trust it' rather than gradient variance optimization

Josh emphasizes thinking in tokens rather than wall-clock time, as token efficiency is the key optimization target. GPT-5 to 5.1 showed significant eval improvements while dramatically reducing token usage—a critical metric for user experience and enabling more tool calls within reasonable serving constraints.

• •Token efficiency is more important than wall-clock time—it's what you can actually optimize
• •GPT-5 to 5.1: evals improved while token usage went way down (dramatic efficiency gains on 2D plot)
• •Token efficiency directly impacts user experience and determines how many tool calls an agent can make
• •Example: Codex does in 10 minutes what would take 4 hours, but the real win is token efficiency, not time

Discussion of long context capabilities, with Josh defending GraphWalks as the key evaluation because it requires complicated transformations across the entire context window, not just sampling from one point. He notes these evals are still climbing and context rot is a temporary issue being actively solved.

• •GraphWalks eval requires multiple transformations across entire context window—much harder than single-point sampling
• •Heat map plots of context utilization miss the complexity of cross-context transformations
• •GraphWalks scores are climbing over time, indicating progress on 'perfect context' utilization
• •Agents with GREP are 'unreasonably effective'—similar to how BM25 was hard to beat in information retrieval

Josh identifies the critical hiring gap: researchers who excel at both distributed systems engineering and ML research. The bottleneck in frontier research shifts constantly between systems and ML, but the education system doesn't produce enough people skilled in both areas.

• •Critical skill gap: people who do both systems work and ML work at high levels
• •The frontier bottleneck changes constantly—sometimes systems, sometimes ML—within single projects
• •Current education system isn't optimized for producing systems-ML hybrids
• •Post-training at OpenAI has great culture of people moving seamlessly between systems and learning work

Josh addresses the 'pre-training is dead' meme and the controversial trend of equal compute investment in pre-training and post-training (citing Grok-4 chart). He draws parallels to the steam-to-electricity transition in factories, arguing we're in a 'fog of war' where declaring anything dead is premature—timelines are short but human adoption is slow.

• •Neither pre-training nor post-training is dead—both are actively scaling
• •Grok-4 chart shows equal compute investment in pre-training and post-training (controversial shift)
• •Historical parallel: factories took decades to optimize for electricity after assuming steam engine patterns
• •Living through technological revolution creates 'fog of war'—ideas will be dormant then spike, expect 'it's so over/we're so back' cycles