Ep 77: Anthropic’s Dianne Na Penn on Opus 4.5, Rethinking Model Scaffolding & Safety as a Competitive Advantage

Dianne explains how Anthropic approaches model development with an ambitious long-range capability roadmap, treating each Claude generation as a vehicle to express advancements in areas like instruction following, coding, and memory. The process balances planned improvements with user-discovered capabilities, exemplified by their investment in Excel/PowerPoint proficiency resonating unexpectedly well with financial services customers.

• •Anthropic maintains a long-range roadmap of capabilities they want to improve, with each Claude generation expressing these advancements
• •Doubling down on capabilities involves both getting more data and doing more RL in specific areas based on user feedback
• •Small investment in Excel/PowerPoint capabilities unexpectedly resonated strongly with financial services customers
• •Development starts with product vision docs imagining future use cases, then translates to evals to measure model improvements
• •Process similar to traditional product management: envision the 'so what' before building evaluations

Discussion of computer use capabilities moving from experimental feature to end-to-end agent, progressing from constrained environments like QA testing to more open-ended browser-based tasks. Dianne shares her personal use of Claude for Chrome and how Opus 4.5's improved vision quality enhanced the interaction quality, positioning computer use as potentially transformative as agentic coding.

• •Computer use has evolved from experimental complement to agentic coding toward standalone end-to-end agent capability
• •Currently works well in constrained environments (browser) but moving toward more open-ended laptop-wide use
• •Opus 4.5's improved vision quality significantly enhanced computer use interaction quality
• •Use cases expanding from QA testing to practical tasks like rescheduling Google Calendar
• •Computer use could become as transformative as agentic coding for how people interact with AI

Dianne reveals that Opus 4.5 was designed from the start to have efficiency gains passed on to users, making it both more capable and cheaper than previous Opus models. She emphasizes the under-hyped 'effort parameter' that can achieve Sonnet 4.5-level intelligence at a fraction of the price, arguing the industry needs to move beyond per-token pricing to focus on end-to-end cost to achieve tasks.

• •Opus 4.5 was intentionally designed for efficiency gains to be passed on to users from the beginning
• •The 'effort parameter' is under-hyped but allows Sonnet 4.5-level intelligence at much lower cost
• •Smaller models often take more tokens or fail to complete tasks, making them more expensive end-to-end
• •Industry needs better education around total cost-per-task vs. per-token pricing
• •Opus 4.5 used internally for strategic thinking - came up with pricing/positioning alternatives spontaneously

Early access customers like Shortcut reported 20% accuracy improvements without changing their harnesses. Dianne discusses the challenge of measuring product-market fit for synchronous agents beyond coding, noting that while the intelligence exists, the industry hasn't figured out the right product features and harnesses for many use cases yet.

• •Shortcut (cell agent) saw 20% accuracy improvement with Opus 4.5 without changing any infrastructure
• •3D gaming use cases showed visible intelligence improvements in early testing
• •Agentic coding has clear PMF; synchronous agents for other domains still need right product features
• •Many use cases remain chat-focused when they could be more proactive/autonomous
• •Opus 4.5 enables new categories of maintenance and monitoring tasks, not just one-time deliverables

Anthropic deliberately chose to focus on expanding intelligence rather than pursuing image/video generation, despite customer requests. This strategic decision reflects their intentional focus on business use cases and enterprise requirements over consumer features, with the ability to iterate quickly on customer feedback through frequent model releases.

• •Anthropic intentionally declined to pursue image generation and video despite customer requests
• •Consistent 2.5-year focus on expanding intelligence over other modalities
• •Product focus deliberately oriented toward business use cases and enterprise requirements
• •Fast shipping velocity allows customer feedback to be incorporated in next model release
• •Building internal systems to automate feedback loops and environment creation for improvements

Dianne discusses how traditional evals like SWE-bench (where Opus 4.5 hit 80.9%) are becoming saturated, necessitating a shift toward more open-ended evaluations. She advocates for evals that measure not just task completion but quality of judgment, time to completion, and ability to handle long-running tasks, using examples like 'vending bench' and Pokemon gameplay.

• •Traditional benchmarks like SWE-bench and TauBench are saturated (Opus 4.5 at 80.9% on SWE-bench)
• •Industry needs to evolve toward more open-ended evals that aren't just yes/no tasks
• •Quality measurements should include time-to-completion and judgment quality, not just task completion
• •Long-running task evals like 'vending bench' (running a vending machine business) show promise
• •End-user quality measurements and customer feedback remain most important evaluation metrics

Dianne describes how scaffolding has evolved from 2022-2024 from being 'training wheels' with extensive rules to keep models on distribution, to intelligence amplifiers that maximize autonomy. Modern best practices involve lightweight scaffolds with generic tools, multi-agent orchestration, and iteratively removing non-amplifying components as models improve.

• •2022-2024 scaffolds were 'training wheels' with extensive do/don't rules to keep models on distribution
• •Modern scaffolds are intelligence amplifiers rather than constraints, maximizing model autonomy
• •Claude Code uses relatively lightweight scaffolds with generic Bash tools rather than specific custom tools
• •Multi-agent orchestration has become viable in 2024 for improving context quality
• •Best practice is iteratively removing scaffold components that no longer amplify intelligence
• •User behavior naturally pushes products to be more complex as they discover new capabilities

Dianne characterizes Anthropic's culture as exceptionally authentic with leaders who 'walk the walk,' featuring the highest talent density she's experienced. The company runs hackathons every 3-4 months to enable discovery of new capabilities, and each Claude generation transforms how Anthropic employees work internally, providing direct feedback on model improvements.

• •Anthropic is the most authentic company Dianne has worked at - leaders genuinely embody stated values
• •Highest talent density environment with people who take radical ownership and are kind but direct
• •Internal hackathons every 3-4 months allow employees to test ambitious prototypes with new models
• •Each Claude generation transforms how Anthropic employees actually work day-to-day
• •Company grew from 150 people to current size in 2.5 years while maintaining culture
• •Early days involved extreme hands-on work: setting up first A/B tests, emailing prospective customers directly

Two critical strategic decisions shaped Anthropic's trajectory: In 2023, despite overwhelming user demand for embedding models for RAG, they invested in agentic coding instead (user-centric vs user-led). Second, shipping computer use as a beta despite it not being perfect, accepting they couldn't capture every edge case upfront and needed real-world feedback to improve safety.

• •In 2023, most common user request was embedding models for RAG, but Anthropic invested in agentic coding instead
• •Distinction between being 'user-led' (giving what's asked) vs 'user-centric' (solving bigger opportunity)
• •Nobody was asking for agentic coding in 2023 - required vision of what AI could become
• •Shipped computer use API as beta knowing it wasn't perfect, accepting need for real-world safety learning
• •Golden Gate Claude shipped from model to UI in less than a day with small grassroots team
• •Internal viral success of features often predicts external success

Dianne argues safety is under-discussed as a product benefit rather than just a constraint - well-aligned models are independent thinkers that push back rather than being sycophantic, leading to better ideas and breakthroughs. She believes we're closer to transformative long-running AI than most think, with the bottleneck being product innovation rather than model capabilities, and her ASI timelines have moved up based on Opus 4.5's building blocks.

• •Safety investments reduce sycophancy, creating independent thinkers that challenge ideas rather than agree
• •Aligned models provide higher quality intelligence by offering alternative perspectives (e.g., pricing strategy)
• •Industry doesn't discuss enough how safety amplifies intelligence quality, not just constrains bad behavior
• •Building blocks for transformative long-running AI are closer than expected - main gap is product innovation
• •Long-running agents need to handle open-ended responsibilities like maintenance and refactoring, not just discrete tasks
• •Dianne's ASI timelines moved up this year - sees high probability in both short-term and long-term (barbell distribution)
• •Developers should maintain libraries of ambitious prototypes that didn't work yet, testing with each new model