Ep 77: Anthropic’s Dianne Na Penn on Opus 4.5, Rethinking Model Scaffolding & Safety as a Competitive Advantage

OPUS 4.5 is a really impressive model. It's had amazing benchmarks. A bunch of companies are getting great outcomes from it. And on supervised learning, I got to sit down with Diane, the head of product for research at Anthropic, to talk about all things OPUS 4.5. We hit on what's newly enabled by the models as well as where Diane thinks we are with computer use.

We talked about how Anthropic does research and gets to models like this, and how they use these models internally. We talked about what's next for models and the evolution of scaffolding over the past years, and we hit on how Anthropic safety focus actually helps them on the product side. This is just a lot of fun to get to go deep into the research process, model capabilities, and a bunch of other things. I think folks really enjoy it. Without further ado, here's our conversation.

And thanks so much for for coming on the podcast.

Thank you for having me. Happy early Thanksgiving.

Yeah. Thanksgiving eve pod is like a real a real treat to get to do. I feel like you guys dropped this this big model as a as a gift to us all right before the holidays. I'm sure a lot I'm sure a lot of founders will be saying they're grateful for it tomorrow at the at the Thanksgiving table.

Love it. That's that's the vibe we're going for.

You guys already had some really impressive models. What does it look like when you begin the work for something like OPUS four point five? You know, how do you think about the you know, what actually is required to improve these models? And like, what does that whole process end up looking like?

I think like in some ways, we have a pretty, I would say, ambitious and, like, long range roadmap around model capabilities that we care about and that we care about making improvements on. So this includes things like better instruction following for users, coding advancements, making the models better at memory, etc. And really, in some ways, every generation of Claude are like the vehicles by which these capabilities are expressed. And so, like, even when we are designing new versions of Cloud, it's really around what are the advancements overall that we want to, like, make sure can be delivered. And, and how do we actually package it also, position it, price it in a way that, like, resonates for the types of use cases that users have today, but also they might not even be aware that, like, AI can help them take on.

And so you gotta pick a set of, like, problems or things that you wanna improve the models on. And then, you know, to what extent is it kinda clear the the research directions to go down to improve that, or are you having, like, a dozen different things and you're kinda figuring out early on which which ones help actually move the needle there?

I think it's both. I think we have, like, had a very, generally, strong sense of what we think, like, this technology can do, which is that, like, we can think, we think that is generally can be extremely transformative across a wide variety of like use cases, whether it's like engineering or others. I think like other things like surprise us with like how users and also like builders discover them. So things like making Cloud really good at Excel and PowerPoint. That was like a relatively small investment earlier on in the year, and like what we found is that it really resonated with financial services customers.

And so we're doubling down and making Cloud really good at that type of, like, general, like, Excel work or, like, PowerPoint work that seems really beneficial. So it's both. What's the right

way to conceptualize, like, doubling down? Is that, like, you know, getting more data around a topic or doing more RL around that area?

Yeah. I think it does for, like, practical applications does start with, like, users and both users who might be coming to us with a use case, but also when we think of like something like computer use. Why should people care? Like, we kind of have to imagine almost a world that we want to go towards. And so there's like almost an imagination phase.

As a PM, the closest thing I could think of is like product vision docs or PRDs, where you're trying to figure out like, what is the so what? Why should somebody come and use this solution? And then also then translating that into practically what are evaluations, what are evals, we call them, right? Like, what are the evals we want to build to know the model's really good or not good at it? Maybe there's already half the way there and like it needs to make certain improvements on data or RRL for those like last 50%.

Maybe it's 20%. We actually need to figure out, like, much more significant changes we wanna make. And so starting with, like, sort of that envision in mind is actually very similar to traditional product management, which I think might be surprising as to hear.

It's really interesting. Think especially as like the kind of classic evals feel like they're more and more divorced from like what actually the value is in the real world or the ways that people are using these things. You have this combination of both customers coming to you with here and now problems that are probably really helpful evals. You obviously have the best seat to what these models might be able to do in the future and really imagining what the what some of those use cases might be. Yeah.

Were there were there any, you know, things that you were imagining that OPUS 4.5 is the first model to be able to do? That maybe maybe we're in the vision doc like a year, year and a half ago, and you're like, maybe one day we could do x or, you know, what were some of the model capabilities that really surprised you?

Yeah. I think some of them have been, like, continuations of, you know, being able to do more complex agentic coding tasks, but also making work that is like more iterative and actually, longer running. So like that's been one thing, like we're starting to see, I think, more of an inflection point on both like complexity and also being able to iterate and continuously improve some of the deliverables. I think computer use has been another one. Like we saw and have been investing in computer use for a long time.

Last year, in November, I think, November or October, we launched the computer use API and since then we continue to invest in it. And like, I use things like Cloud for Chrome, which is like our browser use extension feature pretty often. And like I saw like an improvement just in terms of the quality of that interaction because now Cloud's vision is much better. So I think sometimes it's a combination of like multiple things kind of working together.

I guess there's, you know, a lot of, builders listening to the podcast that, you know, are are curious about computer use. And do you have, like, a rough mental framework today of, like, where computer use works and where it doesn't and kind of where we are on the journey?

Yeah. I think, like, computer use have gone from this, like, very early stage experimental feature when we launched it, and we were initially thinking about it as, like, probably the first stage is, like, a complement or a feature on top of something core, like agentic coding, to now I think more and more between Opus 4.5 and beyond, to be more of an end to end agent just by itself. And so like, can it not just like within a more constrained environment look at the types of like, you know, QA testing that was a popular use case, but actually like, be able to help monitor and be an agent in like a web browser. It's still more constrained than like an open ended, like, let's say your entire like, laptop, but it's like very helpful to be able to have like an agent that like, helps me reschedule my calendar on Google, because that does get pretty complicated. And yeah, it's I think that's kind of just been kind of the arc, like, moving from more constrained environments to towards, things that are a bit more open ended.

Was there anything that you, played around with with Opus 4.5 when you first had it internally that, like, was, was was surprising?

It was actually really helpful from a product team perspective to debate like things like pricing and like how our positioning for the model. Yeah. Because it's such a great, like leap in many ways. So from a product manager perspective, it was actually not just a great writer, but it was great thinker. It came up with like, you know, different, ideas around things like pricing and positioning, where it wasn't just refining my ideas, but it came up with alternatives, more spontaneously than I had seen with other models.

Know, it's like traditionally, a lot of the Opus models were, like, much more expensive, and this is both a really good model and, like, quite cheap, honestly. What what, like, drives that? And I guess, how clear was it from the beginning of this work that, like, this Opus model would be able to be served this efficiently?

From the beginning, we were hoping that this would be a thing for Opus models where we're able to have more efficiency gains and we're able to pass it then on to our users and customers and builders. We intentionally also made it possible, in addition to like the core model, training for things like the effort parameter, which I actually think is like under hyped right now, where you could actually get like SONET 4.5 level of intelligence at a fraction of the price. And I think this is something that like we as an industry haven't gotten really good about, which is like just because a model has a certain token price and sticker price, that's not always a good measure of like the end to end cost to achieve a task. And what we want to do with Opus, and what we want to do with things like the effort parameter, is to make it more accessible, that you could actually achieve a higher quality and a lower cost. And, you know, we're starting with that with with Opus, I'm really excited about this area going forward as well.

Yeah. Do you feel like most most folks, like, get that intuitively? Or, like, what how do you think about educating the market around, yeah, this idea that it's not just the per token cost, but obviously the amount of tokens it takes to to to complete some of these tasks?

Yeah. I think it's something we, as model providers, could be doing more around. So one thing I hear from builders is the smaller models actually just take longer to do a task, or it might actually not even get the task done, but you've spent a bunch more tokens than if you just used an Opus model. You know, for a lot of this year, we were traditionally focused around, like, Sonnet and having a very core flagship offering. And as we move into, like, different tiers of models, it actually is becoming more important for us to educate developers and users on it.

And so, like, I think this is an area we're gonna be investing in from, like, a marketing perspective.

You all have a very good intuitive sense of what these models can do. But then when you put it out there, I'm sure they start getting used in, like, a million different ways that you that you couldn't possibly have conceived of internally. In these first days of release, like, what have been some of the things that have surprised you the most?

Yeah. I think it's like very early. We're also in the middle of Thanksgiving week. So I'm sure my answer will will change a little bit as like, people have like tested the systems. I would say maybe two things.

One, in our early access with customers, we invested in areas, like I mentioned, around making the models better at office, like deliverables. It was really surprising to see how big of an accuracy jump Opus 4.5 was for customers like, Shortcut, which is like,

a

cell agent, I think customers were saying something like 20% accuracy improvement, just like without changing harnesses, without making other changes. So that felt like resonated a ton, because then they can pass that intelligence improvement to their users. I think other things that I've seen just like generally, in the early days, I think people tend to test the models on like gaming use cases. And so like three d games have gotten better in particular. So that's always really cool.

Visualizing some of those have just been an easy way for people to kinda see intelligence bounce. And I'm really excited also for just like the feedback that we're getting around quality. So customers and users saying like, oh, it's actually helped me clear out my entire backlog of bugs.

Now with with OPUS 4.5, as you think about kind of the broader ecosystem of applications and and, you know, I'm curious your mental model of like, what does have product market fit or like really works today on top of these models?

I think the big ones are, you know, agent to coding is, like, very visibly, something that's, like, here to stay. I think we continue to actually have more, like, enterprises reach out about solutions like Cloud Code and coding capabilities. I think synchronous agents have pretty strong reason for PMF generally beyond coding. I think it's more that we've not figured out the, we as a general industry haven't figured out the right harness plus product features to build on top. Like what is agentic coding but for you know, web monitoring and, like, personal agents, a lot of use cases are still very, like, chat focused.

And so I feel like we're kind of at an inflection point that we need a bit more intelligence to kind of really boost the neck, like, large set of use cases.

And do you think that, like, we have that level of intelligence now for those use cases?

I think people will be surprised by how good OPUS 4.5 is, and I think peep there will be new things that are born from from OPUS 4.5. There'll be new features and products.

So more of these, like, proactive experiences or, like, agents going out and doing things on the on your behalf and then servicing it up to you versus you prompting, like, via chat.

Yeah. That was one thing we heard a lot from, like, internally where it felt like the model just got it without having to have explicit human instruction. And I think when you couple that, plus the fact that, one, the context is getting high, the quality of the context is getting higher, and also things like memory start working better, really you get a system that can do not just one deliverable, but like do things that are not possible before, like monitoring, maintenance, like those are actually things that are like really valuable. It's not really just about shipping an MVP, It's about also, like, how do you maintain, something that you build.

I'm curious. You brought up that example of of shortcut and, like, really improving Excel. I imagine there's, like, an endless set of people that come to you, and they're like, we would love you to improve your model on, like, x domain or y task we have. How do you prioritize that? And, like, do you think that the that the foundation models end up going different directions just based on what they prioritize here?

I think a little bit. Yes. Right? Like, there's you know, we do have customers that us about things like image generation or video, and we're very intentional, and I've been here about like two and a half years, we've been very intentional about focusing on expanding intelligence. And so I think you do see that a little bit with the labs already.

In terms of a question of feedback, I think it's kind of a two way flywheel, right? Sometimes there are pain points or paper cuts we hear from customers. What's really good about our shipping velocity today is if a customer comes to us or a potential customer comes to us, we can make changes pretty quickly. Because if it's in 4.5, maybe it's in Cloud five, right? Like you have like multiple chances to get your feedback heard.

And then how do we also automate more of that? Like how do we build the systems on our side to like more organically figure out the right environments to build and, like, build that, like, loop ourselves? And then how do we deliver, like, new advancements, more more generally? So, like, things like computer use and beyond. So I think it's it'll be bidirectional.

Yeah. And it's interesting. You talk about building some of these tools internally, like, you know, more easily spinning up environments and and being able to, lower, I guess, probably the amount of effort that is required to, you know, to improve the models in these specific categories. And I'm sure there's, like, internal tooling that really helps with that.

Yeah. It's definitely a continuous investment area. We just have a lot that we could be doing, and so we have to be really intentional about, like, where we spend, like, to the point, how do we get Claude to help us

Yeah. When we're done? I mean, sounds like multimodal, like, you know, obviously has been below the the the, you know, the focus area. Are there any other things that you think of of, yeah, we could you know, we've explicitly chosen not to, you know, spend too much time on x or y.

I think those are some of the big ones. I think, like, we've also, from a product perspective, been very intentional about focusing on business use cases generally. Right? And so, like, this, like, means that, like, a lot of our focus and investment in product is around things like data security, privacy, meeting, like, enterprise requirements, for them to be able to adopt and use Cloud over, like, let's say, like, consumer use cases.

It feels like there's been this, like, really interesting discourse around enterprise agents over the last few weeks, I think largely driven by the podcast ecosystem, where, you know, you had, I guess, Andre Karpathy going on Darknesses Pod and being like, you know, hey. I don't know if this this is really it. Like, feels like we're still a decade away from from real enterprise agents. Then I guess Ilya most recently came on and was like, this current paradigm only gets us so far, like, we're we're gonna kinda hit a wall. You know, you then obviously square that with all the improvements in something like Opus 4.5.

Where Where do you kinda what what's your opinion on all this discourse on, like, the the feeling, I guess, maybe within Anthropic around it?

I I think intelligence and improvements and model improvements, what I've learned is, like, improvements are not necessarily like always a smooth line the way that you see in an eval benchmark of like, it's always like this. And I think they're more jagged edge. And I think depending on what eval or personal tests that you have, maybe it feels like a small jump, or maybe to someone else it feels like a large jump. So I think there's like a level of like what is exact like framing you're looking at. I think like from a customer and user perspective, what we hear from a Rakuten or a Lovable, etcetera, is that capabilities have actually still continued to improve their team's productivity.

And I think, like, really from like a, is AI transforming technology, and we talk about, like, less AGI, but more like transformative AI internally and anthropic, like, are we making technology as transformative? I think we're very much on the path on that. Claude is like transforming, every generation of Claude transforms internally how Anthropic employees work. And so we feel that because we're adopting it in a different degree. And so maybe if you're not adopting it to the same degree, it might not feel like it's meaningfully, like, make making advancements.

You've obviously gotten very far, and I imagine some of the things that were on the board are off now. But what's on, like, the next board?

Nobody can predict the future, but I think, like, some of the areas that I'm really excited about is, like, this move towards, like, longer running intelligence. So, again, not just it a human gives and delegates a specific task to Claude, but actually like Claude taking responsibilities that are more open ended. So not just like, build me this portion of my website, but maintain it, refactor the code when you think it's correct, and not needing so much of like, handholding. I'm also really excited about capabilities like Office, improvements that I mentioned, things like Excel, PowerPoint, there's a lot more to go. And also computer use, I feel like computer use is actually on the way of adoption and quality where it can really be transformative from enterprise and also just like general user perspective, the way that like a edge out of coding could be.

Like computers, like being able to navigate a computer is how most of us interact with each other. We're on a podcast right now, right? So like being able to have Claude that, like, can interact in these environments where, like, people work makes them much more useful. So I'm really excited about, like, where computers will go.

I mean, you mentioned long running agents, and I feel you know, it feels like this is certainly where the world's headed and, you know, maybe even having a fleet of agents doing work on your behalf. It it feels like in in many ways, you know, obviously, the models will get better. And but in many ways, part of this is solving the product side of it. And, like, what does it actually look like to have a long running agent going on in the background? And what does it look like to have a set of a dozen of them that you're managing?

You're obviously a product guru. Like, how do you think about what the product surface area might look like here? And and have you seen anything that that you thought was particularly interesting?

Yeah, I think long running agent and long running intelligence generally is a, it's not a use case in itself, right? We need to actually have a user problem that it's a good fit for. So I think things that are really valuable are around being able to maintain, iteratively improve, right? Let's say you have, you're an investor and you want to like understand the latest like stock movements or like how you should adjust your portfolio. That's like a one time thing.

And I think like what we kind of lack today is like having these like really long horizon tasks in a way that makes it easy to eval quality improvements. So like the closest one that I think we're starting to like track, it might not be the right one because I think, you know, no eval is perfect, it's like vending bench, which is the Claudius like run a vending machine business, right? But that's a I love

that you guys do this.

Or, you know, Claude plays Pokemon, right? Like these just, you know, Clawd played Pokemon when we first

Yeah. I kinda missed the Pokemon eval. That I I like I don't know if I'm missing it in the reports. I I liked when that was, like, the one one of the main ones people were discussing.

Okay. I am I will take a note for, like, potentially bringing it back. But, like How do

Maybe the models are all too good and they beat the game really quickly. But

Well well, it's also not just about completing tasks, but, like, how long does it take? Right? And, like, can you actually complete the task in much less steps because you're not brute forcing the problem? And, like, you remember the fact that, like, Ash Ketchum was like in Pallet Town at this time and already talked to like some user, Like to nerd out for a second.

We can go deep on Pokemon. I'm all game for that.

Yeah, but like, guess the part of what I want to know here is just like, you know, intelligence isn't just about like, can I complete a task and check the box? More and more, it's also the quality of the judgment and the quality of like, how, what intuitions some models can have in a situation. And so can we actually dramatically reduce the time it takes or the amount of effort to get to a certain outcome? And sometimes this really shows up in things like around long running tasks, right? But really, I'm just like excited about this area.

I think what we need more as an industry is better evals around it.

And what are the evals that still kind of matter?

Oh, it's a very good question. I think we would need to evolve beyond things like sweep bench and like TauBench. I think we're especially with Opus 4.5. I think we got to, like, 80.9% and, like, these these evals are was saturated.

I also love the example of, like, finding a way to upgrade the ticket through, like, you know, changing a fare category, and that was a very fun way to to to to to get around base.

Yeah. It's like it's crazy because, like, when I shipped the tool use API last year, think I we were in maybe below 50% on accuracy. Again, there's so much advancement, it's really how do we measure and how do we productize it. I think the evals that matter continue to be for us, what are the areas where people are using Cloud and how well does it work? So I think like end user measurements of quality continue to matter or feedback that we get from customers.

I do think we will probably see more movement towards evals that are more open ended, like a vending bench sort. It's not gonna look exactly like that because I don't know how realistic running a vending machine is all the time, but that type of direction of it's open ended, there is some quantifiable way to measure quality, but it's not just yesno. Cause how much of like most things in the world are yes, no tasks, right? If we're moving beyond coding into other types of like impact, like, it's hard to say, like, there a yes, no, like, sweepench equivalent in, biology. And you kind

of alluded a few times now to, you know, the harnesses that people are putting around these models. And Yeah. Obviously, you work very closely with teams that are at the cutting edge of using OPUS 4.5. How would you like, know, do you feel like there's a typical kind of set of scaffolding people are building around models today?

I think similar to model intelligence, scaffolds have evolved. I would say in, like, 2022, 2023, even 2024, a lot of the scaffolds are more like training wheels to keep the model on distribution. And they tend to be of the form, do not do this, always do this, right, just like instructions and like 20 rules. And I think more and more this year, we've seen is scaffolds become much more around augmentations for intelligence rather than training wheels, or like the best scaffolds tend to be. And iteratively removing the parts of the scaffold that are no longer intelligence amplifying.

So, for example, on like Cloud Code, our scaffolds are relatively lightweight. The types of tools we give it are things like Bash tools that are not like specific, very unique, and the point of that is that we want to maximize autonomy of like the work as it's being done by the model. And so the, like, types of scaffolds that I think will continue to be valuable might be things that are intelligence amplifying, so things that are, like, giving it, like, generic sets of tools, multi agents, I think, have started to be, like, more viable this year, where it's, like, not just, like, having one one model, but, like, orchestrating a set of models to, like, amplify and improve on things like context, quality, etcetera.

I think a bunch of builders are asking themselves, you know, what set of this stuff becomes obviated by the next generation of models versus, you know, continues to amplify intelligence for that next set? Is it obvious from the inside, or is it also like, you know, we'll we'll see when we get to those models?

I think there is a bit of like, this is why having like a thinner layers of harnesses and like scaffolds is important. I think though like, how much does it change? Like, it's a little bit hard to predict, but we do see improvements in quality by some level of iteration on scaffolds by model. Part of this also users requests also change, right? What we see is when I use Cloud Code or use Claude, if I see that it's really good at editing a document, I might give it a large set of things and like, hey, come up with an alternative strategy, right?

And so like work continuously also as users pushing the products that people build. And so it's not just in service of new models to update your Scaffold, but actually in service of user behavior. And users will naturally push your product to be slightly more complex. And how do you as, like, a developer meet that demand is, like, I think the the the the the right way to think of it.

You've kinda been in anthropic since the early days, and so I I feel like it must just be been a fascinating journey these past two and a half years. I I wonder maybe to to to start, like, how do you kinda compare the culture of Anthropic to, like, other places that you've worked?

I would say a lot of how, like, you know, our leaders show up in and how people talk about the mission and the goal. It's like very much actually how we internally, a big part of how internally we make decisions. Like, it's actually a lot of like walking the walk. And I think Anthropic has been the most like authentic, and as much from like the first day I joined to like now, of a company that I've been in. I think it goes without saying, but like the talent caliber and talent density has been incredible.

It's been the most like talent dense, like environment that I've gotten to work with. And I think people are like just deeply take like radical ownership, deeply thoughtful, kind, but direct, and also in the service of making products, models, capabilities, research better.

I'm sure our listeners will be curious, like, what does your day to day look like?

I think, like, three months, my job changes because it's almost like we're a different company. I think we were like 150 people when I joined. Like two years ago, around this time, I was setting up like our first AB tests, and emailing prospective customers to try new models. It was extremely hands on. As a PM, you do whatever it takes to get this thing done.

I think now more day to day as we have more product managers, more researchers, more engineers, a lot of my day is spent helping the team. And a lot of my PMs are more embedded with their research counterparts. And so coaching, supporting them is a big part of my day. Maybe a third of the time with calls with customers, just better getting a sense of, like, all of the different use cases, particularly what's emerging, like, what is not working today that, like, is really could be really pivotal. So spending a lot of time with, like, either businesses or startups.

And then I think, you know, especially in this environment, thinking about what's ahead.

Obviously, what a wild last and a half years. You know, I I guess as you reflect back on that, like, journey, are there some, like, key decision points that stick out to you as, like, oh, that that really, like, tip things or or or was was pretty pivotal?

What is the thing that they're not even asking for? Nobody's coming down our door in 2023 being like, gave us agenta coding. So I think that's one, just the early days of like being very clear focus around what is the bigger like opportunity. I think another one personally is when we did choose to ship computer use on the API and we shipped it as a beta feature. We knew that it didn't quite work completely yet, but we thought that this was really helpful for showcasing what AI could do and how it's just a different form factor.

I think we're still on that journey, but bringing that to the world. And there was a lot of discussions around like, hey, how do we make sure everything is very safe, etcetera. And so we did a lot of safety work, but it's impossible to capture every edge case. And so we had to put it out into the world a bit to then figure out what else we need to make safe as the capabilities expand. And so I thought that was like, just like a very authentic moment, making a bold decision that I think was the right one.

And yeah, think those are those are probably like two of the large things. There's a lot of other fun moments, but they're not like, it's like, I think pivotal in my mind.

Well, sticks out as a fun one?

So my team and I worked on Golden Gate Claude, which was an

Oh yeah.

Of our interpretability work. And that was very cool because I think the company was still less than 500 people, but it was starting to feel a bit bigger. And we shipped GoldenGate Cloud from model to the UI within less than a day.

Did you know it was gonna be this, like, kinda viral thing?

It went a little viral internally. So I think, like, usually, Anthropic employees have good like product sense and taste. So we were like, okay, if we like it, let's just try it. Maybe it's like, we get like maybe a couple 100 people who like geek out with us. And, but we we didn't know.

But yeah, it was like very grassroots. Some engineers, researchers, PM, and designer, we were like, okay, let's figure out how do we show this to users? And we needed to do it in the week because we had published a paper. So that was probably one of, like, the proudest moments.

It's been a fascinating conversation. We always like to enter interviews with a quick fire round where I basically just stuff in as many questions as I can before we, we run out of time. Yeah. And so maybe to to start, what's one thing you've, like, changed your mind on in AI in the last year?

I actually think we are closer to transformative long running AI than I expected even starting the year. Like, it actually feels like the building blocks are kind of there more. I feel that more now than than than in the beginning of the year.

You obviously have a a a range of sophistication of of end users of of the Cloud API. Are there things that the most sophisticated folks do that you wish that just all of our listeners, would would do with these models to use them more effectively?

I think there's, like, maybe, two two big things. I think the boldest, like, builders and users are constantly thinking about, like, not just what's working today, but actually might have things off the shelf that did not work before or some prototype that doesn't quite work, that they still kind of put together and test with new versions of models that come out. So we call this kind of prototyping, just having library user centric prototyping. And I think that has been something that's been really valuable to see of like because a lot of times these systems are not planned, it's almost like you have to discover the capability. And if you don't have a prototype, then you're not going to discover the thing.

You're always going to wonder, oh, I wonder when it's going to get really good at drug discovery. But if you don't have like some way to actually like test that each time, it's always going to be too abstract. So I think builders having very ambitious prototypes, product ideas, or features that might not work in the past, but just like having things like hackathon where you could actually test these ideas is really important. And then I think like being willing when you have a new version of a technology to like invest and actually potentially changing your product experience to like meet the intelligence tailwinds.

Yeah. What is model taste?

I think of model taste as like, just like, you know, product sense and like product taste is kind of developed over time. It's really like a continuously honed sense of model capabilities and a willingness to discover the capabilities by being hands on and also by understanding what users are trying to do and, like, continuously, like, iterating on that. What do you think model taste is?

Yeah. No. I mean, I love the I love the idea of, like, you know, getting your fingernails dirty with the models. Right? And just, like, you know, developing some intuitions to both, you know, what they can and can't do, and also the right ways to push them or build scaffolding around them, you know, in order to get the most out of them.

There's just some people that you you give everyone the same tools, and they always seem to just get way more out of them.

Yeah. I think it's like your willingness to experiment. I think kind of creatively, problem solving is not the perfect word, but people who are willing to kind of approach models as a new discovery.

Yeah.

And you're constantly trying new things. That's kind of a way of developing model taste.

What I'm struck by too, talked about folks having being able to try new things as the models come out. You know, I imagine it's not as simple as just like plugging in Opus 4.5 and seeing if it works. Because obviously there's lot of scaffolding one has to build. And so I imagine part of the art is also being able to have some intuition as to you know, I'm not just switching out the API key and up, it doesn't work. But I'm but I'm actually have some idea of, like, things to tweak that that actually might make it work.

This is why we actually internally have things like hackathons pretty regularly. I think we do a hackathon maybe every three or four months internally at Anthropic just because people have these, like, pent up ideas of like, I wonder if Claude can now do X. And you really need to give builders, engineers, product people a place to do that creative work and do that discovery work outside of their day to day. Because yeah, like, you know, nobody asks for a thing like agentic coding to work. It has to be a bit discovered, it has to be a bit, you know, hands on to figure out if like something is possible.

Yeah. And giving people the place to do that is really important.

And I guess you you talked about, obviously, long running agents, you know, being being closer, and all these things. What's, like, one thing you think of as a society, like, we're not talking about enough or one of the implications of of all this that maybe is under discussed?

I think we don't talk enough about the benefits of safety, not just of the fact of, hey, it's to make sure the model doesn't do bad things, but more also, like, what is the benefits of an aligned model? One of the key issues that is, like, an active research area is around things like sycophancy.

Right?

That like models will just tell you what you want to hear. And actually a well aligned, safe model, it's not, it's actually the opposite. So it's actually an independent thinker, right? And independent thinkers are actually how we have breakthroughs and how we have better ideas. And I don't think we really talk enough about why safety is actually really good for higher value intelligence too.

It's not just to constrain AI. It's actually a way to amplify quality of intelligence if it works well. Coming back to my example in the beginning, I asked Claude, Hey, how do we think about pricing? Here's two options. And it comes up with a third that was really good, that pushed my thinking.

And if we had a very sycophantic version of Opus 4.5, it might not have done that. Would have just agreed with me, right? And it's just like, I don't think sycophancy is solved by any means, but just like by investing in, like, making AI that's aligned, we could actually get to a better intelligence, I feel like we don't talk about.

I love that example. You know, obviously, I I think folks at Anthropic famously have, like, very aggressive ASI timelines. Like, do you do you fit into that bucket or or how do you think about that?

I would say my timelines have probably moved up this year based on things of what I'm seeing with thing models like Opus 4.5. I feel like the building blocks are actually closer than we think, and that it's actually more of like a product overhang or product opportunities to express it. How do you like again, build the right scaffolds or build the right ways to harness, like, the model quality? And so I'm more of, like, a barbell. I think, like, there's, like, a very large probability in short term and then, like, very large probability in long term.

Well, this has been fascinating. I feel like I I will delay the, the launch of the future generation of models if I keep you any too much longer here. But, I I just wanna make sure to leave the last word to you. Like, where can folks go to learn more about Opus four five, about you, about anywhere you'd like to point them. The mic is yours.

Yeah. I think our blog and our website, I think that, you know, we are pretty like, we like to like the words speak for itself. So I think following on on our website is the right place.

Yeah. I guess you guys famously don't do the, like, coming tomorrow or, like, the cryptic, you know, tweets before model launches that No. Everyone else seems to Well, awesome. This has been fascinating. Thanks so much for for the time.

Thank you so much. It was great chatting.