Anthropic's newest model, Claude 3.5 Sonnet, is its best yet as measured across a range of AI benchmarks.
techcrunch.com
Anthropic claims its latest model is best-in-class
Kyle Wiggers
7:00 AM PDT • June 20, 2024
Comment
Image Credits: Anthropic
OpenAI rival
Anthropic is releasing a powerful new generative AI model called Claude 3.5 Sonnet. But it’s more an incremental step than a monumental leap forward.
Claude 3.5 Sonnet can analyze both text and images as well as generate text, and it’s Anthropic’s best-performing model yet — at least on paper. Across several AI benchmarks for reading, coding, math and vision, Claude 3.5 Sonnet outperforms the model it’s replacing, Claude 3 Sonnet,
and beats Anthropic’s previous flagship model
Claude 3 Opus.
Benchmarks
aren’t necessarily the most useful measure of AI progress, in part because many of them test for esoteric edge cases that aren’t applicable to the average person, like answering health exam questions. But for what it’s worth, Claude 3.5 Sonnet
just barely bests rival leading models, including OpenAI’s recently launched
GPT-4o, on some of the benchmarks Anthropic tested it against.
Alongside the new model, Anthropic is releasing what it’s calling Artifacts, a workspace where users can edit and add to content — e.g. code and documents — generated by Anthropic’s models. Currently in preview, Artifacts will gain new features, like ways to collaborate with larger teams and store knowledge bases, in the near future, Anthropic says.
Focus on efficiency
Claude 3.5 Sonnet is a bit more performant than Claude 3 Opus, and Anthropic says that the model better understands nuanced and complex instructions, in addition to concepts
like humor. (AI is
notoriously unfunny, though.) But perhaps more importantly for devs building apps with Claude that require prompt responses (e.g. customer service chatbots), Claude 3.5 Sonnet is faster. It’s around twice the speed of Claude 3 Opus, Anthropic claims.
Vision — analyzing photos — is one area where Claude 3.5 Sonnet greatly improves over 3 Opus, according to Anthropic. Claude 3.5 Sonnet can interpret charts and graphs more accurately and transcribe text from “imperfect” images, such as pics with distortions and visual artifacts.
Michael Gerstenhaber, product lead at Anthropic, says that the improvements are the result of architectural tweaks and new training data, including AI-generated data. Which data specifically? Gerstenhaber wouldn’t disclose, but he implied that Claude 3.5 Sonnet draws much of its strength from these training sets.
Image Credits:Anthropic
“What matters to [businesses] is whether or not AI is helping them meet their business needs, not whether or not AI is competitive on a benchmark,” Gerstenhaber told TechCrunch. “And from that perspective, I believe Claude 3.5 Sonnet is going to be a step function ahead of anything else that we have available — and also ahead of anything else in the industry.”
The secrecy around training data could be for competitive reasons. But it could also be to shield Anthropic from legal challenges — in particular challenges pertaining to
fair use. The courts have yet to decide whether vendors like Anthropic and its competitors, like OpenAI, Google, Amazon and so on, have a right to train on public data, including copyrighted data, without compensating or crediting the creators of that data.
So, all we know is that Claude 3.5 Sonnet was trained on lots of text and images, like Anthropic’s previous models, plus feedback from human testers to try to “align” the model with users’ intentions, hopefully preventing it from spouting toxic or otherwise problematic text.
Image Credits:Anthropic
What else do we know? Well, Claude 3.5 Sonnet’s context window — the amount of text that the model can analyze before generating new text — is 200,000 tokens, the same as Claude 3 Sonnet. Tokens are subdivided bits of raw data, like the syllables “fan,” “tas” and “tic” in the word “fantastic”; 200,000 tokens is equivalent to about 150,000 words.
And we know that Claude 3.5 Sonnet is available today. Free users of Anthropic’s web client and the Claude iOS app can access it at no charge; subscribers to Anthropic’s paid plans
Claude Pro and
Claude Team get 5x higher rate limits. Claude 3.5 Sonnet is also live on Anthropic’s API and managed platforms like Amazon Bedrock and Google Cloud’s Vertex AI.
“Claude 3.5 Sonnet is really a step change in intelligence without sacrificing speed, and it sets us up for future releases along the entire Claude model family,” Gerstenhaber said.
Claude 3.5 Sonnet also drives Artifacts, which pops up a dedicated window in the Claude web client when a user asks the model to generate content like code snippets, text documents or website designs. Gerstenhaber explains: “Artifacts are the model output that puts generated content to the side and allows you, as a user, to iterate on that content. Let’s say you want to generate code — the artifact will be put in the UI, and then you can talk with Claude and iterate on the document to improve it so you can run the code.”
The bigger picture
So what’s the significance of Claude 3.5 Sonnet in the broader context of Anthropic — and the AI ecosystem, for that matter?
Claude 3.5 Sonnet shows that incremental progress is the extent of what we can expect right now on the model front, barring a major research breakthrough. The past few months have seen flagship releases from Google (
Gemini 1.5 Pro) and OpenAI (GPT-4o) that move the needle marginally in terms of benchmark and qualitative performance. But there hasn’t been a leap of matching the leap from
GPT-3 to
GPT-4 in quite some time, owing to the rigidity of today’s model architectures and the immense compute they require to train.
As generative AI vendors turn their attention to
data curation and
licensing in lieu of promising new scalable architectures, there are signs investors
are becoming wary of the longer-than-anticipated path to ROI for generative AI. Anthropic is somewhat inoculated from this pressure, being in the enviable position of
Amazon’s (and to a lesser extent
Google’s) insurance against OpenAI. But the company’s revenue, forecasted to reach
just under $1 billion by year-end 2024, is a
fraction of OpenAI’s — and I’m sure Anthropic’s backers don’t let it forget that fact.
Despite a growing customer base that includes household brands such as Bridgewater, Brave, Slack and DuckDuckGo, Anthropic still lacks a certain enterprise cachet. Tellingly, it was OpenAI — not Anthropic — with which
PwC recently partnered to resell generative AI offerings to the enterprise.
So Anthropic is taking a strategic, and well-trodden, approach to making inroads, investing development time into products like Claude 3.5 Sonnet to deliver slightly better performance at commodity prices. Claude 3.5 Sonnet is priced the same as Claude 3 Sonnet: $3 per million tokens fed into the model and $15 per million tokens generated by the model.
Gerstenhaber spoke to this in our conversation. “When you’re building an application, the end user shouldn’t have to know which model is being used or how an engineer optimized for their experience,” he said, “but the engineer could have the tools available to optimize for that experience along the vectors that need to be optimized, and cost is certainly one of them.”
Claude 3.5 Sonnet
doesn’t solve the hallucinations problem. It almost certainly makes mistakes. But it might just be attractive enough to get developers and enterprises to switch to Anthropic’s platform. And at the end of the day, that’s what matters to Anthropic.
Toward that same end, Anthropic has doubled down on tooling like its
experimental steering AI, which lets developers “steer” its models’ internal features;
integrations to let its models take actions within apps; and tools built
on top of its models such as the aforementioned Artifacts experience. It’s also hired an Instagram co-founder as
head of product. And it’s
expanded the availability of its products, most recently bringing Claude to Europe and establishing offices in London and Dublin.
Anthropic, all told, seems to have come around to the idea that building an ecosystem around models — not simply models in isolation — is the key to retaining customers as the capabilities gap between models narrows.
Still, Gerstenhaber insisted that bigger and better models — like Claude 3.5 Opus — are on the near horizon, with features such as web search and the ability to remember preferences in tow.
“I haven’t seen
deep learning hit a wall yet, and I’ll leave it to researchers to speculate about the wall, but I think it’s a little bit early to be coming to conclusions on that, especially if you look at the pace of innovation,” he said. “There’s very rapid development and very rapid innovation, and I have no reason to believe that it’s going to slow down.”
/cdn.vox-cdn.com/uploads/chorus_asset/file/25406818/STK051_TIKTOK_CVirginia_C.jpg)
:format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/25406818/STK051_TIKTOK_CVirginia_C.jpg "Vector art of the TikTok logo.")
:format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/25494375/Stock_Avatars.png "A selection of the Symphony Digital Avatars being introduced by TikTok."){kind=avatar}
:format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/25494377/AI_Dubbing_.png "The language translation options for TikTok’s Symphony Digital Avatars")
/cdn.vox-cdn.com/uploads/chorus_asset/file/25498067/Claude_3_5_Model_Selector.png)
:format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/25498067/Claude_3_5_Model_Selector.png "A screenshot of the Claude app showing 3.5 Sonnet selected.")
:format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/25498069/Claude_3_5_Sonnet_Eval_General.png "A screenshot showing various benchmark scores for Claude 3.5 Sonnet and other AI models.")
:format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/25498070/Claude_3_5_artifacts.jpg "A screenshot showing a preview of a document alongside an AI chat.")
/cdn.vox-cdn.com/uploads/chorus_asset/file/25498203/snapchat_on_device_ai.png)
:format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/25498203/snapchat_on_device_ai.png "An image showing Snapchat’s on-device AI model generating a filter based on the prompt “50s sci fi film”")
:no_upscale():format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/25498054/snapchat_lens_gif.gif)
:format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/25498174/snapchat_studio_immersive_ml.png "This is Snapchat’s “Immersive ML” effect using the prompt “Matisse Style Painting.”")
/cdn.vox-cdn.com/uploads/chorus_asset/file/13292779/acastro_181017_1777_brain_ai_0002.jpg)
:format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/13292779/acastro_181017_1777_brain_ai_0002.jpg "An illustration of a cartoon brain with a computer chip imposed on top.")