Claude stress-tested Docs for two weeks; launch had zero rollbacks

AI coding illustration: a robotic arm fitting a red block into a structure

On November 17, Anthropic engineer Vivek Nair published a post-mortem on the Claude developer blog describing how his team kept a single Claude Code session load-testing, hunting for problems and rewriting code for the two weeks before Claude Docs launched, and how the service then held up under real traffic.

Claude Docs is a document editor Anthropic introduced on September 16. Users, colleagues and Claude can work in the same document together, and any Claude session, from the chat interface to Claude Code, can open that document to read, write and comment in real time. Three weeks before launch it was still a proof of concept running on a single node, used by a few thousand employees inside the company.

The author sums up the results after launch:

“Across 2 billion requests we saw only 60 server errors, and we never had to roll back.”

One session, looping for two weeks

Stress testing began on September 4, roughly two weeks before launch. The environment was a staging cluster mirroring production, generating more than a million synthetic documents a day.

The coordinator was a continuously running Claude Code session whose loop had only a few steps: pick a scenario, run the load tool, read the monitoring data, propose a fix, deploy it to staging, then start over. Most of the actual work was handed to subagents; in the busiest week there were typically 11 running at once, peaking at 30. Monitoring data came in through Datadog's MCP, and proposed changes were reviewed in a separate session. The author says the model used was an internal research model with capabilities roughly comparable to Claude Fable 5.2.

The failure scenarios the team wrote down in advance included 10 million new documents created in a single day, 22,000 live connections reconnecting at once, and the server being killed while loading a popular document with several hundred viewers, with the database stalling at the same time.

There was one line on permissions. For easily reversible changes, Claude wrote, deployed and retested on its own; changes touching the database schema, key layout, or how documents are stored and generated needed human approval. The team also gave Claude two constraints it could not break: even documents with very long histories must open almost instantly, and any edit the server has acknowledged must be provably still there in tests.

How one memory problem got fixed

The example the post walks through in full is memory. A single large document took up about 3 GB on one pod.

Claude's first diagnosis was that a certain cache was too large, so it shrank that cache to make the server evict documents from memory sooner. The next round of tests showed that memory was actually stuck in a different cache inside a dependency library, and the change was reverted. After about three days of iteration, finer-grained memory profiling found the root cause: on every keystroke the document was being copied into several data structures, adding a few MB each time. Claude changed those structures to copy-on-write so they share a single reference, and peak memory for that document dropped to about 180 MB.

By the author's estimate, the number of memory-optimized pods needed for launch fell from more than 500 to about 50. After launch the entire Docs service ran on roughly 50 pods; the first million documents appeared in about 30 hours, and by September 22 users had created more than 4.6 million documents.

What the inference bill came to

Using the API list price for Claude Fable 5.2, the author calculates that this loop consumed about $125,000 in inference from late August through launch day. He also admits that a lot of tokens were burned while the workflow was being figured out, and that “we could have spent about half as much.”

A rough back-of-envelope check: the roughly 450 memory-optimized pods saved would, at typical large-memory instance prices from cloud providers, very likely cost more than $125,000 a month in rent. The exact figure depends on instance size, and the post does not give the pod specification, so this is only an order-of-magnitude estimate.

For much smaller teams, the author estimates that similar work on the scale of two weeks would cost roughly $3,000 to $10,000 at list price, noting “I haven't actually run this,” so he has not tested that figure himself.

The author credits recent models' long-task ability for making the approach work: a few months ago, a model running ten or twenty rounds in a long loop would forget what it was optimizing, and he had to keep restating the goal. This time Claude kept hold of every constraint through multiple context compactions. The service is written in Rust with as few dependencies as possible to keep monitoring signals cleaner, though the author also says the team never formally compared Rust with Go.

Sources: Claude developer blog, CocoLoop, Anthropic product documentation. Request counts, error counts, pod numbers and inference costs are self-reported by the Anthropic engineering team and have not been independently verified.