Anthropic's "When AI builds itself" essay makes trust the limit on autonomous agents

On June 4, 2026, Anthropic published an essay called "When AI builds itself" (anthropic.com/institute/recursive-self-improvement). It ends by arguing that a coordinated slowdown or pause on frontier AI would be a good thing if it could be made to work. The pause got the attention. The data behind it matters more, because it shows autonomous agents getting more capable faster than people are learning to trust them. Its headline numbers come from METR's time-horizon measure, which the essay cites: the length of software task, in human working time, that a model can complete on its own with 50% reliability. Claude Opus 3, March 2024: tasks that take a person about four minutes. Claude Sonnet 3.7, March 2025: about an hour and a half. Claude Opus 4.6, March 2026: about 12 hours. That length has been doubling roughly every four months, up from every seven months on the earlier trend. The essay adds more of Anthropic's own data. As of May 2026, Claude authored more than 80% of the code merged into Anthropic's codebase. Claude's success rate on the most open-ended Claude Code tasks reached 76% in May 2026, up 50 percentage points in six months. On a fixed code-optimization test, Claude Opus 4 averaged about a 3x speedup in May 2025, and Claude Mythos Preview reached about 52x by April 2026, though the essay itself warns against anchoring on the absolute multiple. Anthropic lays out three scenarios. In the first, the trend stalls and society catches up with today's AI; Anthropic says it doesn't believe that's likely. In the second, gains compound, and 100-person companies could do the work of 10,000- or 100,000-person organizations; the essay says the evidence points this way. In the third, AI becomes capable of full recursive self-improvement and builds its successors, and the pace is set by available compute rather than human research speed. Hence the slowdown argument. The essay says it would require "multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions," and that each could verify the others had stopped. That's a very high bar in an industry racing toward IPO valuations. I think they're being conservative in one place. The task-length curve doesn't need recursive self-improvement to reshape everything. If the four-month doubling holds, autonomous task length goes from 12 hours to multi-day work within a year, and to week-long work within 18 months. At that point the question stops being whether the model can do the task. It becomes whether I trust an autonomous agent to spend a week on it without a check-in. That trust question is the frontier now, and the block is mostly human. An agent you'd leave alone for a week has to be calibrated, easy to audit and easy to interrupt. The labs that figure out trustworthy week-long agent work will own the next decade. The labs that keep pushing capability without solving trust will ship features nobody uses. Anthropic is already hedging that bet, pause or no pause. A 76% success rate doesn't help me if I can't trust the 24% failure mode. They know this, and the essay is honest about it. Whatever you think of the pause, the essay is the most concrete public account of where a frontier lab thinks the next two years go. Read it for the curve.