AI research group Epoch AI on October 26 published a set of internal experiments: it handed frontier models the work its own team does day to day and scored the results against the same standard used to assess human employees. The top performers were Claude Fable 5.1 and GPT-6 Astra, at an average of 65%.
"A 65% score is still a failed task."
The article's authors are Kelly Hong, Greg Burnham and Lynette Bye, and it summarizes the first installment of Epoch's new "automation reports" series.
The experiment has a precursor. Anson Ho, a senior researcher at Epoch, earlier tried having AI take over his own job. The AI could not complete all of the tasks, and his judgment was that this would change before the decade is out. This time the personal trial was widened into a task list covering the whole team, scored with a single shared rubric, and the field was swapped for the six strongest models available today.
How the test worked
The tasks come from Epoch staff's everyday work and span five categories, including graphic design, planning research projects, coding and data analysis. The models had to gather context on their own, call tools and deal with the various snags built into each task. Scoring looks at how many items on the rubric an output satisfies, and that rubric was originally written to evaluate human employees.
Epoch admits this is far more laborious than running a standard benchmark, with the upside of being closer to real work. The article does not give the total number of tasks.
The results follow a clear pattern. On well-bounded tasks, such as writing code or analyzing data, the models do reasonably well. On open-ended ones, such as drawing out a pattern from a handful of examples or any job that calls for making a judgment call, they struggle noticeably. Asked to propose research directions, the models come up with ideas that are fairly useful; turning a direction into an experiment plan that can actually be run is where things tend to go wrong.
In one pilot experiment, GPT-6 Astra set its token budget too low, so 22% of runs returned no answer. It then wrote those failures up in its conclusions as "an important finding about model capability," without realizing the error was in its own settings.
Set beside standard benchmark scores
The same two models usually look much better on the public leaderboards their makers cite. OpenAI's newly released GPT-6.1 Terra completed 50.3% of tasks on Terminal-Bench 4.0 at its highest setting, and that was already being promoted as a major step forward. Computer-use benchmarks such as Agents' Last Exam and OSWorld likewise lean more and more on "completion rate" to tell their story.
Epoch's 65% cannot be converted directly into those numbers. The former is the share of rubric items satisfied; most of the latter measure the share of problems solved outright. Epoch includes a specific caution in the article: 65% does not mean two thirds of the work has been automated. Much of the score may come from the easy-to-earn items, and in terms of time spent and difficulty, 65% of the items covered is nowhere near 65% of the progress.
"Every one of those tasks would be rejected or revised if a human had submitted it."
There has been one practical change too. Epoch has already dropped a basic chart-generation task from the test set, because Fable 5.1 produces an acceptable chart in one go most of the time, and the team now has AI partly handling that step.
The authors' view is that with each round of model improvement, more tasks will cross the line of being 100% acceptable. Only then will people hand them over to AI, and that is when the economic effects begin to show. Epoch plans to track, item by item in later reports, which tasks cross that line first.
Sources: Epoch AI research column, CocoLoop, OpenAI product announcement; Epoch's rubric notes, which verify how the 65% score is calculated.