Fields medalist Yu Deng faults AI labs for hiding chains of thought

Fields medalist Yu Deng faults AI labs for hiding chains of thought

Yu Deng, a Fields Medal winner and professor of mathematics at the University of Chicago, gave a keynote by video link on October 25 at a forum hosted by The Intellectual and Mr. Science. He publicly criticized the way AI companies release mathematical results. A transcript was republished the next day by Chao News. He does not object to using AI; what he objects to is a company handing over results while explaining nothing about how it got them.

His example was a batch of results an AI company announced on October 6, covering more than 300 problems and more than 700 papers. The company declined to release its chains of thought, saying it wanted to prevent distillation. Deng argues that this reason does not hold up: if an AI has produced a major proof, academic convention says the ideas and the process behind it should be laid out.

The talk did not name the company. Reporting by Retraction Watch lines up with it: on October 6, OpenAI posted 722 preprints on GitHub covering 372 math problems.

What Deng is asking for

Deng's criticism has two layers. The first is academic norms: if an AI is going to be treated as a researcher, it has to explain where its ideas came from and guarantee that its proofs are correct. He noted that these papers were published without any guarantee of correctness, and that more than one has been withdrawn after errors turned up.

The second layer is the commercial relationship. Mathematicians pay to use the product and type their own ideas into it, and that material may end up in training. He said some researchers saw their ideas appear in the company's new releases a few weeks later, with nothing coming back to the users. This is his personal account, and it cannot be checked case by case from public information.

He laid out several principles for how AI should be used: AI is a tool for mathematicians and enthusiasts to use, and companies should not do research on users' behalf; users should decide whether the data they enter can be used for training; and contributions should be traceable, so it is clear what the AI and each person did, which would reduce priority disputes.

“I think the choice belongs to the user.”

He also argued that the mathematics community should develop open-source math models rather than depend on commercial companies. He acknowledged that open-source models are still well behind the strongest commercial ones in mathematical ability, and said that gap has to be closed through training and improvement.

A line running from summer to now

Connect the milestones of the past few months and the relationship between mathematicians and AI companies has shifted quickly.

In July, Deng was quite friendly in an interview. He said he had been stuck on a special case for several days when GPT produced a very simple proof. It did not generalize, but it offered a way to think about the problem.

In early September, OpenAI announced progress by AI on the Navier-Stokes problem, and mathematicians then criticized the announcement as premature. Around September 11, an open letter signed by 25 mathematicians drew press coverage. Deng and Terence Tao were both among the signatories, and the letter criticized AI companies for treating problem-solving as a capability benchmark, which squeezed the writing and verification that proofs require.

On October 7, the day after the preprints appeared, OpenAI withdrew three of them over a notation error and revised 14 others. Alex Townsend of Cornell University said the company should first release the portions that had been formally verified in Lean. Andrew Sutherland of MIT said that withdrawing and correcting papers is the responsible thing to do, while adding that many mathematicians were unhappy with OpenAI's earlier practices. An OpenAI spokesperson said the errors were found in an internal audit and described the process as iterative.

The day after Deng's talk, OpenAI announced three more results. The Yang–Mills one was not formalized in Lean.

The distillation worry is not baseless

The AI companies' concern has a factual basis. OpenAI has accused accounts linked to Moonshot AI of distilling reasoning chains, and Anthropic has also rolled out measures aimed specifically at blocking reasoning distillation. Publishing chains of thought does give competitors extra training material.

Deng's rebuttal rests on context: a product API can be protected against distillation, but when a paper claims to have solved a hard problem, readers have a right to check how it was reached. The two demands could perhaps be handled separately, for example by opening the reasoning records only to a paper's reviewers. That is just an idea; no company has publicly proposed anything like it. As of publication, we found no public response from OpenAI to the talk.

Sources: Chao News, The Intellectual, Retraction Watch, CocoLoop, ITHome. The counts of preprints and withdrawals follow Retraction Watch's figures.