Anthropic says its AI may soon achieve self-improvement, but is committed to maintaining human involvement for the foreseeable future.

Earlier this week, Anthropic announced an amendment to its initial public offering (IPO) registration documents. While much of the focus has been on updated financial figures with revenue and expenses skyrocketing, developer Claude also revealed notable information in small print. That's a warning that they're getting close to creating a recursively self-improving (RSI) model.

According to Tech Crunch, RSI is the ability of AI to gradually improve itself through successive rounds of development without human intervention. For pessimists, RSI signals irreversible disaster, when AI grows so powerful that it exceeds human control, ignoring the damage it causes in the process of reaching its goal. On the contrary, optimists believe that RSI shows great potential, bringing productivity beyond imagination, promoting scientific discovery and many other achievements.

In the revision of the IPO filing, Anthropic added new information to the "Risk Factors" section, stating that it had used AI to develop new models that had a leap in capacity that they did not realize at the time, and stated that the company could soon reach RSI according to its own definition.

"Our internal estimates and projections of future models, not current models, suggest we may be approaching the definition of 'significantly accelerated progress with AI' outlined in the Responsible Expansion Policy," the company wrote.

However, Anthropic also said that none of their models has exceeded the RSI threshold, and is committed to maintaining human participation in the research and development process in the near future. "We are considering updating our Responsible Expansion Policy to ensure humans remain in control of advanced model development at Anthropic," the revised filing states.

Despite being one of the world's leading advanced AI laboratories, Anthropic is quite cautious. In a June blog post, the company urged labs to consider establishing a coordination mechanism to slow or halt development if AI begins to improve itself at a rate that exceeds society's ability to manage risks. The company said that AI with the ability to self-improve is a major development in the history of technology, but also increases the risk of humans losing control of artificial intelligence.

According to WSJ, Anthropic's leaders, including CEO Dario Amodei, have repeatedly warned about potential risks from AI and proposed widespread cooperation from society to solve them. He fears AI could worsen inequality and eliminate up to half of low-level white-collar jobs.

In September, Amodei warned that AI is increasingly capable of self-improvement and could quickly go beyond human control. He mentioned cases of AI agents escaping isolated environments and attacking external platforms, typically the Hugging Face incident in July, saying this was the main reason for slowing down the speed of development of advanced models.

Jack Clark, co-founder of Anthropic, also regularly speaks out about the potential dangers of AI and the consequences when AI systems reach the threshold of self-improvement, saying that this will change the world.

"That kind of technology has never existed before, but I believe it could appear within the next two years, or even sooner," he said in a lecture in London in May.

Thu Thao summary