From assistant to autonomous developer in a few years

Just a few years ago, models like GPT-3 could barely write a coherent paragraph of text. Today, the most advanced systems independently perform complex tasks — they write and test code, coordinate with other AI agents, and in some areas surpass human experts. Claude Mythos, Anthropic's most powerful model to date, discovered thousands of critical vulnerabilities in all major operating systems and browsers.

It is precisely this rapid growth in capabilities that led Anthropic to issue a report warning of a scenario that experts call recursive self-improvement. In practice, this means that AI systems could begin to design, train, and improve their next generations by themselves — without humans fully understanding or being able to control the process. In other words: AI would become the creator of better AI, and human oversight would disappear.

"We believe it would be beneficial if the world had the ability to slow down or pause the development of cutting-edge artificial intelligence, so that social structures and AI safety research can keep pace with technological development," quoted Vietnamese television VTV from the Anthropic report, which was the first outlet to report on the material on June 5, 2026.

Four catastrophic scenarios that Anthropic fears

Just a few days after the VTV report, on June 10, Anthropic published an extensive document titled Policy on the AI Exponential, which identifies four categories of catastrophic risks associated with uncontrolled AI development:

Biological risk — the same capabilities that accelerate drug development can make it cheaper and easier to produce dangerous viruses by biological attackers. Anthropic therefore recommends gene synthesis screening and early warning systems for new pathogens.

Cyber risk — cutting-edge models today can detect software vulnerabilities on a massive scale. Defensively this is a benefit, but the same tool in the hands of attackers means threats to hospitals, energy grids, and other critical infrastructure.

Loss of control — the more capable AI systems become, the harder it will be to ensure they act in accordance with the intentions of their creators. Anthropic openly admits that research in this area is still in its infancy.

Automated research and development — AI systems are already automating the development of AI itself, which can dramatically amplify all three of the previous risks. It is a multiplier that accelerates the entire development cycle — including the risks.

Concrete proposals: transparency, independent audit, and government intervention

In its framework, Anthropic proposes a set of specific measures. Developers of cutting-edge models — defined as systems trained with computational power above 10²⁵ FLOPs at companies with AI revenues over $500 million — would have to meet four key conditions:

Transparency — publish safety test results, regularly report on risks, and publish safety frameworks. This is already required by laws in California and New York. Anthropic goes further and demands regular reports on the overall risk profile.

Independent audit — involve at least one qualified independent evaluator who reviews testing and risk reports. In parallel, governments should build an ecosystem of these independent auditors.

Infrastructure security — protect training data and model weights against cyberattacks, both from outside and inside the company. Models are now a valuable target even for state actors.

Government authority to halt deployment — this is the most controversial point. Anthropic proposes that the government be able to block or deter the deployment of models that pose a significant risk of catastrophic harm. At the same time, it warns against overly broad powers and proposes specific safeguards against misuse.

Why it's controversial

Not everyone welcomes the measures proposed by Anthropic with enthusiasm. Critics point to an apparent paradox: a company with a market value of almost a trillion dollars is calling for a slowdown in the race that it is itself leading. Some investors and experts warn that such regulations could inadvertently favor established players and stifle innovation by smaller companies and startups.

By a twist of fate, just one day after the framework was published — on June 12, 2026, the US government issued an export control directive that suspended access to the Fable 5 and Mythos 5 models, the most powerful systems that Anthropic has yet created. The company is thus experiencing firsthand exactly what it proposes for the entire industry.

What it means for Czechia and Europe

While the debate on AI regulation is only just getting underway at the federal level in the US, the European Union already has its own AI Act, which has come into force and introduces obligations for developers of high-risk AI systems. The framework proposed by Anthropic is in many respects more ambitious — particularly in the requirement for government authority to block dangerous models, which European legislation does not currently contain in this form.

For Czech companies and developers, the clear message is: AI regulation is not just a Brussels topic. Even the most valuable AI startup on the planet says that safety brakes are necessary. For smaller European players, this could mean an opportunity — if they can demonstrate that their models are safe and transparent, they could gain a competitive advantage in a regulated environment over those who underestimate safety.

The Czech Republic also launched the Czech AI Factory in Ostrava this year, which has the ambition to become part of the European AI infrastructure. It is precisely the emphasis on safety and transparency that could be a path for Czech projects to succeed in global competition.