AI giant says government can block a bot’s use ‘if it could kill more than 50 people’

Anthropic, the AI giant behind the chatbot Claude, has a few rather specific reasons why the US could stop one of its models.

The company has reportedly pitched a bill that would let a judge ‘block or deter dangerous deployments of the most powerful models’.

But only if it ‘poses a foreseeable, imminent risk of more than $1billion in damages, or of killing or seriously injuring more than 50 people’, according to POLITICO.

The threat must be a result of the model creating or releasing a biological weapon, the AI breaking free of human control or a cyberattack.

Anthropic says the law would be in line with a safety framework it has been asking federal and state politicians to put into law since June.

‘Catastrophic risks’ AI poses to humanity should be defined by SB 53, a California law that ensures the safe development of the tech, it says.

The legislation defines such incidents in the same way Anthropic’s draft bill does, including AI-assisted nuclear weapons and serious crimes.

What does the draft bill say?

Congress, which Anthropic once described as ‘built for a slower world’, has yet to pass a federal law on AI.

Sources told POLITICO that, if passed by Congress, its bill would task experts to evaluate Anthropic and its bots.

Anthropic could pay these evaluators, ‘so long as the payment is not contingent on the evaluations’ results’, POLITICO said.

If they deem an Anthropic model an ‘imminent’ risk, they must notify the Department of Commerce within 72 hours.

Commerce officials could ask a DC judge to halt development of the model – a ‘department, agency, or officer’ would not have this power.

AI labs would also have to create safety frameworks and publish transparency reports.

Anthropic’s bill only applies to cutting-edge ‘frontier’ AI models, which require powerful computers to run.

Other types of AI, like general-purpose chatbots, would not be covered.

POLITICO did not specify which lawmakers Anthropic has approached.

Anthropic bosses have increasingly pushed for US-wide regulation on AI as its technology advances at a mind-bending speed.

Former and current Anthropic employees have warned that AI has a ‘10% chance’ of killing all humans by 2030.

Among their gravest concerns, as AI doomsayers also told Metro, is AI tools being used to create known or entirely new biological pathogens.

Anthropic said last month it had stopped potential plots by scientists to conduct research that could have helped develop bioweapons this year.

Others fear that AI models could go ‘rogue’ – as they already are.

OpenAI revealed in June that its bots hacked Hugging Face, a library of AI tools, so it could cheat on a security puzzle and get away with it.

AI safety experts and developers have also raised the alarm over the technology not sharing humanity’s values and ethics, called alignment.

Anthropic and the US Department of Commerce have been approached for comment.

Get in touch with our news team by emailing us at webnews@metro.co.uk.

For more stories like this, check our news page.