Andrew Charlton flags requiring AI companies to run tough ‘risk management’ systems

Assistant Minister for Science, Technology and the Digital Economy, Andrew Charlton, has flagged the government is looking to regulate AI companies by forcing them to run rigorous “risk management” processes.

Called “systems regulation”, the approach would place “the onus on companies to build and run a rigorous process for finding, testing, reporting on and managing the risks of their systems”.

They would then be held “accountable for whether that process works,” Charlton said in a speech to the Sydney Trust and Safety Festival on the Regulation of AI.

The government has argued strongly that AI must have guardrails, with this being a major pitch from Prime Minister Anthony Albanese when he was in New York recently.

Charlton said “systems regulation” was a model used in health and safety, security of critical infrastructure, supervision of systemic banking risks and aspects of aviation safety.

“The goal is to ensure companies are meeting expectations of safety, without trying to specify every hazard.

"Government sets the standard those processes must meet, and ensures companies have robust processes in place. The question isn’t only ‘did something go wrong’ – but is there a serious system in place to detect risks and prevent incidents occurring.”

Arguing the benefits of this approach, Charlton said voluntary regulation failed “the incentive test”.

“You cannot ask firms to volunteer against their own competitive interest amid a manic race involving trillions of dollars and then be surprised when they do not.”

He said while detailed prescriptive rules could give clarity, they could become immediately out of date, given how fast the technology was moving.

“For these reasons, there is much that Australia can learn from systems-based regulatory approaches, that puts the onus and accountability on companies for managing how they develop products and services safely.”

We also need to be part of the action

But a regulatory framework was not enough,

“Relying on regulation alone is like building a fence around a horse, only to realise that horse has wings.”

The government was very aware of that reality, Charlton said.

From the perspective of government, regulation cannot be our only strategy when it comes to keeping Australians safe. This must be supplemented by both leverage and capability. If Australia wants a say in how these technologies are built and governed, we need to be a country that can host and build them, not only regulate them.

When it comes to leverage, technology that is housed overseas cannot always be regulated as effectively as technology Australia has a role in developing at home. And this underscores the importance of Australia having a seat at the table when it comes to AI safety, to shape it in our national interests.

Michelle Grattan does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.