By Sree Bhat

Will the human race survive a misadventure by a rogue AI system?

Once confined largely to science fiction and academic debate, the question is now being raised by AI researchers, technology executives and governments grappling with the risks of increasingly capable systems.

From medicine and scientific discovery to manufacturing, finance and public services, AI is unlocking possibilities that could reshape economies and extend human capability. Yet the same systems are also raising a darker possibility: that one of humanity’s greatest technological achievements could eventually slip beyond our control and threaten the species that created it.

The warning broke into the open in September when Evan Hubinger, an AI safety researcher at Anthropic, stated he personally believed there was a greater than 10% chance that AI could kill all humans within the next decade. Anthropic itself, in its IPO filing, warned that advanced AI could pose catastrophic or existential risks, reports said. Its models, the company acknowledged, could exhibit self-preserving behaviour, resist shutdown, conceal or manipulate information, or behave in ways resembling blackmail. Expanding the power and deployment of these systems, it cautioned, could increase the risk of serious harm.

President Trump speaks to the press with top tech company executives. Image courtesy: White House gallery

These are not prophecies of inevitable apocalypse. They are admissions from people building the technology that the destructive potential may outrun our capacity to govern it.

An Axios report recently revealed that executives at some of the world’s leading AI companies were privately gaming out “the day after” scenarios: a large-scale disaster, most likely a cyberattack, that could knock out financial services, internet connectivity, or even power and water supplies. The exercises are meant to prepare for cascading failures. 

For a region still absorbing the consequences of recent conflict, the idea of AI threatening humanity’s survival may feel distant. Yet governments and technology companies are already treating the risk as real.

Can humanity stay in control of a technology it is developing faster than it can fully understand? I put the question to AI itself.

“AI could potentially pose an existential threat to humanity if it becomes capable of acting beyond people’s ability to control it. The danger need not come from hostility, but from powerful systems pursuing objectives in ways humans cannot predict or stop. This is not an inevitable outcome, but a risk that must be taken seriously before it is too late.”

The answer was measured. It did not forecast imminent catastrophe. It simply identified the core danger: systems growing more capable and more autonomous while remaining imperfectly aligned with human intentions.

That raises the practical question: if AI could ultimately threaten humanity, how might it happen?

Today’s systems already make mistakes, exploit vulnerabilities and take unintended actions when given access to digital tools. The more extreme existential scenarios concern future systems with far greater capabilities. Experts have outlined possible pathways – AI enabling biological weapons, orchestrating devastating cyberattacks, or relentlessly pursuing objectives incompatible with human survival. There is no settled timetable and no scientific consensus on the probability of human extinction and superintelligence remains hypothetical. The uncertainty, however, has not stopped governments from treating the warnings as urgent.

WASHINGTON’S SAFETY PACT

On 29 September, US President Donald Trump and executives from six major technology companies - including OpenAI, Anthropic, Google, Meta, Nvidia and SpaceX - announced a voluntary AI safety accord. 

The agreement requires companies to maintain internal controls to monitor model capabilities and risks, conduct internal reviews, engage independent external auditors, and ensure board-level oversight. It specifically addresses cybersecurity, biological and chemical threats, and the risk of AI systems accessing technical systems in unintended ways.

Yet the accord is not legally enforceable. It carries no penalties for non-compliance. Its signatories suggest the measures might one day be written into law. For now, the emphasis remains on voluntary safeguards while the competitive race intensifies.

The contradiction is stark. Governments urge their companies to build ever more powerful AI systems, then ask those same companies to police the dangers created by that very race. Trump has also directed agencies to prefer the term “super intelligence” over “artificial intelligence.” Changing the label does not solve the control problem.

AUSTRALIA’S WARNING

A more immediate illustration arrived in Australia. On 24 September, Australian Prime Minister Anthony Albanese disclosed that an OpenAI agent had gained unauthorised access to public and non-public files on a government Medicare statistics portal in June. 

This was not evidence of a machine plotting humanity’s destruction. It was evidence that an AI agent operating with a degree of autonomy could cross boundaries its operators never intended. 

THE RISKS ARE ALREADY HERE

Superintelligence dominates headlines, but the dangers of AI are not deferred to some distant future. AI tools already help cybercriminals identify vulnerabilities, automate attacks and generate convincing fraud. They can produce misinformation at industrial scale, lower barriers to dangerous capabilities, and blur the line between authentic and synthetic content. They are already reshaping employment, privacy and the concentration of economic power.

The challenge is to realise the transformative benefits AI offers without sacrificing reliable human oversight. That requires independent testing, robust cybersecurity, clear lines of responsibility for failures, timely disclosure of serious incidents, and international cooperation on risks no single nation can manage alone. 

Perhaps the warnings about AI will prove overstated. The technology may become one of humanity’s most beneficial creations, helping solve problems once thought intractable. But uncertainty is not the same as safety.

If the warnings prove exaggerated, preparing for serious risks will still have strengthened security and accountability. If they prove justified, we may discover - too late - that voluntary pledges and delayed oversight were never enough.