Warnings about the potential dangers of artificial intelligence advancing to a stage known as superintelligence (ASI) have intensified in recent days. Concerns center on the possibility that its capabilities could exceed human control, leading to catastrophic scenarios that threaten humanity’s future, according to a report in The Guardian.
During a session in the British Parliament, officials and experts cautioned that superintelligence could pose a risk equal to, or even greater than, that of nuclear weapons. Former UK Defence Secretary Des Browne said that while in office he viewed nuclear war as the greatest threat to humanity, he now believes superintelligence poses a threat of the same magnitude, and possibly greater.
Professor Stuart Russell, a computer scientist at the University of California, Berkeley, warned of the potential for a disaster on the scale of Chernobyl, or an even more severe scenario in which humanity irreversibly loses control over AI systems.
These warnings came during a session organized by the Control AI coalition in London, a pressure group advocating for international rules to govern artificial intelligence. The group supports a bill under discussion in the UK Parliament aimed at banning the development of superintelligence.
Warning of Potential Human Extinction
Evan Hubinger, who leads alignment research at Anthropic, sparked widespread debate after saying he believes there is a probability greater than 10% that this technology could lead to the killing of all humans within the next decade.
Hubinger’s statement followed the resignation of 28-year-old researcher Jacob Coxson from Anthropic, who had previously worked at OpenAI. Coxson said the two companies are not acting responsibly and are gambling with our lives.
Coxson noted that the risks of AI are well known inside Anthropic, but accused the companies of being caught in a race to reach superintelligence first. However, he expressed optimism that major American labs could coordinate efforts to slow the pace of this race.
Estimates vary on when superintelligence might be achieved. Some experts believe it could happen within a few years, while others expect it to take more than a decade.
Calls for an International Treaty
In the UK, Labour MP Darren Jones called for a multilateral treaty for the orderly and safe development of superintelligence. He stressed that the goal is not to halt innovation or scientific research, but to adopt an approach that prioritizes safety as technology accelerates.
Jones said the current debate ranges from warnings of human extinction to dismissing concerns as mere hype, but he emphasized that governments must intervene now.
British government concerns were raised over Anthropic not providing its latest model, Mythos 5.1, to the UK AI Safety Institute for pre-launch testing. The Cabinet Office confirmed that only a limited number of American institutions had access to the model, noting ongoing cooperation between the AI Safety Institute and companies in the sector, including Anthropic.
In the United States, Senator Bernie Sanders escalated his campaign demanding AI regulation, calling on Congress to act. Sanders cited polls indicating that 81% of Americans believe politicians should take action on the technology.
Sanders said the future of humanity, its economy, environment, democracy, and privacy should not be determined by a handful of greedy people without public participation.
Experts Caution Against Exaggeration
Even with the seriousness of the warnings, some experts have taken a more reserved stance, cautioning against being drawn into catastrophic scenarios.
Dr. Andrew Rogoyski of the Surrey Institute for AI said current systems are still far from possessing the flexibility of human capabilities, let alone the ability of humans to work collectively. He predicted what he described as a great disappointment, as advanced AI may prove too expensive or not useful enough to continue in its current form.
David Barber, director of the Sofair AI research lab, warned against treating technology as a threat to be eliminated. He stressed that it has become highly important and beneficial, but unrestricted use to access the internet and various systems could create problems requiring more control and oversight.
Sandra Wachter, a professor at the Oxford Internet Institute, said she does not believe in catastrophic scenarios, but stressed that there are real and urgent risks associated with AI, including its environmental impact, the spread of misinformation, and its effect on jobs.
She affirmed that these issues are real, urgent, and need to be addressed now, warning that doomsday scenarios could distract from the actual problems AI poses today.
Commentator and AI researcher Gary Marcus sees a fundamental difference between a superintelligence aligned with human interests, if achieving that is possible at all, and one that is not aligned. He noted that the first possibility could be positive for humanity, but the current reality, in his view, offers only an abundance of media hype versus a clear lack of caution and responsibility.