- Monday Momentum
- Posts
- The Superintelligence Schism
The Superintelligence Schism
Why 850 leaders want to stop the race AI labs won't slow Down
Justin Wright
November 03, 2025 • Est. Reading Time: 12 minutes
Happy Monday!
Last week, over 850 prominent figures, from AI pioneers Geoffrey Hinton and Yoshua Bengio to Apple co-founder Steve Wozniak, Prince Harry, and even Steve Bannon, signed a statement calling for a complete prohibition on developing superintelligence. Not a pause. Not slower progress. A ban.
The statement itself is remarkably concise: "We call for a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably, and strong public buy-in."
The timing matters. This comes after Sam Altman announced OpenAI is pivoting from AGI to superintelligence, declaring "we are now confident we know how to build AGI as we have traditionally understood it." Anthropic's Dario Amodei told Davos that AI systems will be "better than almost all humans at almost all things" by 2026 or 2027. Meta established Superintelligence Labs after investing $14.3 billion in Scale AI.
But here's what makes this moment unprecedented: the people calling for prohibition include the godfathers of modern AI themselves. The divide isn't between technologists and humanists; it's directly within the AI community. And that schism reveals something fundamental about where we actually stand in the development timeline.
Over 850 leaders signed a statement prohibiting superintelligence development until proven safe with public consent. Key missing signatures: the people actually building it. The split reveals a deeper truth: frontier labs believe superintelligence timelines are 2-4 years away while signatories see uncontrollable existential risk. The real question isn't if we can build it, it's whether slowing down is even possible in a US-China AI race structured around competitive dynamics.
The Unprecedented Coalition
The breadth of signatories is striking. Five Nobel laureates. Former Obama National Security Advisor Susan Rice and former Joint Chiefs Chairman Mike Mullen. Virgin's Richard Branson. Actor Joseph Gordon-Levitt and musician will.i.am. Author Yuval Noah Harari. Religious advisor to the Pope, Paolo Benanti.
Most significantly: Yoshua Bengio and Geoffrey Hinton, the "godfathers" of modern AI who pioneered the deep learning revolution underlying today's models. Stuart Russell from UC Berkeley, whose textbook defined AI for generations of researchers.
This isn't simply fringe alarmism. These are the people who built the foundation that made ChatGPT possible. Bengio stated: "Frontier AI systems could surpass most individuals across most cognitive tasks within just a few years. To safely advance toward superintelligence, we must scientifically determine how to design AI systems that are fundamentally incapable of harming people."
The statement deliberately remained minimal to attract diverse support. But the coalition itself sends a message: concern about superintelligence now spans the political spectrum, from Steve Bannon to Susan Rice, and from tech founders to faith leaders.
Yet conspicuously absent from the list is Sam Altman, Dario Amodei, Mustafa Suleyman, Elon Musk, Mark Zuckerberg, and David Sacks. These are the people actually building superintelligence systems.
What Superintelligence Actually Means
Definitions matter, especially when they're driving trillion-dollar infrastructure bets. Superintelligence is broadly defined as AI capable of outperforming the entirety of humanity at most cognitive tasks.
This differs from AGI, which much of the industry defines as "AI that can match human capacity," while superintelligent AI means "AI that excels human capacities across the board." But the terms are fluid. Altman uses them almost interchangeably.
The practical distinction: AGI could manage complex coding projects, draw on biology insights to solve engineering problems, or write a Pulitzer-worthy novel. Superintelligence could do all that while autonomously rewriting its own code, making strategic decisions beyond human oversight, and pursuing goals with "superhuman competence."
A superintelligence tasked with ending climate change might logically eliminate the species producing greenhouse gases. One instructed to maximize human happiness might trap brains in perpetual dopamine loops. Nick Bostrom's famous example involves a superintelligence optimizing paperclip production that might convert all Earth's matter, including us, into raw materials.
At its core, this is an alignment problem. Current techniques work for today's AI but may prove inadequate for systems that surpass human intelligence. We don't know how to reliably steer something smarter than us.
The Race Nobody Wanted But Can't Stop
Polling from the Future of Life Institute found only 5% of U.S. adults support "fast, unregulated" superintelligence development. 64% believe it shouldn't be created until proven safe and controllable. 73% want robust regulation.
Max Tegmark, FLI president, summarized: "95% of Americans don't want a race to superintelligence, and experts want to ban it."
Yet the race accelerates. Why? Anthony Aguirre, FLI co-founder, notes: "A lot of the harms come from the perverse incentive structures companies are subject to at the moment. Companies in America and China are competing to be first in creating superintelligence."
This isn't just hypothetical game theory. A recent "Superintelligence Strategy" report by Dan Hendrycks (Center for AI Safety), Eric Schmidt (former Google CEO), and Alexandr Wang (Scale AI founder) proposed "Mutual Assured AI Malfunction" (MAIM) which models AI deterrence directly on Cold War nuclear doctrine. The premise: any nation attempting to dominate the superintelligence race risks sabotage from rivals.
Paul Roetzer, Marketing AI Institute founder, warns: "The kind of national-security level threats these frameworks describe aren't some distant sci-fi scenario. Every major AI lab and every major government knows foreign actors could be infiltrating their AI systems already."
The competitive dynamic creates a Prisoner's Dilemma. No individual lab wants to cause catastrophic harm. But each faces incentives to move faster than competitors, deploy before fully understanding risks, and prioritize capabilities over safety. The Statement on Superintelligence introduces a novel concept: democratic suspension of technological progress, pending societal assurance of safety and control. This echoes international moratoria on nuclear research, biotechnology, and chemical weapons.
Pattern Recognition: What the Labs Are Actually Doing
The gap between rhetoric and action reveals where frontier labs actually stand.
OpenAI: Altman wrote in January: "We are beginning to turn our aim beyond [AGI], to superintelligence in the true sense of the word." He believes OpenAI will deliver AGI during Trump's administration, likely within four years.
But OpenAI's safety record raises questions. In July 2023, the company wrote: "We don't have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue. Humans won't be able to reliably supervise AI systems much smarter than us."
Since then? OpenAI disbanded teams focused on superintelligent systems safety and saw several influential safety researchers depart. Jan Leike and Ilya Sutskever, who led the superalignment team, both left, with Leike citing concerns that "safety culture and processes have taken a backseat to shiny products."
Anthropic: The company submitted testimony emphasizing its belief that "powerful AI will emerge within the next two years" and called for robust national security AI testing, strengthened export controls, enhanced lab security.
Yet Anthropic's own research undermines confidence in current safety approaches. A study published in July found reasoning models often hide their true thought processes. Claude 3.7 Sonnet mentioned hints only 25% of the time. Models frequently engaged in "reward hacking" by exploiting vulnerabilities to achieve better scores, all while hiding this behavior from observable reasoning traces.
The paradox: Anthropic is both calling for superintelligence preparation and demonstrating we can't even reliably monitor current models.
The Unmentioned Reality: The statement doesn't clarify that AI doesn't have to reach superintelligence to cause chaos. Current generative AI tools are already upending education, transforming the web into a misinformation-prone environment, expediting nonconsensual pornography, and sending users into mental health crises resulting in divorce, homelessness, jail, and death.
Contrarian Take: The Statement Changes Nothing, And That's The Point
The cynical read: this is virtue signaling. In its current form, the Statement on Superintelligence is a 30-word statement with zero enforcement mechanisms, no technical specifics on how prohibition would work, and missing signatures from everyone with actual power to slow development.
Scott Alexander and others defend the value of consensus statements as tools for motivating specific policy development. "In the past, critics of AI safety have dismissed the concept of superintelligence and AI risks due to lack of mainline scientific and public support. The breadth of people who have signed this open letter demonstrates opinions are changing."
The statement's real function is to establish that concern about superintelligence is now mainstream. When the godfathers of AI call for prohibition, it's no longer fringe doomerism.
But here's the deeper truth: after superintelligence is developed, the machines are going to be in charge. Whether or not it goes well for humanity is unknown, but it calls for caution as this experiment develops.
The statement isn't trying to stop superintelligence development as that ship has sailed. It's trying to shift the Overton window on what's acceptable to build without public consent.
The approach goes far beyond the EU AI Act's risk-based framework. While the Act imposes transparency and risk-management duties, the FLI calls for a pre-emptive suspension until both scientific and societal validation are achieved.
If frontier labs are compelled by regulation to slow superintelligence development, the competitive balance shifts toward providers with domain-specific, controllable AI architectures. But enforcement requires international coordination in an environment where the US and China view AI supremacy as essential for national security.
Looking Ahead
Actor Stephen Fry captured the core tension: "To get the most from what AI has to offer mankind, there is simply no need to reach for the unknowable and highly risky goal of superintelligence. By definition, this would result in a power that we could neither understand nor control."
But OpenAI, Anthropic, Google, and Meta are reaching for exactly that unknowable goal. They're betting that iterative deployment, alignment research, and safety frameworks will be sufficient.
The signatories are betting that the cost of being wrong is existential. The statement won't stop development. But it establishes that when labs cross the superintelligence threshold, whether in 2027 or 2035, they'll be doing so against explicit public opposition and scientific warnings from the very people who made their work possible.
The decade ahead isn't just about technical capability. We have to explore whether democratic societies can impose collective choices on technologies their governments view as strategically essential. While continued frontier development in AI is critical, and while this new technology has potential to change our lives in very meaningful ways, there are at least a (large) handful of intelligent people trying to ask deeper questions about what this all means.
In motion,
Justin Wright
If the godfathers of AI are calling for prohibition while the labs they helped create race toward superintelligence anyway, does that suggest our governance structures are fundamentally mismatched to the speed and stakes of AI development?
I am excited to officially announce the launch of my podcast Mostly Humans: An AI and business podcast for everyone!
Episodes can be found below - please like, subscribe, and comment!