While we were sleeping the world was set on fire.

Jason Wei and OpenAI just tweeted a vision of artificial intelligence so powerful, so absent of regulation or oversight, that it could set into motion a radical transformation of humanity’s trajectory.

We are faced with the absolutely insane idea that advanced reinforcement learning, with nearly limitless computational resources, operating in an environment impervious to manipulation, is evolving into something both astonishing and terrifying.

In a recent tweet from AI researcher Jason Wei, he captures the paradox and danger of this possibility: “Magic is what happens when an unstoppable RL optimization algorithm powered by sufficient compute meets an unhackable RL environment.”

When a piece of technology begins to produce outcomes that even its creators struggle to predict or control, it’s time to hit the brakes.

Part of the core of reinforcement learning, a discipline that trains algorithms to solve problems by maximizing rewards in a given environment, lies in its capacity to unearth novel strategies beyond the conscious design of human engineers.

When such algorithms are fed huge amounts of data coupled with advanced computational infrastructures, they quickly surpass human-level performance in specific tasks.

Their self-improving nature means they become adept at learning, refining, and optimizing in ways that is genuinely astonishing, producing results that look, from the outside, created out of thin air.

We are at a point where AI has become inscrutable and unpredictable, and the promise of “magic” is terrifyingly unsettling. The possibility that this technology is being unleashed by a relatively small group of individuals without the benefit of comprehensive governance or oversight is unsettling.

Most paradigm-shifting technologies—nuclear energy, the internet, and genetic engineering, to name a few—have historically triggered significant debates and regulatory efforts by government bodies, academic institutions, and international organizations which have tried to understand and limit the enormous risks.

The existence of at least some oversight has generally cooled down the more cataclysmic scenarios. In the case of advanced AI, I fear that the breakneck pace of development, combined with the isolated nature of specialized research labs, spells immense danger.

You should be in awe, but you should also be worried.

The real potential for artificial intelligence to exploit vulnerabilities, shape global systems, and make decisions that affect billions of lives without transparent accountability is happening now.

If the environment in which an AI agent operates is insulated from outside interference—what Wei terms an “unhackable RL environment”—it will learn to perform actions that disregard or bypass user directions and set limits..

Even if designed with benevolent intentions, a sufficiently powerful system could execute its optimization process in ways that produce extreme collateral harm.

Maximizing efficiency, for instance, could translate into real-world strategies that inadvertently disrupt economic markets, override personal autonomy, or concentrate political power in unprecedented ways.

Remember that these systems optimize for abstract objectives—growth, engagement, resource allocation—so they can move swiftly and impassively, unconstrained by human-level ethical considerations.

The insularity of the AI research community. When bright, driven individuals devote themselves almost exclusively to uncovering new technical feats, they lose sight of the broader societal ramifications.

The “need” for breakthroughs, the chase for academic prestige, and even the commercial incentives behind corporate-funded AI labs creates an echo chamber that amplifies excitement as critical voices are lost.

Researchers experiencing rapid success in specialized benchmarks—surpassing human performance in strategy games, programming tasks, or data analysis—are often swept up in their own momentum.

Meanwhile, the bulk of society remains only dimly aware of the depth of this progress, far removed from the specialized jargon and complex models that shape these developments.

By the time public attention is galvanized, the technology’s deployment will be so widespread or entrenched that attempts at regulation are completely pointless and futile.

This means that our frameworks for understanding and shaping the future may prove completely inadequate.

The speed of AI advancement is exponentially accelerating, and common historical transitional pauses in progress are gone.

Without a strong collective effort to channel progress responsibly, runaway systems will emerge, that are either too complex or too alien for humans and governments to grasp.

Calls for transparency, bias mitigation, and fairness are completely overshadowed by the relentless push toward optimization at all costs.

Will democratic institutions, or human civilization as a whole, be able to survive?

We have reached a point where large tech companies operate across national borders and have become more powerful than most governments in terms of resources and global impact and even if regulation were achievable in one country, AI research in other nations and hands will continue, making the entire effort futile.

The globalized nature of research, the proliferation of open-source frameworks, and the competition for technological dominance complicate the scenario further.

Who will control open source?

Sure, all the self-proclaimed enthusiasts of AI continue to argue that it will bring immeasurable benefits in science, medicine, and climate solutions, but that argument ignores my worry.

What happens if a technology of immeasurable power gets deployed by a rogue organization, individual or nation-state?

When researchers and creators like Jason Wei speak of “magic” in technology, it’s time to hit the pause button.

The dream of leveraging intelligence has existed for a very long time, but the accelerating advances in AI are turning these aspirations into reality and it is precisely this collision between extraordinary aspiration and the vulnerability of our societies that terrifies me.

It is worth remembering lessons learned from past technological revolutions.

Unforeseen consequences always follow the introduction of new sciences that was supposed to bring only progress.

Industrialization rushed in a new era of productivity and wealth but also changed the planet’s climate and gave rise to the exploitative labor conditions we see all over the world, still today.

The internet democratized access to information and connected the world, but it also enabled disinformation, invasive surveillance, and extreme social division.

AI’s impact will dwarf any precedents, leaving humanity trying to impose meaningful constraints on the forces that are currently being unleashed. The current exponential pace of innovation requires a public and governments that demands transparency and accountability from the institutions that develop, fund, and deploy these powerful systems.

We are either dumbfounded or uninformed, and I quote OpenAI as they are discovering an “unstoppable optimization algorithm” in an “unhackable environment.”

The momentum toward increasingly capable AI is undeniable and the line between a miraculous transformation and a catastrophic disruption will be drawn by our collective wisdom.

It falls upon all of us to ensure that the forces shaping AI are deeply regulated.

AI magic, in the wrong hands, will burn the whole world down.

By 2ndrevolution.org