A note on the species that figured out fire, then nuclear weapons
It is a beautiful thing, in its way, to watch a species discover what it is capable of. We started with rocks. We hit other things with the rocks. We hit, in particular, our enemies. The rocks worked. Encouraged by this success, we developed sharper rocks, then metal versions of the rocks, then larger metal versions, then projectile versions, then explosive versions, then versions that could be delivered from many miles away, then versions that could destroy entire cities, then versions that could destroy entire countries, then versions that could destroy the planet several times over, which is a curious thing to need but which we built anyway, presumably in case the planet had backup planets we did not know about. We have not, in our entire history as a species, ever invented a weapon and then declined to use it. We have, in some cases, declined to use it for a while. We have never, in the strict sense, declined entirely. The pattern is so consistent that historians have come to regard it not as a series of choices but as a kind of natural law. Anything that can be invented will be invented, and anything that has been invented will eventually be deployed, and the deployment will, in nearly every case, be considered, at the time, both reasonable and necessary by the people doing it.
Into this charming pattern we are now introducing artificial general intelligence. The man behind the announcement is Sam Altman, who has the calm and slightly glazed manner of someone who has thought about this a great deal and has arrived at a position of serene confidence that he, personally, is going to be fine. He has said, for the public record, that he is “not sad” about building something smarter than himself. This is presented as a generous concession. The implication is that he, the builder, has reflected on the possibility that his creation might surpass him, and has made his peace with this. The public, observing the announcement, is invited to share his serenity. The public is invited, in particular, not to ask the obvious follow-up question, which is whether Sam being personally not-sad is the relevant criterion for whether the species should proceed, and whether the rest of us, who were not consulted, might have feelings of our own on the matter. The question, somehow, has not made it into the press conference.
Sam is, to be entirely fair, not alone in his confidence. There is a substantial population of intelligent, well-meaning, professionally accomplished people who believe wholeheartedly in what they are building. They believe it will cure diseases. They believe it will end poverty. They believe it will solve climate change. They believe it will, in some attenuated sense, deliver us from the long human predicament of not being smart enough, fast enough, and patient enough to manage our own affairs. The argument is not unreasonable. The argument is, in fact, occasionally moving. The argument has the small structural disadvantage of being made by the people who stand to become the wealthiest individuals in human history if it turns out to be correct, which has, on the historical record, sometimes affected the objectivity of arguments made by people in similar positions.
The case for doing this, as far as we can reconstruct it, is that we have to. The case is not that we want to. The case is not that we have weighed the risks and concluded the risks are acceptable. The case is that we, as a civilization, are physically unable to stop. If we don’t build it, China will build it. If China doesn’t build it, the next country will build it. If no country builds it, some private actor will build it. If no private actor builds it, some lone individual in a basement will build it. The technology is going to be built, the argument runs, because the technology can be built, and therefore our only real choice is who builds it first. We should build it first. We should build it first because we have, as a civilization, the most experience handling powerful technologies, by which is meant the most experience inventing them and immediately using them on each other.
This argument is not entirely wrong. It is also not entirely an argument. It is more of a description, dressed up in the formal clothing of an argument. The description is: we will not stop. The argument is: therefore, we should not stop. The transition from description to argument is performed swiftly, with confident eye contact, and is not generally subjected to careful examination, because subjecting it to careful examination would reveal that the description is the entire substance, and that “we will not stop, therefore we should not stop” is not an argument anyone would accept from a person describing his own behavior in any other context.
If you said to a friend, “I cannot stop drinking, therefore I should not stop drinking,” your friend would, if your friend were a real friend, suggest that you were perhaps confusing two different categories of statement. The first statement is about your capacities. The second statement is about what you should do. Capacities and shoulds are not the same thing. The species, however, has not had this conversation with itself, perhaps because the species lacks a friend in the relevant sense. The species has only itself. The species, as a friend, has, on the historical record, been a poor friend to itself. The species has always told itself, when faced with the question of whether to invent the next thing, that the inventing was not really up to it, that the inventing was an inevitability, that the inventing would happen with or without it. The species has, on inspection, never declined any invention on the grounds that the invention was unwise. The species has only ever declined inventions on the grounds that the invention was, at the moment, technically infeasible.
As soon as feasibility was achieved, the invention proceeded. The species has been, in this sense, the most reliably curious entity in the history of the universe, and the most consistently incapable of self-restraint, and these two qualities have been, on inspection, the same quality.
We have, as a species, decided that we should manufacture, in carefully secured facilities, the most lethal viruses we can engineer. The reason given for this is that we need to understand the viruses in order to defend against them. The reason given is, in some sense, true. The reason given is also, on close inspection, the reason given, which is not the same as the operative reason. The operative reason is that we can engineer the viruses, and therefore we are engineering the viruses.
We have engineered viruses that, if released, could end civilization. We have stored these viruses in facilities. The facilities have, on several documented occasions, leaked. The leaks have, on a few occasions, killed people. We have responded to these incidents with deeper safety protocols, more careful procedures, additional layers of containment. We have not responded by stopping the engineering of the viruses, because the engineering of the viruses is the entire activity, and stopping the activity is not, in any meaningful institutional sense, on the table.
The activity continues. The activity expands. The activity is currently being augmented by AI, which is improving our ability to engineer the viruses faster, more cheaply, and with more sophisticated effect.
We are, as a species, now in the position of having paired our most reliably reckless impulse with our most powerful tool, and the resulting combination has been described in the trade press as a “promising research frontier.”
It is, on inspection, a research frontier. The promising part is more conditional.