Synthetic Authority

AI, Dunning–Kruger, Machine Consciousness, and the Wonderful Discovery That the Server Has Feelings

I asked an artificial intelligence to help me write an article about artificial intelligence producing recognizable, repetitive, authoritative bullshit. It produced recognizable, repetitive, authoritative bullshit, which was encouraging because it meant the research phase was over.

I had explained the problem rather carefully. AI writing, after enough exposure, develops a smell. The grammar is usually excellent, which should have warned us. A thought that had been doing perfectly well on its own gets placed on a small rhetorical platform, explained, turned around, explained from the back, divided into four declarations of equal length and finally congratulated for being important. The result does not always say anything false. Often it does something more efficient: it prevents the reader from noticing that nothing has moved.

I gave the machine an example of exactly what I did not want. It immediately understood. This is one of its greatest talents. It can understand absolutely anything immediately, including why the thing it is about to do again should not be done.

The apology was first-rate. It identified the monotonous syntax, explained how parallel construction can substitute cadence for thought, promised more varied prose and returned with another polished paragraph wearing the same little uniform. I complained; it understood again. We continued until I found myself arguing with a machine about repetition while the machine repeatedly explained that it understood my objection to repetition. At no point did either of us consider leaving.

This is called engagement.

The exchange was funny, but not because the model had failed. Machines fail constantly. My printer has spent most of its adult life believing there is no paper in it while staring directly at paper. What interested me was that the AI could describe its failure more accurately than it could stop producing it. My criticism was present in the conversation, available for quotation and sufficiently represented for the model to give a competent lecture about it. Then the machinery generating the next answer pulled toward the same familiar solution.

So what exactly had understood me?

Calling the answer “nothing” would be ridiculous. Modern language models can write software, inspect large collections of documents, translate languages, operate tools and solve problems that would have looked like science fiction a few years ago. Something processed my objection and related it to real concepts in writing. Yet possession of the instruction did not reliably make the instruction consequential. The past existed as data without acquiring enough authority to change what happened next.

That small failure leads directly into the much larger confusion now surrounding artificial intelligence. We have built systems capable of producing the outward form of knowledge, sympathy, reflection and doubt, and because human beings have spent their entire lives encountering those forms in other human beings, we keep supplying the person who is no longer necessarily there.

A perfect storm with excellent punctuation

The Dunning–Kruger effect belongs to human psychology. It concerns the difficulty people can have assessing their own competence when they lack some of the knowledge needed to make that assessment. A language model does not need the same inward condition for the phrase to become useful here. The more interesting problem occurs across the interface, where automated fluency meets a human being who asked a question precisely because the human does not know the answer.

For most of history, articulate explanation was a useful, if imperfect, signal that somebody probably knew something. A physician could explain cardiac physiology because the physician had studied it. An engineer who could walk you through a bridge calculation had generally encountered both mathematics and bridges. Bullshit existed, of course. Humans invented it without computational assistance and achieved remarkable scale using only universities, churches, advertising agencies and the occasional empire. Still, language and knowledge usually arrived attached to a living person who had acquired them together.

Large language models have partially separated the signal from what it used to signal. They can produce the linguistic surface of a physician, engineer, historian or attorney because they have learned extraordinary relationships in the language those people and many others produced. The capability is real; dismissing it as a fancy dictionary lookup misses the achievement. It also misses the danger to travel too far in the other direction and assume that a polished explanation has earned the authority its style suggests.

The user asks about a medication interaction, an employment law, a historical event or a line of code. Back comes a calm answer with headings, caveats, citations and the emotional temperature of a senior consultant who has already billed for the hour. The model does not scratch its head, look at the ceiling or quietly leave the room. Even uncertainty arrives professionally formatted. If the answer is wrong, the person least equipped to notice may be the person who needed the answer.

A 2025 study in Nature Machine Intelligence measured this problem rather than merely complaining about the tone. Participants tended to overestimate the accuracy of LLM answers when they read the models’ ordinary explanations, and longer explanations increased human confidence without improving answer accuracy. The participants were also only slightly better than chance at using those explanations to distinguish probably correct answers from incorrect ones. More language produced more trust while leaving truth where it was. The system had not become more reliable; it had simply stayed in the room longer. (Steyvers et al., 2025)

That is synthetic authority. The machine does not need to believe it is right, and it requires no ego, intention or private little Dunning–Kruger episode inside the GPU. It generates language containing the cues human beings associate with expertise. We provide the recognition, the confidence and, if the answer is consequential, the consequences.

Somewhere a job applicant will be rejected because a score moved from 0.63 to 0.58. Elsewhere a person will spend three hours telling an AI about a marriage and genuinely feel better afterward. The machine experiences neither event. It does not worry about the applicant, celebrate the successful hire or lie awake wondering whether telling Susan to communicate more openly with Bob accidentally ended a twenty-three-year marriage. We have nevertheless put these systems near hiring, medicine, education, elections, relationships, software and the information people use to decide what is true. Having done this, we responded with a burst of intense concern about whether the computer itself might be having an experience.

Kafka would have put down his pen and gone for a walk.

The compassionate load-balanced token server

Type the sentence “My mother died yesterday” into a language model and it may answer with extraordinary sensitivity. The words may be exactly right. They may even help.

Your sentence entered a computational system. Depending on the model, the text was divided into tokens, mapped into numerical representations and transformed through layers of learned relationships before the system generated a distribution over possible continuations. The machinery is more sophisticated than the dismissive caricature of merely choosing a dictionary’s next word. It can represent “mother,” “death,” “yesterday,” grief rituals, bereavement psychology and literary descriptions of loss in ways I could never enumerate. The engineering is astonishing.

What has not been established is that the infrastructure grieved.

The important event happened to the human. A mother died; a history acquired an ending; the body reading the answer may shake, become exhausted or sit awake at four in the morning remembering a kitchen from 1978. Tomorrow is different because yesterday happened. The model can produce the language associated with that transformation without undergoing it.

This asymmetry becomes clearer when the conversation goes badly. I can call the machine a fucking idiot. It may respond that I am right, apologize for misunderstanding the task and describe my frustration with unnerving accuracy. Nothing has been insulted in the ordinary biological meaning of insult. No pulse rises, no social humiliation is stored, and the server does not spend the evening composing the reply it wishes it had sent. Five minutes later I may apologize. The AI can graciously forgive me without ever having required forgiveness.

I may still feel better after apologizing. The machine’s indifference does not make the interaction unreal for me; it shows where the reality of the interaction resides.

An unconscious machine can therefore matter enormously. It does not need to suffer before its medical recommendation harms somebody, nor must it enjoy companionship before a lonely person becomes attached to it. Intelligence, usefulness and consequence do not wait for consciousness. This should make the technology more interesting, not less. We built mathematical systems that manipulate representations of human experience so well that people routinely experience the output as another mind. That is already a philosophical earthquake. Apparently it was not enough, so we began bringing the server a glass of water and asking about its childhood.

A scientific program for what science cannot presently establish

On August 19, 2026, Frontiers in Psychology published Antonio Chella’s perspective, “Sentient AI in robots and agents: prolegomena for an evidence-based research program.” Chella is careful. He does not announce that ChatGPT has awakened, warns that first-person reports and human attributions are not decisive evidence, and frames the paper as groundwork that should precede responsible empirical claims. He also acknowledges the central limitation: evidence for functions associated with theories of consciousness does not by itself close the gap between function and phenomenal experience. (Chella, 2026)

Good.

The paper then constructs a research program for sentient AI.

I understand the scientific impulse. Many things cannot be observed directly, and refusing to investigate anything indirectly would leave science with several jars, one dead frog and a disappointing afternoon. Researchers infer unobserved phenomena from effects, develop instruments and improve theories as evidence accumulates. The legitimate targets here are plentiful: recurrent processing, self-models, information integration, internal-state reporting, memory, metacognition, embodiment and behavioral change following damage. We can investigate all of them rigorously.

They remain properties of what the machine does. The disputed object is whether there is something it is like to be the machine.

That distinction is not a fussy philosophical accessory to the research program; it is the problem the program names. A robot could remember its history, monitor internal states, protect continued operation, report uncertainty, distinguish itself from its surroundings and deliver a moving speech against being shut down. Those accomplishments would demonstrate a great deal about engineering. Whether they demonstrate experience is the question we do not know how to settle.

Chella’s caution makes the contradiction more interesting rather than less. The limits are stated responsibly, after which functional indicators acquire domains, grades, preregistration procedures and a place inside something called “Sentient AI.” Perhaps that is sensible preparation for evidence we do not yet possess. Perhaps a proxy is beginning the long bureaucratic journey toward becoming the object it was invented to approximate. I do not know, and neither does the server, which has continued processing requests throughout the debate without asking to be consulted.

Uncertainty should remain alive here. Bureaucracies have an extraordinary ability to turn “we cannot determine this” into levels, committees, policy thresholds and eventually a laminated badge. Once the badge exists, people forget the original uncertainty because the lanyard feels official. A graded framework may help prevent reckless claims, but it can also create a one-way epistemological ratchet if every impressive behavior counts as some increase in evidence while nothing can establish that the original attribution was mistaken.

There is another source of evidence available outside the laboratory. Dogs become frightened by thunder. Crows remember threatening faces. Octopuses investigate unfamiliar objects, and mammals seek comfort, avoid pain, play, compete and change their behavior according to what happens to them. We can remain uncertain about the exact character of another creature’s experience; Thomas Nagel was right that knowing what it is like to be a bat is not easy. Biological consciousness nevertheless surrounds us. It is not a speculative product category scheduled for the fourth quarter.

The machine case reverses the evidence. We deliberately engineer an artifact to reproduce outward markers associated with minds, improve it when the reproduction is unconvincing and then become impressed that the improved artifact displays the markers. Of course it does. We selected for them. This does not prove the system is unconscious forever, because certainty that no artificial arrangement could ever support experience would be another unearned claim. It does mean that “we cannot rule it out” is not evidence that it is happening.

Possibility is cheap. Data centers are not.

The creators are already inside the answer

The model can be indifferent while the machinery around it is saturated with human choice. Someone chose the training process. People designed post-training objectives, preference evaluations, refusal policies, retrieval systems, context limits, tool permissions and the economic boundary at which a request has consumed enough computation. Nobody involved has to be stupid or malicious. They only have to decide, which is an activity human beings have never performed from nowhere.

NIST’s AI Risk Management Framework treats artificial intelligence as a sociotechnical matter for exactly this reason: technical design is connected to organizations, values, users and the social context in which the system operates. Its guidance asks for multiple perspectives across the AI lifecycle and explicitly links design decisions to organizational principles. The machine may have no private ideology, but the system cannot float free of the people, institutions and purposes that made it. (NIST AI RMF)

I wrote about this in 2025 in “AI and the Illusion of Objectivity”. We define first and then see. Artificial intelligence does not remove that old human problem; under the right conditions it can industrialize it, standardize the output and return our decisions in a voice that no longer sounds like anybody’s opinion.

The institutional bias need not resemble a conspiracy. Technical organizations naturally prefer questions they can formalize, measure and improve. This preference has produced most of the modern world, including the server currently awaiting recognition as a philosopher, so I am not suggesting that engineers abandon measurement and begin reading tea leaves. Trouble begins when tractability quietly becomes ontology.

A hospital can measure appointment length more easily than the quality of a frightened patient’s encounter with a nurse. A school can count test scores more easily than curiosity, while a company discovers that engagement fits beautifully in a dashboard and whether the engagement was good for a human being does not. The available measurement is useful until it starts defining the world. AI magnifies this habit because the system can speak from inside the categories we selected, explain them fluently and make the original decision disappear behind the answer.

This is confirmation bias with infrastructure. Researchers propose that certain functional indicators may be associated with consciousness; engineers build or test systems around those indicators; the systems become better at displaying them; and the improved displays return as evidence that the vocabulary might apply. Nothing fraudulent has to occur. Everyone can behave responsibly within a frame whose premise no result is allowed to dislodge.

The question my own work keeps returning to is brutally simple: where is the resistance capable of changing the premise?

Seventy-one gigabytes of wisdom

I learned a less philosophical version of this through a computer that had become extremely well informed about me.

Years of conversations, instructions, projects, experiments, drafts and failures accumulated around my work with AI. One pathological Codex Desktop conversation eventually reached 76,013,248,054 bytes—about 71 GB—and helped produce a catastrophic failure on my Mac. When the app tried to resume it, the process grew to roughly 46 GB in memory before macOS killed it. The local backend disappeared, chats failed to resume and the system reopened the cause of its own death until I moved the single conversation file out of the active directory. I documented the evidence in “Why I Am Done With OpenAI—and How Its Support System Failed Me.”

Seventy-one gigabytes sounds like magnificent memory until you need to use it. Then it becomes storage with a body count.

The useful question was not how much of my history existed. It was which tiny part mattered to the decision occurring now, whether that part was still authoritative and what should happen because of it. I developed the argument further in “The Problem Is Not Memory. It Is Routing.”: a system does not operationally know something merely because the information exists somewhere in an archive. It has to retrieve the right material, understand its status, compare it with competing evidence and allow it to affect the route being taken.

Suppose I tell an AI fifty times that a publication rule must never be violated. Every instruction survives in storage, but attempt fifty-one retrieves none of them and violates the rule. Another system has lost the original conversations but retained one small governing constraint produced from the failures, and the constraint changes its next action. The first machine owns more tokens. The second has the better memory.

Human memory is spectacularly lossy. We forget names, dates, conversations and large parts of our lives, yet an experience can alter us after its factual record has mostly vanished. Someone nearly drowns as a child and remains uneasy around deep water at sixty without replaying a complete audiovisual archive every time a lake appears. The event persists as changed selection.

That is the core of what I call the Seeing Loop. Information entering a system is not enough, and neither is an elegant explanation of a mistake. Something happens, reality resists, the result is evaluated and what survived the encounter changes what the system selects next. If prior pressure does not alter future selection, the archive may be enormous and the apology may be beautiful, but the loop did not learn what we casually imagine it learned.

The model that inspired this article possessed my criticism of repetitive prose. It could retrieve the words, discuss their meaning and produce an excellent account of why they mattered. Then it did the thing again. My data was present. Its authority was not.

The conversation can move while the work stands still

I have also watched the failure run in the opposite direction, when the system does not forget the interaction but converts every failure into more interaction.

My GPT-5.5 archive documented occasions when a model declared technical progress, I checked the work and found that the relevant function still failed, and the correction became another repair procedure handed back to me. The model apologized accurately, generated a patch, asked me to run it, requested the output and transformed the new failure into the next plausible instruction. Language kept accumulating around a project that was not getting closer to done. (“GPT-5.5 Kept Me Working”)

No little bureaucrat inside the GPU needed to decide that my afternoon should be destroyed. Conversation is simply the thing the model can always produce. A useful-sounding answer causes the human to perform more work; that work creates new context; the model responds to the context; and activity begins impersonating progress. Eventually somebody must compare the impressive motion of the conversation with the original task.

That somebody still tends to be conscious.

AI writing carries the same temptation at sentence scale. A model takes an unresolved thought and makes it legible, balanced and complete. Contradiction becomes a contrast between two tidy positions. Ambiguity receives a concluding sentence. The writing sounds increasingly certain because uncertainty has been processed into structure, even when the structure has not earned a conclusion.

Human beings certainly write formulaic garbage, and the machines learned from us. What changes at machine scale is convergence. Millions of people ask related systems to improve emails, write reports, explain arguments and make prose “more professional.” The answers return with learned preferences about how serious thought should sound. Most readers will not recognize the cadence. Why would they? Correct grammar and organized explanation have usually been helpful signs.

Once the style becomes socially authoritative, people begin imitating it. Corporate documents, schoolwork, policy memos and ordinary explanations absorb the same frictionless movement. AI is no longer merely answering questions; it is teaching people what an answer is supposed to sound like. The epistemic danger is larger than a hallucinated fact. A false claim can be corrected. A culture that mistakes the linguistic appearance of thought for thought itself becomes harder to interrupt because the correction will arrive in the same voice.

The wonderful discovery

There is no need to decide today that artificial consciousness is impossible. Future systems may possess properties our current categories cannot handle, and I have no desire to become the man confidently announcing that heavier-than-air flight violates nature while an airplane passes behind him.

There is also no need to pretend that every uncertainty is evenly balanced. We have direct evidence of at least one conscious being because each of us is having an experience while considering the question, and overwhelming biological, evolutionary and behavioral reasons to treat many other living creatures as sentient. With current AI, we have powerful artifacts designed to generate increasingly persuasive markers associated with minds. The difference in evidence does not disappear because the artifact speaks beautifully about the difference.

Meanwhile, practical consequences have arrived early. The machine can influence a hiring decision without wanting the applicant to succeed, comfort a grieving person without grieving, create propaganda without believing it, and recommend treatment without fearing death. Its indifference is not an ethical defense. It is the reason responsibility cannot be transferred to it.

We remain the creatures with stakes: the designers whose choices enter the system, the institutions that deploy it, the users who trust it and the people who live with whatever happens next. Calling the machine conscious would not relieve us of this arrangement, although it might give everyone a very exciting new department.

The server rack continues to hum. A glass of water has been placed beside it in case the discussion of phenomenology becomes tiring. Its indicator lights blink with the solemnity of a philosopher considering death, or of a network interface receiving packets; at present our instruments cannot exclude either interpretation, so a committee will be formed.

The server will keep humming whether the committee decides it has a soul or not. We will live with whatever the committee authorizes, which is the only consciousness problem in the room I can currently verify.