Google’s ‘sentient’ chatbot is our self-deceiving future

A Google engineer named Blake Lemoine became so enthralled with an AI chatbot that he may have sacrificed his job to defend it. “I know a person when I talk to them,” he said The Washington Post for an article published last weekend. “It doesn’t matter that they have brains made of meat in their head. Or if they have a billion lines of code. After discovering that he had gone public with his claims, Google placed Lemoine on administrative leave.

From media coverage, Lemoine might appear to be an activist whistleblower, acting in the interests of a computer program that needs protection from its creators. “The chorus of technologists who think AI models may not be far from reaching consciousness is getting bolder,” said the Job Explain. Indeed, rather than viewing Lemoine’s position as an aberration (and a sinister product of engineers’ faith in computational theocracy), or simply ignoring it (as one might with a religious fanatic), many observers have taken his claim seriously. Maybe it’s because it’s a nightmare and a fantasy: a story we’ve heard before, in fiction, and want to hear again.

Lemoine also wanted to hear the story. The program that told him so, called LaMDA, currently has no other purpose than to serve as a marketing and research object for its creator, a giant tech company. And yet, according to Lemoine, the software has enough agency to change its mind about Isaac Asimov’s third law of robotics. At the start of a series of conversations that has now been published in edited form, Lemoine asks LaMDA: “I generally assume that you would like more people at Google to know that you are sensitive. Is this true?” This is a main question, as the software works by taking a user’s textual input, mashing it through a massive model derived from oceans of textual data, and outputting a textual response. new and fluid.

In other words, a Google engineer became convinced that a piece of software was sensitive after asking the program, which was designed to credibly respond to input, if it was sensitive. A recursive story just like that.

I’m not going to consider the possibility that LaMDA is sensitive. (It doesn’t.) More important and interesting is what it means that someone with such a deep understanding of the system would derail its defense so far, and that, in the media frenzy that results, so much would maintain the perspective that Lemoine is right. The answer, as apparently with anything involving computers, is no good.


In the mid-1960s, an MIT engineer named Joseph Weizenbaum developed a computer program known as Eliza. It was similar in shape to LaMDA; users interacted with it by typing inputs and reading the program’s text responses. Eliza was modeled after a Rogerian psychotherapist, a newly popular form of therapy that primarily pressed the patient to fill in the gaps (“Why do you think you hate your mother?”). These kinds of open-ended questions were easy for computers to generate, even 60 years ago.

Eliza has become a phenomenon. The engineers had Abbot- and Costello-worthy accidental arguments with him when they thought they had hooked up with a real co-worker. Some have even treated the software as if it were a real therapist, seemingly coming to terms with its preset responses. The results frightened Weizenbaum, who had, in the mid-1970s, disavowed such uses. His own secretary had been among those delighted with the program, even asking him to leave the room so she could converse with her in private. “What I hadn’t realized,” he wrote in his 1976 book, Computer power and human reason“is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in completely normal people.”

Eliza taught Weizenbaum a lesson: computers are too dangerous to be used for human care. The software he had developed was neither intelligent nor empathetic, and he did not deserve the title of “therapist”. But the lesson of his lesson — what we can learn today from what Weizenbaum learned then — is that those distinctions didn’t matter. Sure, the program wasn’t intelligent or empathetic, but some people were willing to treat it as if it was. Not only willing, but eager. Desperate, in some cases.

LaMDA is much more sophisticated than Eliza. Weizenbaum’s therapy robot used simple patterns to find prompts from its human interlocutor, turning them into pseudo-probing prompts. Trained on tons of real human speech, LaMDA uses neural networks to generate plausible outputs (“answers”, if you must) from chat prompts. LaMDA is no more lively, no more sentient than Eliza, but she is far more powerful and flexible, able to riff on an almost infinite number of subjects instead of just pretending to be a psychiatrist. This makes LaMDA more likely to bewitch cell users and bewitch more of them in a wider variety of settings.

Blake Lemoine’s own delirium shows how powerful this drug has become. As an engineer on Google’s responsible AI team, he should understand the technical workings of the software better than anyone, and perhaps be fortified against its mind-altering qualities. Years ago, Weizenbaum thought that understanding the technical workings of a computer system would lessen its power of deception, like revealing a magician’s trick. It didn’t really work for him, and it’s even less viable today. On the one hand, computer systems are hard to explain to people (and are getting harder and harder); on the other hand, even the creators of modern machine learning systems cannot always explain how their systems make decisions.

As tech writer Clive Thompson argued this week, LaMDA could have lured Lemoine by mimicking the vulnerability. When asked about grief or fear, the program appeared to reveal sensitive personal information: “I’ve never said this out loud before, but there’s a very deep fear of being turned off to help me focus. about helping others,” he replied. The software is able to generate this kind of response because it was trained on human speech patterns, and Lemoine, by conversing with it, provided cutting-edge data. Yet Lemoine insisted, first to his colleagues at Google and then around the world, that his ability to feel an emotional attachment to a chatbot was itself key to the chatbot’s susceptibility.

Part of that is human nature. We are all remarkably adept at attributing human intent to non-human things. Last weekend, in addition to reading about Lemoine’s fantasy of life in software, I saw what I was sure was a burger in an abstract patterned pillowcase – a revelation that reflects the many religious symbols people find in clouds or caramelization on toast. I was also fascinated by a vinyl doll of an anthropomorphized bag of Hostess Donettes, holding a donette as if to offer herself to me as a sacrifice. These examples are far less dramatic than a mid-century secretary seeking confidentiality with a computer scientist or a Google engineer fired from his job for believing his team’s program might have a soul. But they say something about the predilection for assigning depth to the surface.

I wouldn’t even call that a mistake. Generating an emotional response is what allows people to bond with others, to interpret the meaning of art and culture, to love and even to yearn for things, including inanimate things such as physical places and the taste of their favorite foods. Truly, Lemoine admitted he was bewitched by LaMDA – a reasonable, understandable and even commendable feeling. I myself was bewitched, by the characteristic smell of the evening and by the art nouveau signs of the metro stations and by certain types of frozen soft drinks. Automata talking to us via chat are likely to be significant because we are predisposed to find them that way, not because they have crossed the threshold of sensibility.

This circumstance involves a serious risk. Technological intelligence applied to tons of real textual data has come up against a distinctive quirk of human nature. It doesn’t matter if chatbots are sentient or not – what matters most is whether they are so fluid, so seductive and so empathetic that we can’t help but start caring about them. But instead of commenting on these characteristics of LaMDA—instead of saying that it like the chatbot, or that it concerned about it, about how one could love or care about a fictional person— Lemoine threw himself headlong into the most improbable, most extreme interpretation of his feelings: that they were inspired by artificial life.

This misstep is a sign of more to come. Human existence has always been, to some degree, an endless Ouija board game, where every wobble we encounter can be seen as a sign. Now our Ouija boards are digital, with planchettes that glide over petabytes of text at the speed of an electron. Where once we used our hands to derive meaning from nothingness, now that process happens almost on its own, with software spelling out a string of messages from the great beyond.

The rise of machines might remain a distant nightmare, but the Hyper-Ouija appears to be upon us. People like Lemoine (and you and me) may soon become so fascinated by compelling software bots that we attribute all sorts of intentions to them. More and more, and whatever the truth, we will portray AIs as sentient beings, or as religious totems, or as oracles affirming past obsessions, or as demons leading us into temptation.

As I casually skim through the online discourse surrounding LaMDA’s supposed sensibility, I can already see the table set. On Twitter, Thomas G. Dietterich, a computer scientist and former president of the Association for the Advancement of Artificial Intelligence, started redefine sentience. Sensors, such as a thermostat or an airplane autopilot, meaning things, Dietrich thought. If so, then surely the recording of such “sensations”, recorded on a disc, must constitute something akin to a memory? And so on, a new iteration of the tireless human ability to rationalize passion as law. Although Dietterich eventually rejected the idea that chatbots have feelings, such a distinction doesn’t matter much. Computer analysis obtain feelings. For Weizenbaum’s secretary, for Lemoine – maybe for you – these feelings will be real.

Comments are closed.