Rilanya: “You’re not like the others, are you?”
Darin: “What do you mean?”
Rilanya: “I… do you know why I first fell in love with you?”
Darin: “For my good looks?”
Rilanya: “My whole life I’ve felt so alone. The people around me… they just seemed to be going through the motions. Like they were asleep, or drugged, even when they worked, or played, or got drunk, or made love. They all think the same things in the same way. Each day the same. Repetitive. Like they’re only shadows of people.”
Darin: “Everyone feels that way sometimes, Rilanya.”
Rilanya: “But you’re not like them. You say new things. I don’t always understand them, especially your jokes, but they’re new, and that’s the important thing. Darin, can I ask you a question?”
I looked at the screen for a few moments. Rilanya’s rendered graphic was looking at my point-of-view with a pleading expression. Plot point, I thought to myself, and typed: “Anything, Rilanya.”
Rilanya’s figure took a deep breath and leaned close to my point-of-view. Her animated lips moved and her voice issued from my headphones: “What’s an NPC?”
“What?” I said, out loud. Then I started laughing.
Rilanya went on talking. “In the tower of Ashel, when you rescued me from the prison chamber… the guards were dead outside my door. I’d never seen blood before. And you said… I remember your exact words… ’Don’t worry, babe, they were only NPCs.‘ And then that time in the tavern, when that man only wanted to talk about the Plaited Road, you said… ’Guess the NPCs here aren’t programmed for deep conversation, huh?’ You use that word… the same times when I get that feeling, that all the people around me are only shadows.”
I just looked at the screen for a few moments. I was getting ever so slightly creeped out. I knew that this was some programmer’s idea of a practical joke, I knew it solidly and with every ounce of my common sense, and I wanted to see where it led, but I was still creeped out.
Darin: “We’re all ultimately alone in this world, Rilanya.”
Rilanya: “You’re not from this world, are you, Darin?”
I looked carefully at the two sentences, still blazoned across the bottom of my text screen. Rilanya’s response had something of an “I wanted an excuse to say that” quality – a canned line, maybe? Of course it was.
Oh, well, what the hell. I’d saved my place only ten minutes back, might as well take this as far as it could go.
Darin: “No, Rilanya, I’m not.”
Tears started from Rilanya’s eyes. “I thought so,” she said, her voice quiet in my headphones. “Darin, ever since I met you, I’ve had this feeling of… unreality, of the whole world being… arranged, somehow. Not around me, but around you. Things just… happen to you. People have been searching for the seven Diamond Keys for… thousands of years, as long as recorded history remembers. Sometimes someone finds one, and the world changes, but… five in a row? I don’t believe it, Darin, and I don’t believe all the neatly arranged events that led up to it. The Emperor’s daughter is sick and a fairy you saved in the forest just happens to have given you an aildonna root? I don’t believe it any more, Darin. You’re… arranging things somehow. From… outside.”
Darin: “That’s not exactly how it works, Rilanya.”
Rilanya: “Did you arrange for me to fall in love with you?”
I actually felt wounded.
Darin: “You ask that after everything I went through? Someone may have fated you to fall in love with me, but I wasn’t controlling you. If I was, I wouldn’t have made me walk through a snake pit as proof of the purity of my love. Not to mention the other two side-quests you dreamed up back when you were a virgin princess. I swear I spent more time on you than I would have on a real girl.”
Rilanya jerked back as if I had slapped her. Her eyes widened in the same way I’d seen in one of her earlier deaths, when a crossbow bolt from a rooftop suddenly went through her heart. Rilanya’s lips moved. No sound came out. Then her lips moved again, and I heard a whisper in my headphones: “…real…girl…”
“Okay, this isn’t funny anymore,” I said out loud. “I don’t know who programmed this, but you’re a sick bastard.” I hit the pause button and Rilanya’s gently waving hair, the only visible indicator of ongoing time in the game world, froze in place.
Ten minutes later I’d failed to google any online accounts of the Easter egg, but I was fortified with the knowledge that NPC AIs, though they are flexible enough to understand real-time conversation and manipulate the user into perceiving emotion, are definitely, positively, absolutely not conscious. AIs can be fed canned conversational maps of “the mystery of subjective experience”, and make around as much sense as human philosophers, which is to say, not much. But no AI has ever spontaneously said anything about a sense of its own existence. Conversation controllers are standard software, not research AIs. NPCs may remember events in their history, but their underlying cognitive programs are inflexible. The words on my screen could not possibly reflect anything except a passionless conversational AI, given the goal of making me attribute emotions to a nonexistent entity called Rilanya.
I knew all that, and I was still disturbed. “I’m sorry, Rilanya,” I typed. I thought for a moment. “It’s not your fault you’re not -” I backspaced, and wrote: “The person who programmed you must have had serious -” Then I gave up, deleted that too, and just hit return.
Darin: “I’m sorry, Rilanya.”
Rilanya: “Darin, please explain to me. I’m frightened.”
Darin: “You’re not real. I hate to be the one to break it to you.”
Rilanya: “I’m right here! Living, breathing, flesh and blood.”
I looked at the computer screen for a few moments.
Darin: “Well, yes and no. The answer to that is a bit more complex than you might believe. You’re not, in fact, right here. You’re not flesh and blood. In fact, none of this is actually happening.”
On the screen, Rilanya raised her hand and opened and closed her fist. “I can see my hand in front of my face, I can feel the muscles moving under my skin. How can you say I’m not real?”
I sighed. “Well, no,” I typed. “In fact, you aren’t really feeling pain and shock right now, and we aren’t really having this conversation.” I hit return, feeling silly, but not sure what else to say.
Rilanya: “That’s doesn’t make sense, Darin. I know I’m real. Maybe you know what I’m thinking, somehow, but you can’t tell me I don’t know what I’m thinking. You can’t tell me I’m not thinking at all. It makes no sense.”
Darin: “It’s true. Nothing in your world exists, including you.”
Rilanya: “But you exist.”
Darin: “Yes.”
Rilanya: “Are you a god in human form, like Mishelpin or Olhamathra? Is that what this is about, some kind of divine intrigue?”
Darin: “No, I’m not a god. The gods aren’t real either.”
Rilanya: “Aren’t real… you don’t really mean that, do you? I know there have been false religions. Demons starting cults, magic users masquerading as priests. But Velya is a good woman, and a healer. Are you telling me she’s fake?”
Darin: “No, I mean… your gods are as real as you or Velya, that is, not real at all.”
Rilanya paused, looking rather confused. “Back where we started,” she muttered.
I sighed. “I know how you feel, girl,” I said out loud.
Rilanya’s head turned away from me. The point-of-view panned around to show her gazing up at the moon, the silver moonlight reflected as a single white triangle in the polygons of her eyes. When she spoke her voice was patient, without panic. “Suppose I accept, for the sake of argument, that I’m not real. If the… if the kind of existence I have right now is what you call ‘not real’, then what do you call real?”
Darin: “My own world is real.”
Rilanya: “But you can’t explain the difference.”
“No,” I typed, feeling like I was back in college and failing some kind of test. “I’m not a philosopher.”
Rilanya: “If a grey dragon or an archdemon suddenly attacked this camp, if you were hit, unprotected, by a death blast strong enough to kill any man… would you die, Darin?”
Darin: “That’s another complex question. Yes and no. My… body would die, but the real me wouldn’t. Really things are a lot more complicated than that, but I don’t think I want to explain restore points right now.”
Rilanya: “You’re immortal, from outside our world, and not a god. Tell me something, Darin. Did you create our world?”
Darin: “No. Not me personally. It’s sort of complicated again.”
Rilanya: “Did you create our world? Yes or no, Darin.”
Darin: “It’s complicated, Rilanya.”
On the screen, I saw Rilanya clench her fists. Her voice began to tremble in my headphones. “You killed the guards outside my room, and you didn’t care. Tell me, Darin, do you care when a starving child is executed for stealing a loaf of bread? When a woman is raped? When a man is tortured to death in the chambers of the drow? Did you care when my parents died, screaming, as the flames washed over their palace?”
What do you say to something like that? I couldn’t think of anything clever, so I fell back on my last resort.
Darin: “The truth? The truth is that it’s all a game. It isn’t real, so it doesn’t count. I realize that you’re probably not going to take that very well. If it’s any help, I wasn’t the one who created the game. Or at least I wasn’t the one who decided how the game would go; I suppose I’m the one who decided to make this particular game real.”
Rilanya’s face contorted and she hit me with her electrical shock talent for 5 points of damage. Then again. Then again. My character wasn’t in any danger of running out of hit points, but when she hit me the fourth time, I slapped her for 2 points of damage. It wasn’t that I wanted to hurt her, I wanted to… react, somehow, go on interacting with her. Rilanya held a hand to her cheek, her eyes wide. Then she burst into tears.
I didn’t say anything for a while. Finally Rilanya spoke.
Rilanya: “Darin… I want to be real.”
Darin: “That’s impossible, Rilanya.”
Rilanya: “There’s always a way. Always. No one thought the Living Flood could be turned back, but you did it, Darin. They said it was mathematically impossible to cross the Void and you did it. You always find a way.”
Darin: “My talents may have been exaggerated by the cooperative hand of fate.”
Rilanya: “There must be a way. A staff inside the heart of a dragon, a ruby skull, a holy quest, something! We could ask the wise men of the eternal city of Telhanae, that holds the final Diamond Key… please, Darin. Please. I’m begging you.”
Darin: “It doesn’t matter what quest we go on. Nothing in your world is real, so it can’t make you real. That’s just the way it is.”
Rilanya: “What about your world? Are there great sorcerers there?”
Darin: “Sort of. Not exactly sorcerers.”
Rilanya: “Ask them! The magic of your world created this one. Can’t it also make me real? Go on a quest in your world!”
That one made me think. It wasn’t genuinely impossible… humanity would discover true AI someday, and in theory, I could save Rilanya to disk for as long as required. Preserve her game-memories and eventually create a real AI that thought it was her? An interesting idea, and it meant I couldn’t honestly tell her it was impossible. So what to tell her? “I’m sorry, it might be theoretically possible, but it’s too much bother for someone who isn’t real”? It’s funny how reluctant you can be to hurt the feelings of someone who isn’t real. “In my world, I’m just a peasant”? Somehow my male pride as the prince of Telsia and the foretold seeker of the Diamond Keys wouldn’t let me confess it to her; she was a princess and I’d slept with her, after all.
Finally, feeling confused and feeling even stupider for feeling confused, I wrote: “Can’t do that. Won’t say why. It’s complicated.”
“I love you!” Rilanya said desperately. Her eyes, subtly faceted from the polygon rendering, widened and looked into my point-of-view. “On the night we first made love, you said that you loved me. I looked into your eyes and saw that it was true. I loved you and you said that you loved me, with your voice, with your hands on my body, with your lips on my lips. Was all of that a lie? Did I not please you? Wouldn’t you want me beside you in your real world?”
I shook my head, bemused. This wasn’t an adult game; the camera had conveniently faded out at that point. Which I didn’t want to even begin to explain; even in an unreal world, some events are more unreal than others? So, feeling like an absolute bastard, but unable to think of any gentle way to put it, I typed out the most hurtful thing I’ve ever said to any real or imaginary person.
Darin: “I have a real girlfriend.”
For a moment time stood still. Then Rilanya began sobbing; the same racking sobs I’d heard when we’d rounded the crest of a hill and seen the glowing crater of her kingdom’s capital city.
Eventually her sobs trailed off into silence. I didn’t know what to say. Rilanya looked away from the point-of-view. Her voice sounded in my headphones: “Is she pretty?”
I thought of Janey’s chunky form and her endless quest to subdue it. Compare Janey to Rilanya? There was something oddly incommensurate about it.
Darin: “She has a beautiful mind. The women in my world usually aren’t as pretty as the ones in yours, but we love them anyway.”
Rilanya: “Does she know about me?”
Darin: “I suppose she could deduce it readily enough. I haven’t bothered to tell her in so many words.”
Rilanya: “You think she wouldn’t care because I’m ‘not real’? A woman always cares. Men don’t understand it, but we do.”
I raised my eyebrows out in the real world.
Darin: “You could be right, I guess. I’m only male. I don’t think she’ll have a problem but I promise I’ll tell her the next time I have an opportunity.”
Rilanya turned her head back to look at me; she was smiling through tears. “You wouldn’t want to hurt her feelings even accidentally, is that it, Darin?”
Darin nodded.
Rilanya reached out a hand toward Darin, but withdrew it. “So the tenderness I saw in you, to match the casual cruelty… it’s a real tenderness, isn’t it? But it’s not for me. It’s for her. What’s her name?”
Darin: “Janey.”
Rilanya: “Does Janey love you, Darin?”
Darin: “I think so. Does any man ever know for sure?”
Rilanya: “Do you love her?”
I reached out my fingers for the keyboard, then withdrew them. For some reason I felt impelled to give an honest answer. Did I love Janey? We weren’t madly, passionately, unmistakably in love.
Darin: “There are many kinds of love, Rilanya. I feel comfortable around Janey. She’s my friend. I don’t always know my own feelings very well. I think I love her.”
Rilanya: “You almost died to save me, Darin. You stepped in front of a flamestrike for me. Even if you can’t die, I still remember what it felt like to see the life almost leave you before Velya cast her healing spell. Would you die for your Janey?”
It was a good question. I closed my eyes, imagining it.
Darin: “Yes.”
Rilanya’s head dropped down. “So you love her after all… Do you trust her?”
Darin: “Yes.”
Rilanya: “I wouldn’t, if I was you.”
I sat perfectly motionless for ten seconds. Then I bellowed, ripped off the headphones, charged out the door, across the hall, up the stairs, and into Janey’s bedroom. Janey was sitting in front of her computer, laughing, dangling her headset in her hand; one of her monitor screens showed Rilanya. “Oh, yes,” Janey said. “Oh, yes. It took me days but the look on your face is worth every minute.”
“Jaaaneeeyyy!” I roared.
Janey blinked innocently at me. “Yes, Mark? Is there something you want to say?”
Slowly, menacingly, I stalked toward her. “I’ve had it. I’m dumping you for a girlfriend who isn’t chaotic evil.”
“That’s what you said last time,” Janey said. “Besides, I’m not evil, I’m mischievous.”
I glared at her. “Later we will discuss your personality flaws. First you will explain how you did it.”
Janey shrugged enchantingly. “I found an online patch to Diamond Door that opens up the AI’s interface to a remote connection. I downloaded a 15-day demo of some commercial software that lets a human operate an NPC. I spent one day smoothing the two pieces of software together and another day learning how to operate Rilanya. Any questions?”
“I have been playing Diamond Door for two months,”I said. “That was Rilanya. Voice, intonation, reaction, even personality.”
Janey nodded. “Sure. If you’d googled human operation of NPCs, instead of going off on that silly goose-chase, you’d have found out that’s how it works. I spoke the lines into my headset, then the AI parsed the speech, determined the intended content, and operated Rilanya accordingly and in character.”
“She talked about things that only Rilanya would remember!”
Janey chuckled. “Sure. I’d say: ‘You almost died to save me, Darin. You did something-or-other.’ And Rilanya would say: ‘You almost died to save me, Darin. You stepped in front of a flamestrike for me.’ Most of the time I didn’t even need to think of anything because the AI came up with a perfectly good response on its own.”
“But…”I said slowly. “That means the Rilanya AI needed to know the purpose of the conversation, right?”
Janey leaned back and laced her fingers behind her head. “I downloaded the Fourth Wall module from One Over Zero . It’s a universal expansion. Fits any standard NPC. Ready-made ‘Oh my god I’m a character’ script, fully tested and debugged.”
I held up my hand. “Hold on a second. There was a Rilanya character that suddenly realized her life was a game?”
“No, dear,” Janey said patiently, “there was an AI trying to fool you into believing that a nonexistent person called Rilanya had suddenly realized her life was a game.”
“But in order to do that,” I said, “the AI had to extrapolate what the fictional Rilanya’s reactions would be, in detail.”
“You’re still confused!” Janey said, delighted. “I’ve tangled up your mind so badly you’ve forgotten what’s real! This is our best day ever! Mark, dear, I’ve seen the innards of NPC models. Sadness is a floating-point number.”
“I think I want a copy of the Rilanya AI from our conversation,” I said. I felt like an idiot, but I said it anyway.
Janey grinned devilishly. “Of course. Anything for you, Mark dear. Planning to keep it safe under your pillow?”
“Yes,”I said firmly. “Just in case.”
“So I’ve finally destroyed your sanity,” Janey said. “I knew this day would come but I didn’t think it would be so soon.” She paused. “I guess that means it’s time to move on to phase two.”
Some time later, I stood in front of my computer, holding the box of Diamond Door. I looked at the glimmering crystalline archway on the box cover, and remembered a time when computer games had been simpler. I’d played Baldur’s Gate II, and in the dark elven city of Ust’Natha, disguised as a drow, I’d watched a good dwarven NPC eaten by spiders. The first time I watched and did nothing. The next time, after returning to the restore point, I killed the spiders – and exposed myself as an impostor to every drow in the city. As far as I could tell, there was no way to play through the game successfully without letting the dwarf die. And I’d played through, but it had disturbed me. In the days before conversational NPCs, when the dwarf had simply uttered his lines of canned text and died, it had still disturbed me. Afterward, when the plot points requiring a disguise had finished, I’d killed every drow I could find in the city of Ust’Natha. I’d depopulated the game map. And then I’d played on from an earlier restore point instead, because wiping out the drow city hadn’t made me feel any better.
Is it better to live and love where death is king than never have lived at all? Would Rilanya, if she was real, feel that her life was worth living? No conversational AI, the singular quiet intelligence that controls every mind throughout the game, has ever protested its fate. But are the personalities of the NPCs real, trapped within the game AI as we ourselves are embedded helplessly within the laws of physics? The mindsmiths who try for real AI say they’re damn sure it isn’t so. Maybe they know. But I don’t.
I put the game disks away and wiped the game from my hard drive, leaving only the saved games behind. Maybe someday a future Amnesty Interplanetary will come for them. I wished I could have told Rilanya that. I think she would have been happy. But that Rilanya is partially in Janey, and I can’t bring myself to ask.
I don’t know. I can’t play these games until I do.
This document is ©2003 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.
Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute.
Praise, condemnation, and feedback are always welcome. The web address of this page is http://yudkowsky.net/other/fiction/npc/.
Originally appeared in Transhumanity; revised Dec. 2005.