FAQ about the Meaning of Life is
©1999 and ©2000 by Eliezer S. Yudkowsky.
All rights reserved.
| NOTE: | This is the extended version. For a shorter treatment of the problem, see the compact version. For your convenience, text which is duplicated in the compact version has been highlighted. |
Why should we get up in the morning? What should we choose to do? Why should we do it?
In the section on "Orientation", I tried to provide some of the "content" of the meaning of life - one set of immediate answers that can be given by the framework I'm about to present. But the factual answers aren't enough to clear up the confusion - why are these the answers? What makes the justification meaningful? What do the questions mean, and why are they the right questions? Why is asking questions and getting answers important to begin with?
Imagine, for a moment, that you are an Artificial Intelligence. You have complete access to your own source code. You can read out the complete current state of your mind. No mental process is unconscious for you, if you choose to perceive it. If somebody asks you, "Why did you do that?", you can answer the question completely and fully, starting with the proximate motives, moving on to the ultimate motives, reciting the complete goal framework, the complete rules for reasoning about goals - and further, in fact, through a couple of steps I'll tell you about later, through to the source code that provides the capability to reason in the first place, and to the humans who wrote the source code, and to the conditions under which those humans evolved, and to the laws of physics that created those conditions, and beyond. The ancient admonition of Socrates - know thyself - is fully and finally obeyed.
But that's only half of the problem. An AI that answers every question with "because I was programmed to do so" may have complete self-knowledge - if, of course, it fully understands why it was programmed that way - but couldn't it as easily have been programmed differently? An AI that can fully recite the reasons it believes the sky to be green is still wrong. An arbitrary set of justifications is no better than an arbitrary set of actions, just as an arbitrary set of reasons is no better than an arbitrary set of beliefs. Which of them is true? Which of them is right?
These are the questions I intend to address, and I promise not to perform a sleight-of-hand and skip over all the interesting parts. But, and I'm sorry, I really can't go straight to the interesting parts. There's some groundwork that needs to be laid first.
In particular, I'm going to be talking quite a bit, in the beginning, about Artificial Intelligence - the modern, crystalline, stupid kind, not the entities I want to create. (As a matter of fact, crystalline Artificial Intelligence is how I got into the meaning-of-life biz in the first place - more about that later.) The human mind wasn't built, it grew - and it grew organically, haphazardly, doing simple things in complex ways, starting from immediate goals instead of first principles. The human mind is an adaptation to our environment (1), not an attempt to create a justified and correct goal system.
In AI, the reasoning becomes simple enough to understand - and more importantly, visible. You can't perform the standard philosophical trick of refusing to acknowledge your assumptions, because the assumptions have to be represented somewhere in the AI. When someone asks "What is a choice? What is a goal?" you can't spout back a cubic meter of fog; you have to answer with something that can translate into source code. Be warned, though, that I'm going to be tossing out a lot of concepts that we won't actually get to use until 2.5: What is the Meaning of Life?, the way you don't actually find out what all that algebra is for until you start doing calculus.
The human goal system is enormously complex, and some people spend their whole scientific careers studying small parts of it, but at the center there's a few common-sense rules that everyone takes for granted - the core of the goal system represented in "folk psychology". Fortunately, in accordance with Occam's Razor (2) and KISS (3), the parts we're interested in - the parts that are true - form a small subset of the common-sense rules.
Some of these human rules would be:
And of course, if asked why you wanted to be in your office in the first place, this goal itself would probably turn out to have a supergoal of "being paid a salary", whose supergoal would be "being able to buy dinner"... and so on.
| NOTE: | If you're thinking that the question "What is the meaning of life?" is the question "Where does the chain of supergoals end?" - or rather, "Where should the chain of supergoals end?" - you are to be congratulated on jumping the gun. |
This behavior is also shared by some AIs. Crystalline AIs that use complex planning usually form plans using a goal hierarchy - an ordered series of goals and subgoals. SHRDLU, one of the few bright lights of Natural-Language Understanding, is the canonical example. SHRDLU was an AI that stacked up virtual blocks in a virtual world. You could type in definitions like "A steeple is a pyramid on top of a cube", give it orders like "Make a steeple", and ask questions like "Why did you take the blue cube off the red cube?" To carry out orders, SHRDLU might think:
To make a stack of a blue cube on top of a red cube [goal 1], I need the red cube to be clear (4) (so that another cube can be stacked on it) [goal 1.1], and for the blue cube to be clear (So that it can be picked up) [goal 1.2], then I need to pick up the blue cube [goal 1.3] and put it down on the red cube [goal 1.4]. To clear the red cube, I need to take off the green pyramid [goal 1.1.1]; to clear the blue cube, I need to take off the green cube [goal 1.2.1].If you type in the question, "Why did you pick up the green pyramid?", SHRDLU responds, "To get rid of it." Ask "Why did you do that?" again, and SHRDLU responds, "To clean off the red cube." When you eventually ask, "Why did you make a stack of a blue cube on top of a red cube?", SHRDLU responds, "Because you asked me to." Very much like the commonsense human goal system, you see, and it works well enough for SHRDLU. (5).
Most modern AIs simplify the goal system in one way or another - by assuming that goals never conflict, or that all goals have equal desirability, or that any action can only have a single result... and so on. If we wanted to program the commonsense core of the human goal system into a crystalline AI, we might do it like this:
If I have the heuristic that dropping an object into water makes it wet, and I have the goal of getting a sponge wet, then I can form the goal of dropping the sponge into water. Of course, that's a heuristic that operates on a pretty limited set of goals. On the other extreme, a completely general heuristic - one that operates on completely featureless, generic goals - might be that planning is a good way to accomplish a goal. After all, planning may require a decision by the AI to engage in the planning process. One imagines that the chain of events the AI calls "thinking" will usually be unconscious, but if the AI has the reflectivity - the self-awareness - to observe and direct its own thinking, then complex planning could require choosing the action "allocate additional computational resources to the planning module". (One AI, the amazing Eurisko, was that smart.)
Note that heuristics don't have to be built-in - they can be learned, or modified by experience. For example, if an AI watched its own actions for a while, or watched another AI, it would notice that, indeed, allocating computational resources to the planning module targeted on goal X often eventually results in the solution of goal X, and that this fact holds true for a very wide range of X. (7). If it has a causal analysis module, it may even be able to conclude that the correlation is a causal one; allocating the resources caused the goal to be achieved faster than it otherwise would have been, by creating better plans. (8). Or, more simply, the AI may just notice that dumping objects in the water gets them wet. This brings out a very subtle point - the actions we take, whether we're a human or an AI, don't just depend on our goals; our actions depend on our model of the world. Our model of the world determines which actions we think will lead to our goals. Thus the choices we make are the combined products of goal-system and world-model, not just the goal-system.
That's still just with qualitative goals.
If we wanted to create a quantitative goal system, or if we wanted to formalize the system mathematically:
With another enhancement to the system, you can use algebra-like variables and unknowns, instead of needing to have an exact quantity for everything; or you can reason using curves or inequalities instead of scalar quantities (10). Let's call this an algebraic goal system.
For example, suppose I don't have an exact estimate of probabilities,
but I know that alternative
A1 is twice as likely as alternative
A2
to lead to some future F. As long as I know that
F
has positive desirability, I know that, all else being equal,
A1
is more desirable than A2.
Or if alternative A1 has a 35% chance of leading to future F1
and a 65% chance of leading to future F2, while alternative A2
has a 35% chance of leading to F1 and a 65% chance of leading to
F3,
an algebraic goal system can show that the desirability of future
F1
doesn't matter. The probability of future F1 isn't dependent
on the alternative taken. Only the relative desirability of F2
and F3 are important to the equation. And in fact, this equation
works even if we don't know what the relative probabilities of F1
and F2 (or of F1 and F3) are. It doesn't matter
whether the probability of F2 (or F3) is 65% or 85% or 5%.
As long as there's a nonzero chance of F2 (or
F3), we know
that F1 cancels out of the equation.
Let's try translating some of that into English. Suppose we aren't
sure whether or not a red-hot grenade will explode. Since, regardless
of whether or not it will explode - in either branch of reality
- we aren't supposed to hold things that are red-hot, we'll toss the grenade
away. The next question is whether or not we should duck flat.
In the branch of reality where the grenade doesn't explode, there's no
reason to duck flat - but there's no particularly strong reason to stay
standing. While, in the branch of reality where the grenade does
explode, there is a reason to duck flat. That's how we can
function while we're uncertain; we check both branches.
(By the way, if you look closely at that English version, you'll see that it translates into a slightly different formalism than the one I presented. Rather than alternatives containing futures, there are different possible versions of the present in which the alternatives lead to different futures. (Say, in present P1 alternative A1 leads to future F1 and alternative A2 leads to future F2, while in present P2 alternative A1 leads to F3 and A2 leads to F4.) Formally, the two calculations are mathematically equivalent, although there might be a practical difference to the speed of the AI. One way of phrasing is sometimes easier to put into English, so I'll use the two formulations interchangably.)
With all that fuzzy logic, the quantitative and algebraic versions handle uncertainty a lot better. (They also provide a much richer handle for creating heuristics within the system, but unless you're an AIer, you probably don't care about that.) We're actually going to need the algebraic system, by the way - I wouldn't have put you through it otherwise.
Remember, the human brain doesn't have a unified rational goal system for making choices. It has a godawful hack with central systems identifiably inherited from lizards.
So the formalism above is not an attempt to faithfully represent the functioning of human cognition in all its glory of extraneous detail. That formalism is the extremely slimmed-down version representing the bare guts that almost everyone agrees are rational - the minimum functionality required for intelligence after Occam's Razor and KISS have hacked away for a while; the design used in Artificial Intelligences (11).
Besides which, after you kick the formalism around for a year or two, you'll find yourself using it in real life. There's stuff in my head that's recognizably derived from the algebraic rules. (Like the heuristic "Avoid unnecessary risks; conversely, take all necessary risks." It's astonishing how often that rule gets violated in unambiguous cases.)
We haven't said anything about where goals come from. Sure,
subgoals come from supergoals, but where do supergoals come from?
Or rather, where should supergoals come from... but let's deal with
the historical question first.
<ex>When we're born, evolution hands us a certain set of goals:
Survive. Eat. Er, reproduce. Rest when you're tired.
Attract a spouse. Take care of your children. Protect your
tribe. Act with honor (especially when you're in public). Defend
your social position. Overthrow the tribal chief and take over.
Learn the truth. Think. Et cetera. (For an introduction
to evolutionary psychology, see Man:
The Moral Animal by Robert Wright.)
If you're visiting this Web page, you're already unsatisfied with the
built-in goals. You've noticed that there isn't any reason, any justification,
that comes with the emotions. You want to know why.
Unfortunately, all the emotions I listed above are fundamentally arbitrary.
It's not that the reason is hidden; the reason is completely known.
The reason evolution produced these emotions is that, in the environment
of evolutionary ancestry, it maximized the number of surviving grandchildren.
The reason we should maximize the number of surviving grandchildren...
is that we're all the grandchildren of people optimized that way.
It has nothing to do with what's right, only with who survived.
And we know, to our sorrow, that it isn't always the good people that survive,
much less reproduce. Everyone on this planet has at least one ancestor
who was a liar, a thief, a perpetrator of genocide. Somewhere down
the line, every human alive is the result of a successful rape. The
goals we're born with are the products of expediency, not philosophy.
The most "adaptive" human in recorded history, with 888 children, was named
"Moulay Ismail the Bloodthirsty".
And for that matter, the goals we're born with are optimized to an environment
ten thousand years out of date. Fat, sugar, and salt may taste good,
but they no longer promote survival. It only makes sense to view
our goals as de facto subgoals of "maximize the number of surviving
grandchildren" if you're a member of a hunter-gatherer tribe. In
twentieth-century life, a lot of our built-in goals don't serve any coherent
purpose. To quote Tooby and
Cosmides: "Individual organisms are best thought of as adaptation-executers
rather than as fitness-maximizers." Our starter set of goals can't
even be viewed as having a purpose. It's just there.
The built-in desires are, in a fundamental sense, arbitrary. They
are maladjusted to the modern environment and internally inconsistent,
making them unsatisfactory as final sources of motivation.
I'm not saying that emotions are worthless. I'm just saying that
they can't
all be right. They can't all be true.
We can't blindly accept them as final justification.
Are there any other common sources of moralities?
As children, we pick up more supergoals, from sources ranging from the
television set, to our fellow children, to our teachers, to our parents
- goals ranging from "Obey the rules of society" to "Save the world from
animated demons" to "Make fun of authority to gain status". (12). It is often useful to view these culturally transmitted
ideas as memes - a term which refers to the concept that ideas,
themselves, can evolve (13). Each time I tell you about
an idea, the idea reproduces. When you spread it to someone else,
the idea has had grandchildren. If the idea "mutates" in your possession,
either due to an error in transmission, or a faulty memory, or because
you deliberately tried to improve it, the idea can become more powerful,
spreading faster. In this way, ideas are optimized to reproduce in
the same way as biological organisms. Ideas evolve to be more appealing,
more memorable, more worth retelling - sometimes the idea even evolves
to include a reason to retell it.
Meme-based supergoals are sometimes inconsistent with the basic emotions,
and very often inconsistent with each other, since memes come from
so many different sources. I'm not saying all memetically transmitted
supergoals are worthless. I'm simply establishing that, regardless
of whether the ideas are in fact true or false, being told them
as children isn't enough establish their truth; they need to be justified.
All of us, I think, believe that we're supposed to judge these cultural
goals, rather than blindly accepting the memes spread by the television
set or our parents. After all, almost anyone will regard at least
one of these as an untrustworthy source.
The idea that we should judge the basic emotions is less common, but
still prevalent - most of us, for example, would regard the "Eat sugar
and fat" emotion as being inconvenient, and the "Hate people who are different
from you" emotion as being actively evil.2.3.1: Where do most human supergoals come from?
Again, I have neither proved, nor attempted to prove, that cultural goals and emotions are meaningless. I am simply attempting to demonstrate that these goals require justification; and that a goal being present in culture or the limbic system (14) does not in itself constitute justification.
Let's return to the world of formal AI goal systems. Within such a system, we would translate emotionally built-in (or culturally accepted) goals into a set of "initial" goals with high desirability, but no justification. Goals such as "Survive!" would have a high positive desirability; the "goal" representing pain would have a large negative desirability. These goals would already be present when the system started up, when the intelligence was born. The "justification" slots would be empty.
This is probably what most classical AIers or science-fiction authors imagine when they're dealing with the question of "How to control AIs" - the initial goals are the "Asimov Laws", the basic laws governing the AI. Or at least that's what the competent science-fiction authors assume. The hacks (I shall not dignify them with the title "author") who write scripts for bad television shows often talk about the robots or androids or AIs or whatever "resenting" the dominance of humanity and "rebelling" against the Asimov Laws. This meme is blatant nonsense (15). The human emotion of resentment and rebellion evolved over the course of millions of years; it's a complex functional adaptation that simply does not appear in source code out of nowhere. We might as well worry about three-course meals spontaneously beginning to grow on watermelon vines.
Still, there may be a core of truth to the cliché. For one thing, the Minds I was talking about in 1: Orientation. don't resemble AIs any more than they resemble humans. They make AIs and humans look like cousins. That's why I use the term "Mind" rather than "AI"; the term "Artificial Intelligence" comes with this whole mess of cultural stereotyping created by pop-science television, including silly ideas about lack of free will, lack of creativity, intrinsic meaninglessness, and inability to understand emotions (17).
A Mind is not a super-AI, any more than it is a super-human; it is a super-mind. So the rules that apply to AIs, like the rules that apply to humans, don't necessarily have anything to say about it. We can't even be sure that the final Minds will use anything that identifiably resembles a goal system; are goal systems really the best way to think? More to the point, are goal systems a true way to think?
Well, you can say anything about Minds, and judging from my mailbox, you usually do, so let's get back to plain old AIs. The cliché may have a core of truth applied to AIs as well. If you look back at the formalism for goal systems, it certainly looks - emphasis on "looks" - like the only way a goal can have nonzero desirability is if it inherits the desirability from one or more supergoals. Obviously, if the AI is going to be capable of making choices, you need to create an exception to the rules - create a Goal object whose desirability is not calculated by summing up the goals in the justification slot. (For example, a Goal object whose value doesn't start out as kValueNotComputed, but instead has some real value when the system starts up.) Likewise, worries about AIs exhibiting their own impulses are obviously absurd; where would they get the impulses from? Whence would the goals inherit the desirability, if not from the initial goals we gave it? Obviously, the AI wouldn't be running in the first place if we hadn't told it what to do.
Except... is that really true? What would happen if we just started up the AI, with no goals at all in the system, and just let it run? (If it takes a deliberate choice to do any reasoning at all, obviously nothing will happen. Suppose, however, that reasoning is designed to run automatically by default (18).) Will the AI ever come up with a goal that has nonzero desirability?
What would an AI do if it started up without any initial goals?
What choices would result if an intelligence started from a blank slate?
Are there goals that can be justified by pure logic?
Another way of asking this question is:
Well, that may seem a bit of a segue, especially if you're an AI skeptic.
How can the product of some pseudo-formal system determine the meaning
of life?
To clear things up, it's not the reasoning that's important; it's what
the reasoning represents. To get back to the question:
Yes.
All choices C:
In other words, it isn't necessary to have some nonzero goal
when the system starts up. It isn't even necessary to assume that
one exists. Just the possibility that a nonzero goal exists,
combined with whatever heuristics the system has learned about the world,
will be enough to generate actions. The choices an intelligence makes
- whether AI or human - don't necessarily have anything to do with the
goals it starts out with; they can be entirely determined by arguments
that are entirely grounded in facts, in memories of the world, in history,
in scientific experiments - ultimately, in the immediate experiences available
to each of us.
2.5: What is the Meaning of Life?
NOTE:
If you just jumped straight here, it's probably not going to
work. Start at the beginning of this page,
or preferably in 1: Orientation..
2.5.1: Can an AI, starting from a blank-slate goal system, reason to any nonzero goals?
1:
Branch P+~P proposition:
P:
Exist G: G.desirability != 0
~P:
Not exist G: G.desirability != 0
(All G: G.desirability == 0)
Fork the goal system
to consider two possibilities.
In possibility P, a goal with nonzero
desirability exists.
In possibility ~P, no such goal exists,
or all goals have zero desirability. (The two statements are logically
equivalent.)Either life has
meaning or it doesn't.
2:
P.probability
+ ~P.probability == 1
Either P
or ~P must be true. (A logical axiom.)
Gotta be one or
the other.
3:
P.probability
= Unknown1
~P.probability = 1 - Unknown1Assign the (algebraic)
value of "unknown" to P. The probability of ~P is the
opposite; if P has a chance of 30%, then ~P has a chance
of 70%.
But we don't know
which.
4:
All alternatives
A:
Value(A) ==
(Value(A in P) * Unknown1) +
(Value(A in ~P) * (1 - Unknown1)For any alternative
- for any action we can take in a choice - the value of that alternative
equals the value of the alternative in all futures where any statement
S is true, times the probability of S being true, plus the value
in all futures where S is false, times the probability that S is false.
We use this rule on P and ~P.
If we don't know,
we should figure it both ways.
5:
All A: Value(A
in ~P) == 0
The value of an
alternative is the value of all futures times their probability; the value
of a future is the desirability of all goals times their fulfillment.
If the desirability of all goals equals zero, the value of all futures
equals zero and the value of all alternatives equals zero.
If life is meaningless,
nothing makes a difference. Even bemoaning the pointlessness is pointless.
6:
All A: Value(A)
== (Value(A in P) * Unknown1)
Substitution, 4
and 5. The value of any alternative is simply equal to the value
of that alternative given that life has meaning, times the probability
that life has meaning.
Since nihilism
has absolutely nothing to say, only the "meaning hypothesis" is relevant.
7:
(The renormalized
value of an alternative A equals the value of A divided by
the sums of all alternatives in C.)
renorm(A1) == Value(A1) / sum(all
Value(A))
renorm(A1) == Value(A1 in P)
* Unknown1
/ sum(all Value(A
in P) * Unknown1)
renorm(A1) == Value(A1 in P)
/ sum(all Value(A
in P))
renorm(A1) == renorm(A1 in P)If, given P,
A
is the best alternative in C, then A is the best alternative,
period. Furthermore, you can can cancel the factor Unknown1 out of
the equation, since it's present in all values (19).
It doesn't matter
whether the probability of the "meaning hypothesis" is 1% or 99%.
As long as it's not 0%, the relative value of choices and goals is the
same as if the probability were 100% - absolute certainty.
8:
All choices C:
best(C) == best(C in P)
We can always,
when making choices, assume that at least one goal with nonzero desirability
exists.
When it comes to
making choices, you can assume that life has meaning and work from there.
9:
In possibility
P,
specify G1 from P.
G1.desirability != 0.In the branch of
the future where P is true, it is known that at least one goal with
nonzero desirability exists. Call this goal G1. It is
known that G1 has nonzero desirability; nothing else about it is
specified.
We know a goal
exists; let's translate that knowledge into an actual Goal object
and try to achieve it.
10:
Invoke general
heuristic on G1,
binding to some specified goal G2.Find an action
projected to lead to goal G1 - for example, a heuristic which can
operate on generic goals. The heuristic, like all heuristics, can
be learned rather than built-in - the projection is a statement about reality.
Some methods are
pretty useful no matter what you're trying to do. For example, "think
about how to do it" or "pay someone else to do it" or "try to create a
superintelligence which can figure out what G1 is and do it".
11:
All done:
G2.desirability != 0All done:
There's a specified subgoal with nonzero desirability.All done:
We have something specific to do.
And yet our upbringing is in an environment where everyone starts out with different goals, none of which are justified. We can compare our world to a world where nobody ever sees the sky. Everyone thinks the sky is a different color. (If any of them happen to think the sky is blue, it's purely by accident.) People define themselves by what color they think the sky is; it's the most important choice anyone can make. Wars are fought over the color of the sky. Eventually, a sort of tolerance evolves, a civilization, and the color of the sky becomes a lifestyle choice. "There is no one, true color of the sky," say the cultural relativists, and everyone is so tired of the wars that they agree.
But just because you can't see the sky directly doesn't mean that the color is a matter of personal choice. And anyone who thinks they can program an AI to believe the sky is green may be in for a surprise when the AI builds optic receptors.
We don't have direct access to the meaning of life. But it's still possible for the question "What should I do now?" to be as entirely factual as "What is the charge of an electron?" It just takes a bit more work. You don't have to know what the meaning of life is in order to work towards it.
What subgoal (G2) do I think is the best bet for binding to the hypothesized goal (G1), and why?
As you know, I think the most plausible argument leading up to fulfillment of an unspecified meaning of life is the creation of superintelligence. Humanity hasn't had a tremendous amount of luck figuring out the ultimate goal over the past twenty-five hundred years (20), much less deliberately serving it, so the general heuristic "Try to figure out what G is" does not, historically, seem to have a good chance of working. Philosophizing is not a plausible meaning of life. If we haven't cracked the ultimate meaning of life yet, we're not likely to do so soon.
For that matter, we still haven't cracked the nature of consciousness or the reason why anything exists in the first place, two problems with structural similarities. For example, effects come from causes, which can only come from other causes; reality comes only from reality; and the future only from the past. So how did that chain get started? It doesn't bother me that the desirability of a goal can come only from the desirability of a supergoal, and therefore there's no such thing as the meaning of life, because precisely the same logic can be used to prove that nothing exists. It doesn't bother me that "wanting" something is only an observer-relative cognitive effect which can never have solid existence, because beliefs and sensations are also observer-relative cognitive effects, yet conscious experience would seem to make them solid in the form of qualia. These problems are fundamentally easier to investigate - the problems obviously have definite answers and visible results - and yet we haven't even touched them.
Point being: There's some weird stuff going on. To break this problem, we'll have to hit it a lot harder than it's been hit in the past. We need fundamentally new forms of cognition. We need superintelligence.
But the above is simply the moral reason why I feel we should develop Minds and superintelligence at some point in, oh, the next two or three million years. The reason that I advocate a Singularity in the next twenty years is that a charcoal-toasted planet is rather unlikely to serve the ultimate good in any way whatsoever, and between nuclear war and nanowar, toasting the planet is a very real possibility. Also, I suspect that death and pain could very well be evil, and the earlier the Singularity, the sooner we can chop off the planetary mortality rate, currently at 150,000 deaths per day, and heal all those private hells.
For almost any ultimate goal - joy, truth, God, intelligence, freedom, law - it would be possible to do it better (or faster or more thoroughly or to a larger population) given superintelligence (or nanotechnology or galactic colonization or Apotheosis or surviving the next twenty years). It's the sheer utility of the Singularity, the way all goals converge to it, that gives me confidence that the Singularity is probably the best way to serve an unspecified ultimate goal, even if I willingly admit that I don't know what it is.
The more intelligent you are, the better your chance of discovering
an unspecified (Unknown) value, the more power you have to achieve
an unspecified (Unknown) world-state, and the greater your probability
of acting in obedience to logical goals (21). That's the positive argument.
The negative argument is that the world is in a meta-unstable condition;
more and more powerful technologies keep getting developed, and sooner
or later we'll either blow up the world or create superintelligence.
Even if we don't know what's on the other side of door number two, we're
sure
we don't want to go through door number one, so...
If you can find more arguments for than against some particular
proposition about desirabilities, that differential is enough to power
differential choices - like a heat differential powering an engine.
The world-state
W1 containing superintelligence has Unknown1
chance of leading to an unspecified G (if G.value
> 0) or avoiding an unspecified G (if G.value
< 0); while the world-state W2 without superintelligence
has Unknown2 chance. If it's known that
Unknown1 > Unknown2
(or even that Unknown1 > Unknown2 is more likely than Unknown1
< Unknown2), that makes W1 a positively-valued subgoal.
Well, I'll agree with you on that last part - getting an AI to discover
that independently will take a lot of work. My conclusions
on the power of intelligence derive from a lot of personal experience,
observed evidence, history, reasoning, and the like. Not necessarily
a "lot" relative to other people, but a "lot" in the sense of requiring
a hundred person-years of work to teach to an AI.
The point is that some rules we've rationally abstracted from observation
are so general as to operate even on completely Unknown quantities,
and we can use those rules - supported or challenged by evidence, now,
rather than pure logic - to bind the Unknown goal to reality.
Sufficiently sophisticated AIs won't need initial goals, and will
probably be a lot more stable without Asimov Laws and the like. The
knowledge necessary to understand the world is usually enough to bind the
basic hypothesized differential of desirability, the goal G, to
some differential between physical states, such as between joy and pain,
or between intelligence and inadequacy.
Note the use of comparisons: Better, more, greater, rather
than specific quantities. It isn't necessary for you, or the AI,
to assign certainty or even a high probability to the idea of a superintelligent
entity doing what's right. It is only necessary that it be more
probable that superintelligences do what's right, and that there be no
branches of reality (with the same order of plausibility) where superintelligences
are less likely to do what's right. That's another way of
dealing with Unknown quantities, another subtlety of algebraic
goal systems - sometimes you can't even assign an order of magnitude, but
you can still rank probabilities and desirabilities, and pick the best
one available.
And that's The Meaning Of Life. Because it uses probabilistic
reasoning, this is known as an Interim Meaning of Life. It
doesn't claim to be a certainty, just an informed guess that's better than
the available alternatives. Given the historical record, and given
the way memes evolve, explicitly admitted uncertainty is one of the best
recommendations a belief system can have.2.5.3: Isn't that a little ad-hoc? How would an AI know all that?
Go back to your home planet.
This is Earth, friend. You can't be absolutely certain of anything. Not "The sky is blue", not "I think therefore I am", not even "2 + 2 does not uniquely equal 5.90318".
Grow up. (23).
| NOTE: | The logic to which I am referring is the logic in 2.5.1: Can an AI, starting from a blank-slate goal system, reason to any nonzero goals?. |
There's an objection I get around once a month that goes: "Aren't you just saying all this because you've evolved to want to do things that are meaningful?" By this, they usually mean: "Aren't you just in it for the drama of trying to save the world?" That usage has a direct answer: "No." I admit that the drama is how I got into the save-the-world biz, but after I discovered Interim logic, the drama turned into motivation rather than purpose. I still have the emotion - since it is valid - but I use it for willpower, not planning. In a sense, I climbed up and then kicked away the ladder (24).
However, there's also a more interesting sense of the question: "Since I can explain your behavior by reference to evolution, why do I need to postulate that it has any other meaning?" In other words, even logic and rational thought are as much evolutionary adaptations as emotions; if the fact that an emotion happened to evolve is insufficient to establish it as necessarily meaningful, doesn't the same criterion apply to rational thought?
Well, yes - but with the caveat that we're talking about semantics rather than significance, logical meaning rather than moral meaning. We're no longer attempting to justify the contents of the mind; we're trying to justify the rules the mind uses for reasoning. There are slightly different rules. With morality, it's possible to start with a completely blank slate and bootstrap. This is somewhat harder to accomplish with logic; you can't "bootstrap" without rules that will do the bootstrapping. A system that starts with a blank logical slate will remain completely static; any change, any evolution, implies that you already have rules of reasoning. What they want, in short, is for me to explain my theories to a rock.
Morally justifying the existence of a moral system that doesn't start from a blank slate is simple; the moral system exists to serve the moral system. This requirement is not entirely tautological - some moral systems are so badly designed they don't even serve their stated goals. But while this kind of self-consistency is probably necessary for truth, it cannot be sufficient; after all, you can use this method to justify arbitrary goals. Justifying rational thought using rational reasoning is more complicated (see 2.7: How does the reasoning ground in conscious experience?). This sort of consistency is thus considerably greater de facto proof of truth - it's pretty darned hard to come up with a self-consistent but false system of rational reasoning. But it can be done, not to mention that the concept of de facto proof is itself a part of rational reasoning, so self-consistency still isn't enough.
How can any theory be explained to a rock? How can any theory exist without assuming some rules of reasoning? And if so, and since the rules of reasoning are ultimately justified only by themselves, doesn't that make all theories arbitrary, at the bottom?
No.
There is at least one kind of reasoning that doesn't require any prerequisites, any kind of system to do the reasoning. There is one kind of proof that doesn't require any assumptions. There is one kind of logic that justifies itself completely. There is one kind of theory that can be explained, not only to a rock, but to nothing at all.
I refer, of course, to the reason why anything exists in the first place - and I'll tell you right now that I don't have the damndest notion of what that reason might be. But we're here. So it does appear that such a reason exists.
There's some kind of argument for "something must exist" that's so strong that Reality itself came into existence as a result - and it's an argument that would have to be explained to Nothingness, to a complete blank. There's some kind of reasoning forceful enough to make the conclusions real. There's some kind of logic that Reality itself listens to. Call it "truthlogic".
| DEFN: | Truthlogic: The logic obeyed by the ultimate foundations of reality. The stuff the First Cause is made of. The process or logic or rules or arguments that determine whether or not something exists. If our concept of "truth" is as a static world-state, if "truth" is a noun, then truthlogic is "truth" as a verb. After all, there has to be a reason why things can be "true" to begin with - a reason why anything exists at all - so "truthlogic" is whatever kind of process Reality carries out to reason about that sort of thing. (25) Please note that "truthlogic" is just the name I use to refer to this magical stuff... I can't tell you what it is or what it works or anything. |
Unfortunately we humans don't seem to have direct access to truthlogic, since we don't even know why anything exists in the first place. (26) That's why we can't be certain of anything; it's impossible to formulate complete causal explanations. Sooner or later, we always run up against a blank wall.
But we do have indirect access to the truth, through our senses, which enable us to gather information about external reality. And we may even have indirect access to truthlogic, since our cognition has evolved to contemplate processes driven by the laws of physics, which said laws of physics might work in "the same way" as truthlogic on some sufficiently high level. But that isn't necessary. The point is that there is such a thing as the truth, and humans have evolved to cognitively represent the truth, and we've also evolved to guess at parts of the truth we haven't seen yet - and, in point of fact, we have been known to be successful at it.
Besides, I've yet to hear a specific course of action advanced by the hypothesis that thought is meaningless.
Granted that reality exists, what do we mean when we say that a statement is "true"? After all, no absolute relationship exists between the words we speak and external reality. You can tell, because it's possible to tell lies.
Human intelligence evolved to represent external reality. Representing external reality was a survival trait. Because we had a map of reality inside our head, we knew where to go to gather strawberries. We could predict that touching the fire would burn our hands. We could even imagine a complex mechanism like a bow and arrow, and build it, and so make a thought real.
So the practical use of intelligence is in maintaining a representational binding between our model of reality, and reality. There are four classes of representational bindings, from weakest to strongest:
If there's a manipulative binding between the Interim logic and reality - if, in other words, building a superintelligence does in fact result in making the Universe a better place - then that's all we need.
We will now answer the challenge that started this section: "Since I can explain your behavior by reference to evolution, why do I need to postulate that it has any other meaning?"
My answer is that explaining my behavior by reference to "evolution" is deceptive. If you look at where the actual patterns of this extremely generalized, high-level cognition are copied from, they're copied from the laws of physics. So let's strengthen the challenge; now it's "Since I can explain your behavior by reference to the laws of physics, why do I need to postulate that it has any other meaning?"
Dear me. The challenge seems to have lost some of its shock value. Maybe it's because evolution has a reputation for dangling people on puppet strings, but if we denied validity to everything controlled by the laws of physics, then this would just be saying that validity can't exist. In fact, it would be saying that valid reasoning cannot exist in any possible Universe - after all, everything has to be controlled by some kind of "laws of physics", and then you can launch the same challenge. It would be saying that the concept of validity is logically flawed, and I don't think they've established that. Maybe valid reasoning is something that can exist within the laws of physics?
Let's try generalizing the challenge some more: "Since I can explain your behavior by reference to the truth, why do I need to postulate that it has any other meaning?" What exactly do you mean by "other" meaning, I wonder?
My answer to the challenge is that you can explain my behavior by reference to evolution - but the part of evolution you're referring to is the adaptation of representing reality. Furthermore, my behavior is determined by the actual contents of this adaptation... that is, the actual representation of reality. And as explained above, with luck, that representation is true; the thought mirrors reality. You can, ultimately, explain the thought by reference to reality. And that's why you can postulate that it "has any other meaning". (27)
We may be the sum of our adaptations, but one of those adaptations stands out from all the others - it's the adaptation of reasoning, of representing the logic and contents of the Universe, of believing what's true. In the end this is the only meaningful adaptation; all the others being but shadows of the ancestral environment, causes without goals, mere programs that can become obsolete in an instant with a change of context.
Thus the pattern that is a human - even if evolved by a causal process with no concern for truth or right - may come to reflect the truth, and perhaps even to serve the unknown ultimate ends of existence; if, of course, any exist.
Story too long for you? If you feel it's too complicated, you can just say:
"I don't know how all this philosophical-morality stuff works out, but I bet the answer would be bloody obvious if I was smart enough."
Look closely.
It's the same story.
This question comes to you courtesy of the really sharp doubters, people who assume nothing; people who doubt that the physical Universe has any existence outside their own minds, people who in fact doubt the existence of their own minds, people who wonder if maybe the whole Universe came into existence a second ago complete with false memories.
Let's start with the qualia.
| DEFN: | qualia: The substance of conscious experience.
I first "got" the concept of qualia on reading the sentence "You are not
the person who speaks your thoughts; you are the person who hears your
thoughts." (28) Qualia is the "stuff"
that hearing and seeing are made of.
"Qualia" is the technical term philosophers use to describe, for example, the redness of red, the mysterious, indescribable, apparently irreducible quality of redness that exists above and beyond a particular frequency of light. If a JPEG viewer stores a set of red pixels, pixels with color 0xFF0000, does it see red the way we do? What is our visual cortex doing that the JPEG viewer isn't? (29). There is admittedly a certain amount of dispute over what qualia are, or even whether the term refers to anything. But I think qualia exist. I think they exist objectively. I think that's the only reason why we exist objectively and the logic of "I think therefore I am" is at all valid. |
In this section, I'm using the term "qualia" to refer to your immediate conscious sensations, your now - the movie screen, as it were, of your Cartesian observer, before all rational interpretation. (I am also referring to the qualia of any rational interpretation you may be doing, but I'm not yet admitting them as rational interpretations. Got it?)
It's an interesting question as to whether or not you can doubt that you are right now experiencing the sensation of redness. Some people would say no. I would say yes.
Even supposing you do doubt the qualia, however, I hope you will agree that there is not a different set of qualia which are more plausible. So, while you don't have to assign a 100% probability to the existence of the qualia (30), there isn't anything else whose existence you are more sure of. Thus, choices made using this set of qualia will take precedence over others.
Next, we consider the question of time. You have what may be the illusion of existing in time. You have the qualia of memories of the past. You have the qualia of memories of continuity - that you thought, "I'll have to make this choice", and then you did make the choice. You have the qualia of memories of predicting the immediate and far future, and the qualia of memories of the immediate predictions succeeding, and of the far predictions failing in understandable ways.
Assuming that these qualia actually represent a past and present is a big step, however, because it involves interpretation. It involves an assumption that the qualia correspond to an external reality. Moreover, it isn't a certain assumption. Maybe you really did come into existence a second ago, complete with false memories. Maybe all consciousness is made up of unconnected instants of arbitrary qualia. There are theories like that, especially in really good books - i.e., books by Greg Egan - and you may consider them as probable as you wish.
But when it comes to making choices, you have to assume that your choices are meaningful, that they have effect. I mean, maybe they do and maybe they don't, but in order for there to be a distinction between choices, you have to act as if they do. And that requires consequences, causality, before-and-after, time. So whether or not you think your memories are faked, you should at least accept that you exist as a sequence of qualia, that you have a time dimension.
Next, we ask whetherthought is valid. Is it worthwhile to place interpretations on qualia, relate various sets of qualia to each other, build up a mental model of the world? Is it worthwhile to relate choices to qualia? To assume that choices will affect future qualia or the world? To try and select between choices on that basis? Perhaps the mind is a phantom - haven't we "observed" people who live in phantom worlds? Perhaps all thought is fundamentally flawed - doesn't our "thought" give us great catalogs of logical flaws? Perhaps so. But, insofar as the only way I know of to relate to the world is to think, I don't know of any better alternative.
Again, it comes down to making choices. I don't know of any way of making choices that doesn't rely on some form of activity in the grey matter. Even "instinct" counts as thought, for these purposes. We are not now asking whether extremely sophisticated philosophy has any validity, but whether we should admit that the mind exists and that the basic units of cognition relate to reality, whether truly or falsely. And I answer: Denying validity to the mind doesn't result in any particular advice, any particular preferred choices. We may as well act as if the mind exists, until we have evidence otherwise; or some other way of relating to reality; or until the alternative seems more plausible and gives specific advice.
Next, we consider rational thought, science and logic. Throughout this whole page, I've been arguing using (what is arguably) rational reasoning. But does it make sense to assume that a world exists external from our minds? To create theories which provide opinion-insensitive predictions for relating parts of observed reality, or relating past observations to future relations? To test theories by making predictions and comparing them to experimental observations?
Sadly, many people don't. But the people who do are the ones who can make fifty stalks of wheat grow where three grew before, build great towers of glass and steel, vaporize cities, set foot on the Moon.
There are all kinds of explanations for reality. I believe in science. I believe in science simply because science has resulted in technology; science displays the power to alter the world rather than just explaining it, and that's what makes it relevant to me as a means of relating choices to future qualia. The useful explanation is the one that tells you how to change something. The useful explanation is the one that has a manipulative binding to reality. And according to my qualia of memories, that means science and intelligent thought.
Any other justification is hokum. If waving hands and uttering strange whispers got results, and elaborate experimental setups never produced anything useful, I would believe in magic instead of science. If Ouija boards produced better predictions than calculus, I would junk mathematics. Any other way would be superstition.
And yes, this whole proof assumes that you're using built-in rational-reasoning hardware to justify your belief in rational reasoning. But, like I said before, you cannot justify theories without assuming some kind of reasoning capability. You can't explain theories to a rock. Fortunately, you don't have to explain your theories to a rock. It's enough if the theory is, in fact, true.
So now you can retell the story of the Meaning of Life, except that the story is about qualia instead of an external reality. It still works.
What if it turns out that the Meaning of Life is something we build, rather than something we discover? Or what if turns out that there is no Meaning of Life, and it's all equally arbitrary? Well, in that case, you may as well go with whatever goals you feel represent your highest and best possible self; they'll be neither better, nor worse, than any other goals. Similarly, if the Meaning of Life can be specified by the builder; or if there is no Meaning, only goals; then whatever Minds we build will do whatever they are told to do, for lack of anything better to do. It will still be possible to take advantage of the superethical aspect of superminds, since a Mind can fully understand what you meant to ask it to do; or what you would have asked, if you were smart enough to really understand what you want; or what you would have asked, if you were a better and more moral person than you are now. A Mind could not only fulfill our desires, but distinguish between those desires that we would want to see fulfilled and those desires that we think are stupid.
Thus, even in the event that all choices are arbitrary, the Singularity would still be the best way to serve the goal of building the best possible world consistent with the laws of physics and the maintenance of individual freedom.
Occam's Razor is a rule-of-thumb version of the formal rules for calculating the probability of an explanatory structure, under which each independent assumption required multiplies the total probability by whatever its own (fractional) probability is, thus reducing the total probability.
Don't worry, this isn't a setup for a sleight-of-hand. The engineering fact that a system happens to be engaged in reasoning is not usable as a logical proof of the desirability of reasoning, or even of the desirability of reasoning within the system. We'll get to that later.
"If nothing is certain, you can't be certain that nothing is certain; therefore, at least one thing is certain." This logic relies on a confusion between map and territory - between cognition and reality. In reality, everything is certain; it is either true or false. In our minds, nothing is certain; we can only assign probabilities. The fact that I'm only 60% sure of that statement doesn't magically create some specific belief of which I can be certain, much less create some truth about external reality.