Example Feed

3 | Overcoming Bias 4 | 5 | 6 | tag:typepad.com,2003:weblog-549792 7 | 2009-01-18T04:03:29-05:00 8 | A forum for those serious about trying to overcome their own biases in beliefs and actions. 9 | TypePad 10 | 11 | In Praise of Boredom 12 | 13 | 14 | tag:typepad.com,2003:post-61530602 15 | 2009-01-18T04:03:29-05:00 16 | 2009-01-18T12:21:05-05:00 17 |

Previously in series: Seduced by ImaginationIf I were to make a short list of the most important human qualities -- and yes, this is a fool's errand, because human nature is immensely complicated, and we don't even notice all the...

18 | 19 | Eliezer Yudkowsky 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 |

Previously in series: Seduced by Imagination

If I were to make a short list of the most important human qualities -

- and yes, this is a fool's errand, because human nature is immensely complicated, and we don't even notice all the tiny tweaks that fine-tune our moral categories, and who knows how our attractors would change shape if we eliminated a single human emotion -

- but even so, if I had to point to just a few things and say, "If you lose just one of these things, you lose most of the expected value of the Future; but conversely if an alien species independently evolved just these few things, we might even want to be friends" -

- then the top three items on the list would be sympathy, boredom and consciousness.

Boredom is a subtle-splendored thing. You wouldn't want to get bored with breathing, for example - even though it's the same motions over and over and over and over again for minutes and hours and years and decades.

Now I know some of you out there are thinking, "Actually, I'm quite bored with breathing and I wish I didn't have to," but then you wouldn't want to get bored with switching transistors.

According to the human value of boredom, some things are allowed to be highly repetitive without being boring - like obeying the same laws of physics every day.

Conversely, other repetitions are supposed to be boring, like playing the same level of Super Mario Brothers over and over and over again until the end of time. And let us note that if the pixels in the game level have a slightly different color each time, that is not sufficient to prevent it from being "the same damn thing, over and over and over again".

Once you take a closer look, it turns out that boredom is quite interesting.

28 | 29 |

One of the key elements of boredom was suggested in "Complex Novelty": If your activity isn't teaching you insights you didn't already know, then it is non-novel, therefore old, therefore boring.

But this doesn't quite cover the distinction. Is breathing teaching you anything? Probably not at this moment, but you wouldn't want to stop breathing. Maybe you'd want to stop noticing your breathing, which you'll do as soon as I stop drawing your attention to it.

I'd suggest that the repetitive activities which are allowed to not be boring fall into two categories:

Things so extremely low-level, or with such a small volume of possibilities, that you couldn't avoid repeating them even if you tried; but which are required to support other non-boring activities. You know, like breathing, or obeying the laws of physics, or cell division - that sort of thing.
Things so high-level that their "stability" still implies an immense space of specific possibilities, yet which are tied up with our identity or our values. Like thinking, for example.

34 |

Let me talk about that second category:

Suppose you were unraveling the true laws of physics and discovering all sorts of neat stuff you hadn't known before... when suddenly you got bored with "changing your beliefs based on observation". You are sick of anything resembling "Bayesian updating" - it feels like playing the same video game over and over. Instead you decide to believe anything said on 4chan.

Or to put it another way, suppose that you were something like a sentient chessplayer - a sentient version of Deep Blue. Like a modern human, you have no introspective access to your own algorithms. Each chess game appears different - you play new opponents and steer into new positions, composing new strategies, avoiding new enemy gambits. You are content, and not at all bored; you never appear to yourself to be doing the same thing twice - it's a different chess game each time.

But now, suddenly, you gain access to, and understanding of, your own chess-playing program. Not just the raw code; you can monitor its execution. You can see that it's actually the same damn code, doing the same damn thing, over and over and over again. Run the same damn position evaluator. Run the same damn sorting algorithm to order the branches. Pick the top branch, again. Extend it one position forward, again. Call the same damn subroutine and start over.

I have a small unreasonable fear, somewhere in the back of my mind, that if I ever do fully understand the algorithms of intelligence, it will destroy all remaining novelty - no matter what new situation I encounter, I'll know I can solve it just by being intelligent, the same damn thing over and over. All novelty will be used up, all existence will become boring, the remaining differences no more important than shades of pixels in a video game. Other beings will go about in blissful unawareness, having been steered away from studying this forbidden cognitive science. But I, having already thrown myself on the grenade of AI, will face a choice between eternal boredom, or excision of my forbidden knowledge and all the memories leading up to it (thereby destroying my existence as Eliezer, more or less).

Now this, mind you, is not my predictive line of maximum probability. To understand abstractly what rough sort of work the brain is doing, doesn't let you monitor its detailed execution as a boring repetition. I already know about Bayesian updating, yet I haven't become bored 35 | with the act of learning. And a self-editing mind can quite reasonably 36 | exclude certain levels of introspection from boredom, just like 37 | breathing can be legitimately excluded from boredom. (Maybe these top-level cognitive algorithms ought also to be excluded from perception - if something is stable, why bother seeing it all the time?)

No, it's just a cute little nightmare, which I thought made a nice illustration of this proposed principle:

That the very top-level things (like Bayesian updating, or attaching value to sentient minds rather than paperclips) and the very low-level things (like breathing, or switching transistors) are the things we shouldn't get bored with. And the mid-level things between, are where we should seek novelty. (To a first approximation, the novel is the inverse of the learned; it's something with a learnable element not yet covered by previous insights.)

Now this is probably not exactly how our current emotional circuitry of boredom works. That, I expect, would be hardwired relative to various sensory-level definitions of predictability, surprisingness, repetition, attentional salience, and perceived effortfulness.

But this is Fun Theory, so we are mainly concerned with how boredom should work in the long run.

Humanity acquired boredom the same way as we acquired the rest of our emotions: the godshatter idiom whereby evolution's instrumental policies became our own terminal values, pursued for their own sake: sex is fun even if you use birth control. Evolved aliens might, or might not, acquire roughly the same boredom in roughly the same way.

Do not give into the temptation of universalizing anthropomorphic values, and think: "But any rational agent, regardless of its utility function, will face the exploration/exploitation tradeoff, and will therefore occasionally get bored with exploiting, and go exploring."

Our emotion of boredom is a way of exploring, but not the only way for an ideal optimizing agent.

The idea of a steady trickle of mid-level novelty is a human 38 | 39 | terminal value, not something we do for the sake of something else. 40 | Evolution might have originally given it to us in order to have us explore as well as exploit. But now we explore for its own sake. That 41 | steady trickle of novelty is a terminal value to us; it is not the most efficient instrumental method for exploring and exploiting.

Suppose you were dealing with something like an expected paperclip maximizer - something that might use quite complicated instrumental policies, but in the service of a utility function that we would regard as simple, with a single term compactly defined.

Then I would expect the exploration/exploitation tradeoff to go something like as follows: The paperclip maximizer would assign some resources to cognition that searched for more efficient ways to make paperclips, or harvest resources from stars. Other resources would be devoted to the actual harvesting and paperclip-making. (The paperclip-making might not start until after a long phase of harvesting.) At every point, the most efficient method yet discovered - for resource-harvesting, or paperclip-making - would be used, over and over and over again. It wouldn't be boring, just maximally instrumentally efficient.

In the beginning, lots of resources would go into preparing for efficient work over the rest of time. But as cognitive resources yielded diminishing returns in the abstract search for efficiency improvements, less and less time would be spent thinking, and more and more time spent creating paperclips. By whatever the most efficient known method, over and over and over again.

(Do human beings get less easily bored as we grow older, more tolerant of repetition, because any further discoveries are less valuable, because we have less time left to exploit them?)

If we run into aliens who don't share our version of boredom - a steady trickle of mid-level novelty as a terminal preference - then perhaps every alien throughout their civilization will just be playing the most exciting level of the most exciting video game ever discovered, over and over and over again. Maybe with nonsentient AIs taking on the drudgework of searching for a more exciting video game. After all, without an inherent preference for novelty, exploratory attempts will usually have less expected value than exploiting the best policy previously encountered. And that's if you explore by trial at all, as opposed to using more abstract and efficient thinking.

Or if the aliens are rendered non-bored by seeing pixels of a slightly different shade - if their definition of sameness is more specific than ours, and their boredom less general - then from our perspective, most of their civilization will be doing the human::same thing over and over again, and hence, be very human::boring.

Or maybe if the aliens have no fear of life becoming too simple and repetitive, they'll just collapse themselves into orgasmium.

And if our version of boredom is less strict than that of the aliens, maybe they'd take one look at one day in the life of one member of our civilization, and never bother looking at the rest of us. From our perspective, their civilization would be needlessly chaotic, and so entropic, lower in what we regard as quality; they wouldn't play the same game for long enough to get good at it.

But if our versions of boredom are similar enough - terminal preference for a stream of mid-level novelty defined relative to learning insights not previously possessed - then we might find our civilizations mutually worthy of tourism. Each new piece of alien art would strike us as lawfully creative, high-quality according to a recognizable criterion, yet not like the other art we've already seen.

It is one of the things that would make our two species ramen rather than varelse, to invoke the Hierarchy of Exclusion. And I've never seen anyone define those two terms well, including Orson Scott Card who invented them; but it might be something like "aliens you can get along with, versus aliens for which there is no reason to bother trying".

42 | 43 | 44 | 45 | 46 | 47 | 48 | Beware Detached Detail 49 | 50 | 51 | tag:typepad.com,2003:post-61531546 52 | 2009-01-17T21:15:00-05:00 53 | 2009-01-18T09:21:46-05:00 54 |

Yesterday I talked about how "social minds must both make good decisions, and present good images to others" and suggested "the near-far brain division can be handy when facing this problem; let the far system focus more on image, and...

55 | 56 | Robin Hanson 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 |

Yesterday I talked about how "social minds must both make good decisions, and present good images to others" and suggested "the near-far brain division can be handy when facing this problem; let the far system focus more on image, and the near system focus more on decisions." But I didn't follow this thought very far down the game tree. To an economist going down the game tree is like going down the rabbit hole; it shows us just how deep and strange are the underlying drivers of top behavior.

If our far thoughts are more distorted to present good images, then the next step down the rabbit hole is this: to judge how we will typically act, others should prefer to see our near thoughts, at least if they can distinguish near versus far thoughts. After all, near thoughts drive most day to day actions. And we should each look more to our own near thoughts to judge our own sincerity. 65 | 66 |

Once we evolved to weigh near others' thoughts more heavily, the next step would be to look for cheap ways to have good-looking near-thoughts, without paying the full price of distorting important actions. That is, our mind designer would look for ways to show "detached" near thoughts, consistent with good-image far-thoughts, but not actually impacting much on important near decisions. This could be accomplished by vivid engaging detail that can clearly occupy our near thought systems, but which isn't much connected to substantial personal decisions. 67 | 68 | 69 |

70 | 71 |

For example, abstract religious beliefs are usually tied to specific concrete, but largely irrelevant, details. The gods and prophets have names and birth-dates and stories of who did what when. Churches have pictures and idols people and events, and religious rituals are full of rich sensory details, like incense, plush materials, stained-glass windows, and angelic singing. Daily routines regarding what to eat and when to pray and what to wear all let you show how much near thought there is in your religious thought. Of course most of these details hardly matter to your important decisions; they let you assure yourself you will be a good Samaritan, without actually much influencing whether you be a good Samaritan. 72 | 73 |

Similarly, fiction lets us tie lots of concrete detail to our abstract social beliefs about what sorts of people are good or bad, who helps who how much, and which of them tend to win in the end. Stories presented as plays, movies, or video games have especially rich sensory detail, to give us experiences not just of abstracting having certain far reactions, but of having our near systems fully engaged and in agreement with our far expectations of what one should do in various situations. 74 | 75 |

A movie can let you feel not only that people in distant times and places should fight Nazis or free slaves, but that if you were actually in such situations with near systems fully engaged, you would actually do such things. But of course since the movie's scenario has little overlap with your real life, there is little risk that near habits acquired would interfere with your usual near actions. You rarely watch movies about, say, helping poor neighbors or illegal immigrants, since those stories are less detached from your ordinary life. There is a reason they call it "escapism," after all. 76 | 77 |

This detached detail dodge may explain why inside views seem worse at schedule forecasting than outside views. Yes inside-view schedulers consider more detail that engages near mental systems, but I suspect this is mostly detached detail, driven as with fiction detail more by far thought agendas than by consistency with relevant personal detail. Little girls may imagine their future wedding in vivid detail, without that being much constrained by actual weddings they have seen. 78 | 79 | 80 |

Don't assure yourself that your vivid near thoughts mean you are being more realistic, if those vivid thoughts are little connected with real personal experience. It might just be detached detail, there more to flesh out far thoughts than to advise important decisions. When you stand at a real street corner looking to see if it is safe to cross, that is attached detail; when you imagine being tempted by a grinning ten foot devil after fasting for weeks in the hot red desert, that is detached detail.

81 | 82 | 83 | 84 | 85 | 86 | Getting Nearer 87 | 88 | 89 | tag:typepad.com,2003:post-61505446 90 | 2009-01-17T04:28:54-05:00 91 | 2009-01-18T12:29:54-05:00 92 |

Reply to: A Tale Of Two TradeoffsI'm not comfortable with compliments of the direct, personal sort, the "Oh, you're such a nice person!" type stuff that nice people are able to say with a straight face. Even if it would...

93 | 94 | Eliezer Yudkowsky 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 |

Reply to: A Tale Of Two Tradeoffs

I'm not comfortable with compliments of the direct, personal sort, the "Oh, you're such a nice person!" type stuff that nice people are able to say with a straight face. Even if it would make people like me more - even if it's socially expected - I have trouble bringing myself to do it. So, when I say that I read Robin Hanson's "Tale of Two Tradeoffs", and then realized I would spend the rest of my mortal existence typing thought processes as "Near" or "Far", I hope this statement is received as a due substitute for any gushing compliments that a normal person would give at this point.

Among other things, this clears up a major puzzle that's been lingering in the back of my mind for a while now. Growing up as a rationalist, I was always telling myself to "Visualize!" or "Reason by simulation, not by analogy!" or "Use causal models, not similarity groups!" And those who ignored this principle seemed easy prey to blind enthusiasms, wherein one says that A is good because it is like B which is also good, and the like.

But later, I learned about the Outside View versus the Inside View, and that people asking "What rough class does this project fit into, and when did projects like this finish last time?" were much more accurate and much less optimistic than people who tried to visualize the when, where, and how of their projects. And this didn't seem to fit very well with my injunction to "Visualize!"

So now I think I understand what this principle was actually doing - it was keeping me in Near-side mode and away from Far-side thinking. And it's not that Near-side mode works so well in any absolute sense, but that Far-side mode is so much more pushed-on by ideology and wishful thinking, and so casual in accepting its conclusions (devoting less computing power before halting).

103 | 104 |

An example of this might be the balance between offensive and defensive 105 | nanotechnology, where I started out by - basically - just liking nanotechnology; until I got involved in a discussion about the particulars of nanowarfare, and noticed that people were postulating 106 | crazy things to make defense win. Which made me realize and say, "Look, the 107 | balance between offense and defense has been tilted toward offense ever 108 | since the invention of nuclear weapons, and military nanotech could use nuclear weapons, and I don't see how you're going to build a molecular barricade against that."

Are the particulars of that discussion likely to be, well, correct? Maybe not. But so long as I wasn't thinking of any particulars, my brain had free reign to just... import whatever affective valence the word "nanotechnology" had, and use that as a snap judgment of everything.

You can still be biased about particulars, of course. You can insist that nanotech couldn't possibly be radiation-hardened enough to manipulate U-235, which someone tried as a response (fyi: this is extremely silly). But in my case, at least, something about thinking in particulars...

...just snapped me out of the trance, somehow.

When you're thinking using very abstract categories - rough classes low on computing power - about things distant from you, then you're also - if Robin's hypothesis is correct - more subject to ideological bias. Together this implies you can cherry-pick those very loose categories to put X together with whatever "similar" Y is ideologically convenient, as in the old saw that "atheism is a religion" (and not playing tennis is a sport).

But the most frustrating part of all, is the casualness of it - the way that ideologically convenient Far thinking is just thrown together out of whatever ingredients come to hand. The ten-second dismissal of cryonics, without any attempt to visualize how much information is preserved by vitrification and could be retrieved by a molecular-level scan. Cryonics just gets casually, perceptually classified as "not scientifically verified" and tossed out the window. Or "what if you wake up in Dystopia?" and tossed out the window. Far thinking is casual - that's the most frustrating aspect about trying to argue with it. 109 | 110 |

This seems like an argument for writing fiction with lots of concrete details if you want people to take a subject seriously and think about it in a less biased way. This is not something I would have thought based on my previous view.

Maybe cryonics advocates really should focus on writing fiction stories that turn on the gory details of cryonics, or viscerally depict the regret of someone who didn't persuade their mother to sign up. (Or offering prizes to professionals who do the same; writing fiction is hard, writing SF is harder.)

But I'm worried that, for whatever reason, reading concrete fiction is a special case that doesn't work to get people to do Near-side thinking.

Or there are some people who are inspired to Near-side thinking by fiction, and only these can actually be helped by reading science fiction.

Maybe there are people who encounter big concrete detailed fictions process them in a Near way - the sort of people who notice plot holes. And others who just "take it all in stride", casually, so that however much concrete fictional "information" they encounter, they only process it using casual "Far" thinking. I wonder if this difference has more to do with upbringing or genetics. Either way, it may lie at the core of the partial yet statistically outstanding correlation between careful futurists and science fiction fans.

I expect I shall be thinking about this for a while.

111 | 112 | 113 | 114 | 115 | 116 | 117 | A Tale Of Two Tradeoffs 118 | 119 | 120 | tag:typepad.com,2003:post-61445194 121 | 2009-01-16T06:00:00-05:00 122 | 2009-01-18T09:46:26-05:00 123 |

The design of social minds involves two key tradeoffs, which interact in an important way. The first tradeoff is that social minds must both make good decisions, and present good images to others. Our thoughts influence both our actions and...

124 | 125 | Robin Hanson 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 |

The design of social minds involves two key tradeoffs, which interact in an important way.

134 | The first tradeoff is that social minds must both make good decisions, and present good images to others. Our thoughts influence both our actions and what others think of us. It would be expensive to maintain two separate minds for these two purposes, and even then we would have to maintain enough consistency to convince outsiders a good-image mind was in control. It is cheaper and simpler to just have one integrated mind whose thoughts are a compromise between these two ends.

135 | When possible, mind designers should want to adjust this decision-image tradeoff by context, depending on the relative importance of decisions versus images in each context. But it might be hard to find cheap effective heuristics saying when images or decisions matter more.

136 | The second key tradeoff is that minds must often think about the same sorts of things using different amounts of detail. Detailed representations tend to give more insight, but require more mental resources. In contrast, sparse representations require fewer resources, and make it easier to abstractly compare things to each other. For example, when reasoning about a room a photo takes more work to study but allows more attention to detail; a word description contains less info but can be processed more quickly, and allows more comparisons to similar rooms. 137 | 138 |

139 |

140 | It makes sense to have your mental models use more detail when what they model is closer to you in space and time, and closer to you in your social world; such things tend to be more important to you. It also makes sense to use more detail for real events over hypothetical ones, for high over low probability events, for trend deviations over trend following, and for thinking about how to do something over why to do it. So it makes sense to use detail thinking for "near", and sparse thinking for "far", in these ways.

141 | It can make sense to have specialized mental systems for these different approaches, i.e., systems best at reasoning from detailed representations, versus systems best at reasoning from sparse abstractions. When something became important enough to think about at all you would first use sparse systems, graduating to detail systems when that thing became important enough to justify the added resources. Even then you might continue to reason about it using sparse systems, at least if you could sufficiently coordinate the two kinds of systems.

142 | A non-social mind, caring only about good personal decisions, would want consistency between near and far thoughts. To be consistent, estimates made by sparse approaches should equal the average of estimates made when both sparse and detail approaches contribute. A social mind would also want such consistency when sparse and detail tasks had the same tradeoffs between decisions and images. But when these tradeoffs differ, inconsistency can be more attractive.

143 | The important interaction between these two key tradeoffs is this: near versus far seems to correlate reasonably well with when good decisions matter more, relative to good images. Decision consequences matter less for hypothetical, fictional, and low probability events. Social image matters more, relative to decision consequences, for opinions about what I should do in the distant future, or for what they or "we" should do now. Others care more about my basic goals than about how exactly I achieve them, and they care especially about my attitudes toward those people. Also, widely shared topics are better places to demonstrate mental abilities.

144 | Thus a good cheap heuristic seems to be that image matters more for "far" thoughts, relative to decisions mattering more for "near" thoughts. And so it makes sense for social minds to allow inconsistencies between near and far thinking systems. Instead of having both systems produce the same average estimates, it can make sense for sparse estimates to better achieve a good image, while detail estimates better achieve good decisions.

145 | And this seems to be just what the human mind does. The human mind seems to have different "near" and "far" mental systems, apparently implemented in distinct brain regions, for detail versus abstract reasoning. Activating one of these systems on a topic for any reason makes other activations of that system on that topic more likely; all near thinking tends to evoke other near thinking, while all far thinking tends to evoke other far thinking.

146 | These different human mental systems tend to be inconsistent in giving systematically different estimates to the same questions, and these inconsistencies seem too strong and patterned to be all accidental. Our concrete day-to-day decisions rely more on near thinking, while our professed basic values and social opinions, especially regarding fiction, rely more on far thinking. Near thinking better helps us work out complex details of how to actually get things done, while far thinking better presents our identity and values to others. Of course we aren't very aware of this hypocrisy, as that would undermine its purpose; so we habitually assume near and far thoughts are more consistent than they are.

147 | These near-far inconsistencies seems to me to reasonably explain puzzles like:

150 | we value particular foreign-born associates, but oppose foreign immigration
152 | we say we want to lose weight, but actually don't exercise more or eat less
154 | we say we care about distant future folk, but don't save money for them

156 |

157 | So which of near or far thinking is our "true" thinking? Perhaps neither; perhaps we really contain an essential contradiction, which we don't want to admit, much less resolve.

Added: The key puzzle I'm trying to address here is the fact that hypocrisy is hard. It is hard enough to manage a mind with coherent opinions across a wide range of topics. To manage two coherent systems of opinions, one for decisions and one for image, and then only let them differ where others can't see, that seems really hard. I'm saying the near-far brain division can be handy when facing this problem; let the far system focus more on image, and the near system focus more on decisions.

158 | 159 | 160 | 161 | 162 | 163 | 164 | Seduced by Imagination 165 | 166 | 167 | tag:typepad.com,2003:post-61450634 168 | 2009-01-15T22:10:22-05:00 169 | 2009-01-17T14:14:36-05:00 170 |

Previously in series: Justified Expectation of Pleasant Surprises"Vagueness" usually has a bad name in rationality - connoting skipped steps in reasoning and attempts to avoid falsification. But a rational view of the Future should be vague, because the information we...

171 | 172 | Eliezer Yudkowsky 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 |

Previously in series: Justified Expectation of Pleasant Surprises

"Vagueness" usually has a bad name in rationality - connoting skipped steps in reasoning and attempts to avoid falsification. But a rational view of the Future should be vague, because the information we have about the Future is weak. Yesterday I argued that justified vague hopes might also be better hedonically than specific foreknowledge - the power of pleasant surprises.

But there's also a more severe warning that I must deliver: It's not a good idea to dwell much on imagined pleasant futures, since you can't actually dwell in them. It can suck the emotional energy out of your actual, current, ongoing life.

Epistemically, we know the Past much more specifically than the Future. But also on emotional grounds, it's probably wiser to compare yourself to Earth's past, so you can see how far we've come, and how much better we're doing. Rather than comparing your life to an imagined future, and thinking about how awful you've got it Now.

Having set out to explain George Orwell's observation that no one can seem to write about a Utopia where anyone would want to live - having laid out the various Laws of Fun that I believe are being violated in these dreary Heavens - I am now explaining why you shouldn't apply this knowledge to invent an extremely seductive Utopia and write stories set there. That may suck out your soul like an emotional vacuum cleaner. 181 | 182 |

183 |

I briefly remarked on this phenomenon earlier, and someone said, "Define 'suck out your soul'." Well, it's mainly a tactile thing: you can practically feel the pulling sensation, if your dreams wander too far into the Future. It's like something out of H. P. Lovecraft: The Call of Eutopia. A professional hazard of having to stare out into vistas that humans were meant to gaze upon, and knowing a little too much about the lighter side of existence.

But for the record, I will now lay out the components of "soul-sucking", that you may recognize the bright abyss and steer your thoughts away:

Your emotional energy drains away into your imagination of Paradise:
- You find yourself thinking of it more and more often.
- The actual challenges of your current existence start to seem less interesting, less compelling; you think of them less and less.
- Comparing everything to your imagined perfect world heightens your annoyances and diminishes your pleasures.
190 |
You go into an affective death spiral around your imagined scenario; you're reluctant to admit anything bad could happen on your assumptions, and you find more and more nice things to say.
Your mind begins to forget the difference between fiction and real 193 | life:
- You originally made many arbitrary or iffy choices in constructing 196 | your scenario. You forget that the Future is actually more 197 | unpredictable than this, and that you made your choices using limited 198 | foresight and merely human optimizing ability.
- You forget that, in real life, at least some of your amazing good ideas are guaranteed not to work as well as they do in your imagination.
- You start wanting the exact specific Paradise you imagined, and worrying about the disappointment if you don't get that exact thing.
202 | 203 |

205 |

Hope can be a dangerous thing. And when you've just been hit hard - at the moment when you most need 206 | hope to keep you going - that's also when the real world seems most 207 | painful, and the world of imagination becomes most seductive.

It's a balancing act, I think. One needs enough Fun Theory to truly and legitimately justify hope in the future. But not a detailed vision so seductive that it steals emotional energy from the real life and real challenge of creating that future. You need "a light at the end of the secular rationalist tunnel" as Roko put it, but you don't want people to drift away from their bodies into that light.

So how much light is that, exactly? Ah, now that's the issue.

I'll start with a simple and genuine question: Is what I've already said, enough?

Is knowing the abstract fun theory and being able to pinpoint the exact flaws in previous flawed Utopias, enough to make you look forward to tomorrow? Is it enough to inspire a stronger will to live? To dispel worries about a long dark tea-time of the soul? Does it now seem - on a gut level - that if we could really build an AI and really shape it, the resulting future would be very much worth staying alive to see?

208 | 209 | 210 | 211 | 212 | 213 | 214 | Data On Fictional Lies 215 | 216 | 217 | tag:typepad.com,2003:post-61383844 218 | 2009-01-15T06:00:00-05:00 219 | 2009-01-17T12:57:58-05:00 220 |

221 | 222 | Robin Hanson 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 |

A speculator paper analyses a dataset of 519 Victorian literature experts describing 382 characters from 201 canonical British novels of the nineteenth century. Characters were described by gender, as major or minor, as good or bad, by role (protagonist, antagonist, friend of p, friend of a, or other), by a five factor personality type (from a ten-question instrument), as their (5-point-scale) degree of twelve different motives (converted to five factors: social dominance, constructive effort, romance, nurture, subsistence), and as the degree of ten different emotions they arouse in readers (converted to three factors: dislike, sorrow, interest). Experts agreed 87% of the time. They found:

231 |

Antagonists virtually personify Social Dominance - the self-interested pursuit of wealth, prestige, and power. In these novels, those ambitions are sharply segregated from prosocial and culturally acquisitive dispositions. Antagonists are not only selfish and unfriendly but also undisciplined, emotionally unstable, and intellectually dull. Protagonists, in contrast, display motive dispositions and personality traits that exemplify strong personal development and healthy social adjustment. Protagonists are agreeable, conscientious, emotionally stable, and open to experience. ... The male protagonists in this study are relatively moderate, mild characters. They are introverted and agreeable, and they do not seek to dominate others socially. They are pleasant and conscientious, and they are also curious and alert. They are attractive characters, but they are not very assertive or aggressive characters. ...
232 |

233 |

In Hierarchy in the Forest: The Evolution of Egalitarian Behavior, Boehm (1999)... argues that ... humans developed a special capacity, ... for enforcing moralistic or altruistic norms. By enforcing these norms, humans succeed in controlling "free riders" or "cheaters," and they thus make it possible for genuinely altruistic genes to survive within a social group. Such altruistic dispositions, enforced by punishing defectors, would enable social groups to compete more successfully against other groups and would thus make "group selection" or "multi-level selection" an effective force in subsequent human evolution. ...

If Boehm and others are correct, ... by derogating dominance and enacting the triumph of the communitarian ethos, ... agonistic structure in the novels would articulate real features of human nature, but like culture in general, the novels would exaggerate the magnitude of those features. Agonistic structure in these novels seems to serve as a medium for readers to participate vicariously in an egalitarian social ethos. ... as prosthetic extensions of social interactions that in non-literate cultures require face-to-face interaction. ...

234 | Could it not plausibly be argued that the novels merely depict social dynamics as they actually occur in the real world? If that were the case, one would have no reason to suppose that the novels mediate psychological processes in the community of readers. The novels might merely serve readers' need to gain realistic information about the larger patterns of social life. To assess the cogency of this challenge, consider the large-scale patterns revealed in the present study and ask whether those patterns plausibly reflect social reality: 235 | 236 |

The world is in reality divided into two main kinds of people. One kind is motivated exclusively by the desire for wealth, power, and prestige. These people have no affiliative dispositions whatsoever. Moreover, they are disagreeable, emotionally unstable, undisciplined, and narrow minded. The second kind of people, in contrast, have almost no desire for wealth, power, and prestige. They are animated by the purest and most self-forgetful dispositions for nurturing kin and helping non-kin. Moreover, they are agreeable, emotionally stable, conscientious, and open-minded. Life consists in a series of clear-cut confrontations between these two kinds of people. Fortunately, the second set almost always wins, and lives happily ever after. This is reality, and novels do nothing except depict this reality in a true and faithful way. 237 |

In our view, this alternative hypothesis fails of conviction. The novels do contain a vast fund of realistic social depiction and profound psychological analysis. In their larger imaginative structures, though, the novels evidently do not just represent human nature; they evoke certain impulses of human nature. Vicarious participation in the novel stirs up the reader's impulses to derogate dominance in others and to affirm one's identity as a positive, contributing member of his or her social group. It may not be too much of a leap to suggest that the emotional impulses aroused by the novel carry over when the novel is put down, actually encouraging people to suppress dominance and cooperate with others in real life.

This agrees a lot with William Flesch's Comeuppance, but like it focuses too much on group selection, instead of individual selection, pressures. As I suggested ten days ago,

Both religion and fiction serve to reassure our associates that we will 238 | be nice. In addition to letting us show we can do hard things, and 239 | that we are tied to associates by doing the same things, religious 240 | beliefs show we expect the not nice to be punished by supernatural 241 | powers, and our favorite fiction shows the sort of people we think are 242 | heroes and villains, how often they are revealed or get their due 243 | reward, and so on.

Hat tip to Fortune Elkins.

244 | 245 | 246 | 247 | 248 | 249 | 250 | Justified Expectation of Pleasant Surprises 251 | 252 | 253 | tag:typepad.com,2003:post-61388720 254 | 2009-01-15T02:26:51-05:00 255 | 2009-01-16T22:14:11-05:00 256 |

Previously in series: Eutopia is Scary I recently tried playing a computer game that made a major fun-theoretic error. (At least I strongly suspect it's an error, though they are game designers and I am not.)The game showed me -...

257 | 258 | Eliezer Yudkowsky 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 |

Previously in series: Eutopia is Scary 267 | 268 |

I recently tried playing a computer game that made a major fun-theoretic error. (At least I strongly suspect it's an error, though they are game designers and I am not.)

The game showed me - right from the start of play - what abilities I could purchase as I increased in level. Worse, there were many different choices; still worse, you had to pay a cost in fungible points to acquire them, making you feel like you were losing a resource... But today, I'd just like to focus on the problem of telling me, right at the start of the game, about all the nice things that might happen to me later.

I can't think of a good experimental result that backs this up; but I'd expect that a pleasant surprise would have a greater hedonic impact, than being told about the same gift in advance. Sure, the moment you were first told about the gift would be good news, a moment of pleasure in the moment of being told. But you wouldn't have the gift in hand at that moment, which limits the pleasure. And then you have to wait. And then when you finally get the gift - it's pleasant to go from not having it to having it, if you didn't wait too long; but a surprise would have a larger momentary impact, I would think.

This particular game had a status screen that showed all my future class abilities at the start of the game - inactive and dark but with full information still displayed. From a hedonic standpoint this seems like miserable fun theory. All the "good news" is lumped into a gigantic package; the items of news would have much greater impact if encountered separately. And then I have to wait a long time to actually acquire the abilities, so I get an extended period of comparing my current weak game-self to all the wonderful abilities I could have but don't.

Imagine living in two possible worlds. Both worlds are otherwise rich in challenge, novelty, and other aspects of Fun. In both worlds, you get smarter with age and acquire more abilities over time, so that your life is always getting better.

But in one world, the abilities that come with seniority are openly discussed, hence widely known; you know what you have to look forward to.

In the other world, anyone older than you will refuse to talk about certain aspects of growing up; you'll just have to wait and find out.

I ask you to contemplate - not just which world you might prefer to live in - but how much you might want to live in the second world, rather than the first. I would even say that the second world seems more alive; when I imagine living there, my imagined will to live feels stronger. I've got to stay alive to find out what happens next, right?

The idea that hope is important to a happy life, is hardly original with me - though I think it might not be emphasized quite enough, on the lists of things people are told they need.

I don't agree with buying lottery tickets, but I do think I understand why people do it. I remember the times in my life when I had more or less belief that things would improve - that they were heading up in the near-term or mid-term, close enough to anticipate. I'm having trouble describing how much of a difference it makes. Maybe I don't need to describe that difference, unless some of my readers have never had any light at the end of their tunnels, or some of my readers have never looked forward and seen darkness.

If existential angst comes from having at least one deep problem in your life that you aren't thinking about explicitly, so that the pain which comes from it seems like a natural permanent feature - then the very first question I'd ask, to identify a possible source of that problem, would be, "Do you expect your life to improve in the near or mid-term future?"

Sometimes I meet people who've been run over by life, in much the same way as being run over by a truck. Grand catastrophe isn't necessary to destroy a will to live. The extended absence of hope leaves the same sort of wreckage.

People need hope. I'm not the first to say it.

But I think that the importance of vague hope is underemphasized.

"Vague" is usually not a compliment among rationalists. Hear "vague hopes" and you immediately think of, say, an alternative medicine herbal profusion whose touted benefits are so conveniently unobservable (not to mention experimentally unverified) that people will buy it for anything and then refuse to admit it didn't work. You think of poorly worked-out plans with missing steps, or supernatural prophecies made carefully unfalsifiable, or fantasies of unearned riches, or...

But you know, generally speaking, our beliefs about the future should be vaguer than our beliefs about the past. We just know less about tomorrow than we do about yesterday.

There are plenty of bad reasons to be vague, all sorts of suspicious reasons to offer nonspecific predictions, but reversed stupidity is not intelligence: When you've eliminated all the ulterior motives for vagueness, your beliefs about the future should still be vague.

We don't know much about the future; let's hope that doesn't change for as long as human emotions stay what they are. Of all the poisoned gifts a big mind could give a small one, a walkthrough for the game has to be near the top of the list.

What we need to maintain our interest in life, is a justified expectation of pleasant surprises. (And yes, you can expect a surprise if you're not logically omniscient.) This excludes the herbal profusions, the poorly worked-out plans, and the supernatural. The best reason for this justified expectation is experience, that is, being pleasantly surprised on a frequent yet irregular basis. (If this isn't happening to you, please file a bug report with the appropriate authorities.)

Vague justifications for believing in a pleasant specific outcome would be the opposite.

There's also other dangers of having pleasant hopes that are too specific - even if justified, though more often they aren't - and I plan to talk about that in the next post.

269 | 270 | 271 | 272 | 273 | 274 | 275 | Disagreement Is Near-Far Bias 276 | 277 | 278 | tag:typepad.com,2003:post-61326928 279 | 2009-01-14T10:30:00-05:00 280 | 2009-01-16T15:21:19-05:00 281 |

Back in November I read this Science review by Nira Liberman and Yaacov Trope on their awkwardly-named "Construal level theory", and wrote a post I estimated "to be the most dense with useful info on identifying our biases I've ever...

282 | 283 | Robin Hanson 284 | 285 | 286 | 287 | 288 | 289 | 290 | 291 |

[NEAR] All of these bring each other more to mind: here, now, me, us; trend-deviating likely real local events; concrete, context-dependent, unstructured, detailed, goal-irrelevant incidental features; feasible safe acts; secondary local concerns; socially close folks with unstable traits.

293 | 294 |

[FAR] Conversely, all these bring each other more to mind: there, then, them; trend-following unlikely hypothetical global events; abstract, schematic, context-freer, core, coarse, goal-related features; desirable risk-taking acts, central global symbolic concerns, confident predictions, polarized evaluations, socially distant people with stable traits. 295 |

Since then I've become even more impressed with it, as it explains most biases I know and care about, including muddled thinking about economics and the future. For example, Ross's famous "fundamental attribution error" is a trivial application.

296 | The key idea is that when we consider the same thing from near versus far, different features become salient, leading our minds to different conclusions. This is now my best account of disagreement. We disagree because we explain our own conclusions via detailed context (e.g., arguments, analysis, and evidence), and others' conclusions via coarse stable traits (e.g., demographics, interests, biases). While we know abstractly that we also have stable relevant traits, and they have detailed context, we simply assume we have taken that into account, when we have in fact done no such thing.

297 | For example, imagine I am well-educated and you are not, and I argue for the value of education and you argue against it. I find it easy to dismiss your view as denigrating something you do not have, but I do not think it plausible I am mainly just celebrating something I do have. I can see all these detailed reasons for my belief, and I cannot easily see and appreciate your detailed reasons.

298 | And this is the key error: our minds often assure us that they have taken certain factors into account when they have done no such thing. I tell myself that of course I realize that I might be biased by my interests; I'm not that stupid. So I must have already taken that possible bias into account, and so my conclusion must be valid even after correcting for that bias. But in fact I haven't corrected for it much at all; I've just assumed that I did so.

299 | 300 | 301 | 302 | 303 | 304 | 305 | She has joined the Conspiracy 306 | 307 | 308 | tag:typepad.com,2003:post-61260822 309 | 2009-01-13T14:48:22-05:00 310 | 2009-01-15T18:05:38-05:00 311 |

I have no idea whether I had anything to do with this.

312 | 313 | Eliezer Yudkowsky 314 | 315 | 316 | 317 | 318 | 319 | 320 | 321 |

I have no idea whether I had anything to do with this.

322 | 323 | 324 | 325 | 326 | 327 | Building Weirdtopia 328 | 329 | 330 | tag:typepad.com,2003:post-61236322 331 | 2009-01-12T15:35:27-05:00 332 | 2009-01-16T18:22:49-05:00 333 |

Followup to: Eutopia is Scary"Two roads diverged in the woods. I took the one less traveled, and had to eat bugs until Park rangers rescued me." -- Jim Rosenberg Utopia and Dystopia have something in common: they both confirm the...

334 | 335 | Eliezer Yudkowsky 336 | 337 | 338 | 339 | 340 | 341 | 342 | 343 |

Followup to: Eutopia is Scary

"Two roads diverged in the woods. I took the one less traveled, and had to eat bugs until Park rangers rescued me."
-- Jim Rosenberg 344 | 345 |

Utopia and Dystopia have something in common: they both confirm the moral sensibilities you started with. Whether the world is a libertarian utopia of the non-initiation of violence and everyone free to start their own business, or a hellish dystopia of government regulation and intrusion - you might like to find yourself in the first, and hate to find yourself in the second; but either way you nod and say, "Guess I was right all along."
346 | 347 |

So as an exercise in creativity, try writing them down side by 348 | side: Utopia, Dystopia, and Weirdtopia. The zig, the zag and the zog.

I'll start off with a worked example for public understanding of science:

Utopia: Most people have the equivalent of an undergrad degree in something; everyone reads the popular science books (and they're good books); everyone over the age of nine understands evolutionary theory and Newtonian physics; scientists who make major contributions are publicly adulated like rock stars.
Dystopia: Science is considered boring and possibly treasonous; public discourse elevates religion or crackpot theories; stem cell research is banned.
Weirdtopia: Science is kept secret to avoid spoiling the surprises; no public discussion but intense private pursuit; cooperative ventures surrounded by fearsome initiation rituals because that's what it takes for people to feel like they've actually learned a Secret of the Universe and be satisfied; someone you meet may only know extremely basic science, but they'll have personally done revolutionary-level work in it, just like you. Too bad you can't compare notes.

355 | 356 |

Disclaimer 1: Not every sensibility we have is necessarily wrong. Originality is a goal of literature, not science; sometimes it's better to be right than to be new. But there are also such things as cached thoughts. At least in my own case, it turned out that trying to invent a world that went outside my pre-existing sensibilities, did me a world of good.

Disclaimer 2: This method is not universal: Not all interesting ideas fit this mold, and not all ideas that fit this mold are good ones. Still, it seems like an interesting technique.

If you're trying to write science fiction (where originality is a legitimate goal), then you can write down anything nonobvious for Weirdtopia, and you're done.

If you're trying to do Fun Theory, you have to come up with a Weirdtopia that's at least arguably-better than Utopia. This is harder but also directs you to more interesting regions of the answer space.

If you can make all your answers coherent with each other, you'll have quite a story setting on your hands. (Hope you know how to handle characterization, dialogue, description, conflict, and all that other stuff.)

Here's some partially completed challenges, where I wrote down a Utopia and a Dystopia (according to the moral sensibilities I started with before I did this exercise), but inventing a (better) Weirdtopia is left to the reader.

Economic...

Utopia: The world is flat and ultra-efficient. Prices fall as standards of living rise, thanks to economies of scale. Anyone can easily start their own business and most people do. Everything is done in the right place by the right person under Ricardo's Law of Comparative Advantage. Shocks are efficiently absorbed by the risk capital that insured them.
Dystopia: Lots of trade barriers and subsidies; corporations exploit the regulatory systems to create new barriers to entry; dysfunctional financial systems with poor incentives and lots of unproductive investments; rampant agent failures and systemic vulnerabilities; standards of living flat or dropping.
Weirdtopia: _____

363 |

Sexual...

Utopia: Sexual mores straight out of a Spider Robinson novel: Sexual jealousy has been eliminated; no one is embarrassed about what turns them on; universal tolerance and respect; everyone is bisexual, poly, and a switch; total equality between the sexes; no one would look askance on sex in public any more than eating in public, so long as the participants cleaned up after themselves.
Dystopia: 10% of women have never had an orgasm. States adopt laws to ban gay marriage. Prostitution illegal.
Weirdtopia: _____

369 |

Governmental...

Utopia: Non-initiation of violence is the chief rule. Remaining public issues are settled by democracy: Well reasoned public debate in which all sides get a free voice, followed by direct or representative majority vote. Smoothly interfunctioning Privately Produced Law, which coordinate to enforce a very few global rules like "no slavery".
Dystopia: Tyranny of a single individual or oligarchy. Politicians with effective locks on power thanks to corrupted electronic voting systems, voter intimidation, voting systems designed to create coordination problems. Business of government is unpleasant and not very competitive; hard to move from one region to another.
Weirdtopia: _____

375 |

Technological...

Utopia: All Kurzweilian prophecies come true simultaneously. Every pot contains a chicken, a nanomedical package, a personal spaceship, a superdupercomputer, amazing video games, and a pet AI to help you use it all, plus a pony. Everything is designed by Apple.
Dystopia: Those damned fools in the government banned everything more complicated than a lawnmower, and we couldn't use our lawnmowers after Peak Oil hit.
Weirdtopia: _____

382 |

Cognitive...

Utopia: Brain-computer implants for everyone! You can do whatever you like with them, it's all voluntary and the dangerous buttons are clearly labeled. There are AIs around that are way more powerful than you; but they don't hurt you unless you ask to be hurt, sign an informed consent release form and click "Yes" three times.
Dystopia: The first self-improving AI was poorly designed, everyone's dead and the universe is being turned into paperclips. Or the augmented humans hate the normals. Or augmentations make you go nuts. Or the darned government banned everything again, and people are still getting Alzheimers due to lack of stem-cell research.
Weirdtopia: _____

388 |

389 | 390 | 391 | 392 | 393 | 394 | 395 |