A scene I once saw

(inaccurately recounted)

Ms. Knox: When any of you feels ready, you can move in to the center of the circle, hold the stone, and tell us all about your feelings about what we are doing. Listen to the trees moving, encouraging you.

Sarah: I feel really proud. Young people are so passionate about the environment. Everyone will have to believe in us when they see how much we care.

Amanda: Excited! I feel like we are going to be part of a really positive change, across the world. It’s so great to be here now, when this is happening.

Marie: I’m just really glad to be here with so many likeminded people. When nobody around you sees what’s possible, it can be really disillusioning, but here I feel like everyone cares so much.

Linda: I feel really inspired by what the others are saying!

Becky: I’m so hopeful when I see all this engagement. I believe we can all stay passionate and keep the movement going until we are old, and inspire the new youth!

Odette: Irritated! I have so many things I would enjoy doing more than saving the environment, both this weekend and for the rest of my life. Preventing ecological catastrophe is very important, but I’d obviously much much prefer that someone else had done it already, or that it never needed doing. It’s extremely disappointing that after this many generations nobody’s got around to the most obvious solutions like taxing the big externalities. These things are not even interesting to think about. In a perfect world it would be nice to play video games most of the time, but I’m at least as frustrated that I won’t even get to work on the interesting altruistic endeavors.

***

Why is this so rare?

More convergent instrumental goals

Nick Bostrom follows Steve Omohundro in exploring the types of instrumental goals that intelligences with arbitrary ultimate goals might converge on. This is important for two reasons. First, it means predicting the behavior of arbitrary intelligences might be a tiny bit easier than you’d think. Second, because it draws attention to the difficulty of creating a creature that doesn’t want to get mixed up in taking resources and seeking longevity and that sort of thing.

Between Nick and Steve we have these convergent instrumental goals:

  1. Self-preservation
  2. Preserve your values
  3. self-improvement
  4. rationality
  5. other cognitive enhancement
  6. technological perfection
  7. get resources
  8. Avoid counterfeit utility

I think acquiring information is included in cognitive enhancement here, though to me it seems big and different enough that I’d put it by itself.

I’d like to add three more, incidentally all to do with interacting with other intelligences. So not relevant if you are the one all powerful singleton, but  handy if there are a number of AIs which are of a similar enough level that they might be useful to one another (like sheep and humans are, or humans and humans are). Here’s my list:

Spread your values
This can be achieved by making other entities which have at least some of the same values, or more traditionally by buying advertising space, exercising social influence, or having children. This makes sense for the same reason that ensuring that you maintain your values in the future does. If you think of yourself as just your values (as Bostrom mentions might be appropriate for some kinds of intelligences), spreading values can come under the heading of self-preservation perhaps (or reproduction). But usually we don’t think like this. If I convince you that Bach is pretty good, we don’t think of this as me replacing part of you with part of me. So I think it’s worth having a different category.

Social influence
If you can’t get others to have your values, the next best thing is to get them to do what you want anyway. This can be done in many ways, e.g. deceit, deals, forming relationships with useful people, sending good signals, being likable. Perhaps manipulating others comes under perfection of technology: making friends is in a way like better programming your computer to do what you type in. Both can be seen as investing some effort in arranging a complex system such that in future you will get what you want from it with less input. But not that close, so I give these things a new category. The extent to which it is worth influencing other intelligences depends on how useful they are. If the world only had me and sheep and stupider things in it, influencing sheep would be handy, but not nearly as handy as social skills are to me in this world. The next can be thought of as a subcategory of this one.

Acquire commonly valued goods
Nick and Steve mention acquiring ‘resources’, but from their elaboration it seems like they mean goods that are direct inputs to the intelligence’s goals. An instrumental goal to acquiring directly useful resources is acquiring stuff that is valuable to someone else. This apparently roundabout route is common because of comparative advantage. So while an AI would presumably like to have energy and fast hardware, it would also like a yacht or intellectual property rights for a new kind of frozen dinner, assuming anyone exists who will trade. With few assumptions, acquiring something like the latter will be easier than acquiring the former directly. This is pretty much what humans spend their working lives on.

Updated blog summary

I have updated the summary of every post on my blog, to include all the posts since last time, more than a year ago.

Epistemology of evilness

Most everyone seems to think that a big reason for bad things happening in the world is that some people are bad. Yet I almost never see advice for telling whether you yourself are a bad person, or for what to do about it if you seem to be one. If there are so many bad people, isn’t there a very real risk that you are one of them?

Perhaps the model is one where you automatically know whether you are good or bad, and simply choose which to be. So the only people who are bad are those who want to be bad, and know that they are bad. But then if there is this big population of bad people out there who want to be bad, why is so little of the media devoted to their interests? There’s plenty on how to do all the good things that a good person would want to do, such as voting for the benefit of society, looking after your children, buying gifts, expressing gratitude to friends, holding a respectable dinner, pleasing your partner. Yet so little on scamming the elderly, effectively shaking off useless relatives, lying credibly, making money from investments that others are too squeamish to take, hiding bodies. Are the profit-driven corporate media missing out on a huge opportunity?

If there aren’t a whole lot of knowingly bad people out there who want to be bad, and could use some information and encouragement, then either there aren’t bad people at all, or bad people don’t know that they are bad or don’t want to be bad. The former seems unlikely, by most meanings of ‘bad’. If the latter is true, why are people so blase about the possibility that they themselves might be bad?

***

Prompted by the excellent book Harry Potter and the Methods of Rationality, in which there is much talk of avoiding becoming ‘dark’, in stark contrast to the world that I’m familiar with. If you enjoy talking about HPMOR, and live close to Pittsburgh, come to the next Pittsburgh Less Wrong Meetup.

Value realism

People have different ideas about how valuable things are. Before I was about fifteen the meaning of this was ambiguous. I think I assumed that a tree for instance has some inherent value, and that when one person wants to cut it down and another wants to protect it, they both have messy estimates of what its true value is. At least one of them had to be wrong. This was understandable because value was vague or hard to get at or something.

In my year 11 Environmental Science class it finally clicked that there wasn’t anything more to value than those ‘estimates’.  That a tree has some value to an environmentalist, and a different value to a clearfelling proponent. That it doesn’t have a real objective value somewhere inside it. Not even a vague or hard to know value that is estimated by different people’s ‘opinions’. That there is just nothing there. That even if there is something there, there is no way for me to know about it, so the values I deal with every day can’t be that sort. Value had to be a function of things: the item being valued and the person doing the valuing.

I was somewhat embarrassed to have ever assumed otherwise, and didn’t really think about it again until recently.  It occurred to me recently that a long list of strange things I notice people believing can be explained by the assumption that they disagree with me on whether things have objective values. So I hypothesize that many people believe that value is inherent in a thing, and doesn’t intrinsically depend on the agent doing the valuing.

Here’s my list of strange things people seem to believe. For each I give two explanations: why it is false, and why it is true if you believe in objective values. Note that these are generally beliefs that cause substantial harm:

When two people trade, one of them is almost certainly losing

People don’t necessarily say this explicitly, but often seem to implicitly believe it.

Why it’s false: In most cases where two people are willing to trade, this is because the values they assign to the items in question are such that both will gain by having the other person’s item instead of their own.

Why it’s believed: There’s a total amount of value shared somehow between the people’s posessions. Changing the distribution is very likely to harm one party or the other. It follows that people who engage in trade are suspicious, since trades must be mostly characterized by one party exploiting or fooling another.

Trade is often exploitative

Why it’s false: Assume exploiting someone implies making their life worse on net. Then in the cases where trade is exploitative, the exploited party will decline to participate, unless they don’t realize they are being exploited. Probably people sometimes don’t realize they are being exploited, but one is unlikely to persist in doing a job which makes one’s life actively worse for long without noticing. Free choice is a filter: it causes people who would benefit from an activity to do it while people who would not benefit do not.

Why it’s believed: If a person is desperate he might sell his labor for instance at a price below its true value. Since he is forced by circumstance to trade something more valuable for something less valuable, he is effectively robbed.

Prostitution etc should be prevented, because most people wouldn’t want to do it freely, so it must be pushed on those who do it:

Why it’s false: Again, free choice is a filter. The people who choose to do these things presumably find them better than their alternatives.

Why it’s believed: If most people wouldn’t be prostitutes, it follows that it is probably quite bad. If a small number of people do want to be prostitutes, they are probably wrong. The alternative is that they are correct, and the rest of society is wrong. It is less likely that a small number of people is correct than a large number. Since these people are wrong, and their being wrong will harm them (most people would really hate to be prostitutes), it is good to prevent them acting on either their false value estimates.

If being forced to do X is dreadful, X shouldn’t be allowed:

Why it’s false: Again, choice is a filter. For an arbitrary person doing X, it might terrible, but it is still often good for people who want it. Plus, being forced to do a thing often decreases its value.

Why it’s believed: Very similar to above. The value of X remains the same, regardless of who is thinking about it, whether they are forced to do it. That a person would choose to do a thing others are horrified to have pressed on them, that just indicates that the person is mentally dysfunctional in some way.

Being rich indicates that you are evil:

Why it’s false: On a simple model, most trades benefit both parties, so being rich indicates that you have contributed to others receiving a large amount of value.

Why it’s believed: On a value realism model, in every trade, someone wins and someone loses, anyone who has won at trading so many times is evidently an untrustworthy and manipulative character.

Poor countries are poor because rich countries are rich:

Why it’s false: In some sense it’s true—the rich countries don’t altruistically send a lot of aid into the poor countries. Beyond that there’s no obvious connection.

Why it’s believed: There’s a total amount of value to be had in the world. The poor can’t become richer without the rich giving up some value.

The primary result of promotion of products is that people buy things they don’t really want:

Why it’s not obviously true: The value of products depends on how people feel about them, so it is possible to create value by changing how people feel about products.

Why it’s believed: Products have a fixed value. Changing your perception of this in the direction of you buying more of them is cheatful sophistry.

***

Questions:

Is my hypothesis right? Do you think of value as a one or two place function? (Or more?) Which of the above beliefs do you hold? Are there legitimate or respectable cases for value realism out there? (Moral realism is arguably a subset).