Category Archives: 1

Crossposted from world spirit sock puppet.

I’ve really wanted more good short accounts of why AI poses an existential risk. Working on one myself has been one of those incredibly high priorities I keep putting off.

Meanwhile award-winning journalist Ben Bradford of NPR has made a podcast version of my case for AI x-risk that I am thrilled with!

(Bonus within the 39 minutes: what Hamza Chaudhry of FLI thinks we should do about it—who I was delighted to later meet as a consequence!)

If you or anyone you know could do with a quick and gripping rundown of why this is a problem, try this one.

Get it on any podcasting app here: https://pod.link/1893359212

The NPR press release has more context on the rest of the series, assessing different possible sources of doom.

Crossposted from world spirit sock puppet.

Some things you might learn from games are pretty blatant: Trivial Pursuit might teach you trivia, MasterType might teach you about typing, Grand Theft Auto might teach you about driving or crime.

But sometimes games teach people less obvious things—things that are more experiential or ineffable, things that you didn’t know you didn’t know, concepts that stick in your mind, deep things. Here’s my list of games and their interesting real-world updates, as experienced by me or my friends:

Dominion: Don’t invest for eternity. When casually improving or protecting or investing in things, it’s easy for me to treat life (and perhaps even the present period) as basically eternal. In fact I shouldn’t, but it can take many years of living to really feel how likely it is that you’ll leave your perfectly wonderful house within two years, or just keep on aging. Dominion lets me feel that in a matter of hours, by tempting me to invest in a beautiful and effective deck that will do amazingly for the rest of eternity, then making the other player win by haphazardly buying a handful of provinces before I’m done. Which is very annoying, and I do hold against it.

**The Witness: **there is nothing in The Witness (at least near the start, I haven’t played it all) that you can pick up and take with you. No objects, no points, no manna, no health. It’s just you, walking around in a world. Something about that feels like it would be deeply unsatisfying—like what is a game, if you can’t get, y’know, things, dings? Part of me thinks that GETTING is equivalent to satisfaction, in spite of all the evidence to the contrary I keep pointing out to it. And The Witness is not where I came to realize that. What The Witness made me feel is that knowledge is a REAL thing you can GET, like an object. Not some hand-wavey second-rate bullshit thing that philosophers pretend to get off on. In The Witness, while your character walks around, impermeable to the world, you come to know more things. And knowing more things lets you go to places you couldn’t go to when you knew fewer things. The game on the computer concretely changes from you picking up knowledge, that ethereal thing in your mind. This is of course how everything is, but I suppose the absence of any other form of ‘picking up things’ in The Witness made me actually feel it.

**Minecraft: **How many of my difficulties in life are not this-life specific. How to live as a creature with different boundaries of personal-identity, e.g. the world spirit. Much more about these in my previous post, Mine-craft.

Return of the Obra Dinn: If at an event where lots of people are saying their name and what they do or something, I am usually bored and don’t expect to remember these things. Return of the Obra Dinn is a game where you have to figure out from minute clues the names and causes of death of a lot of characters. Once at a networking event, I decided to think of it as like a sequel of Return of the Obra Dinn—I could see all these people sitting around the table, and my quest was to pin a name and a deal to each of them, and this introductory section was currently showing me crucial information. I found that this was a very different mental state. So I suppose I learned that whatever I was normally doing in ‘trying to learn’ things about the other attendees, it is an extremely pale cousin of the curiosity I can feel in a different mental state, and that different mental state is actually fairly different, and naturally invoked in RotOD and not networking introductions.

**Dungeons and Dragons: **Caitlin Elizondo says DnD has given her a few concepts that make a difference to her thinking more generally. The concept of ‘will saves’ has given her more empathy for situations where someone wanted to but failed to do something. The six DnD stats helps her access the framework where there are different types of competency that are valuable for different tasks—obvious in theory, but easier to think in terms of with this structure.

**Poker: **the feeling of being ‘on tilt’

Boggle, Set, Ragnarock: the feeling of flow. Ragnarock is mine, and I would have said I’d experienced ‘flow’ elsewhere, but Ragnarock is sometimes more like an altered state than other such experiences I’ve had.

Civilization IV: I used to lose at a scenario then go back and play it again over and over changing things slightly until I won, which gave me a vivid sense of how suboptimal my native strategy is, presumably also in life. Which is obvious in theory, but it’s different to really feel how much better I would live this day if I was doing it the twentieth time with a laser focus on winning.

Games in general: the experience of addiction, sadly. I’ve always struggled to keep up habits of taking addictive substances, so I infer I’m unusually safe from chemical addictions (I used to play Civilization for five minutes as a reward if I remembered to take my amphetamines). Games are I think the thing I find most seriously addictive. Which has definite downsides, but it is certainly also an interesting experience that helps me understand the wider world better, and where I would be missing something if I just read about addiction in the abstract.

Do you have any to add?

[ETA May 1: I’m adding more I hear in the above list, and also see many good additions in the comments!]

Crossposted from world spirit sock puppet.

often worry that people are being too deferential about their beliefs. I also hear others worrying about this, and nobody seemingly worrying about the reverse, except perhaps my friends and therapists (and I guess honestly people who know cranks, so that’s a bit troubling).

Which leads me to wonder, supposing it’s true that many people are too deferential, what might people do to change it? And can I offer them useful advice, as a person who might be not deferential enough?

Tonight I talked to Joe Carlsmith about this; here are some ideas mostly from the conversation:

1. A thing that has discouraged me from having independent views and broadcasting them is the concern that my views are extremely ignorant. At the normal pace of new information appearing, I am just too slow a reader to be acceptably up on it. At the AI-news rate, it’s very hopeless. And even if you recognize that situation at a high level, it can be easy to get to thinking ‘I want to write something about W, but I’ll need to read X, Y and the sequence on Z and all the responses to it first’.

It was helpful to me to give up on this kind of expectation, and accept that I’m going to be ignorant and have views anyway. I think this is the right thing to do because a) everyone is fairly ignorant and we don’t want the public discussion to be only the few people who don’t realize they are ignorant or care, and b) saying what you guess is true then letting people point you to why you are wrong is often more efficient than scouring all writing on the topic, and c) there’s value from more independent thinking on a topic, and being informed comes with being less independent. Bringing us to—

1. Being sufficiently out of the loop can actually help, as long as you are bold enough not to be silenced by this—if you don’t know what others’ views are, you have to come up with your own.
2. Focus on having your own beliefs at a relatively high level. For instance, “Shouldn’t we be stopping AI though?.. Wait, does that argument make sense?” is the kind of thing you can think about and discuss reasonably well without needing to know a lot of technical or fast-moving details, until a more manageable few are brought up in the argument. And my sense is that these kinds of questions—e.g. is our basic strategy what it should be?—are actually neglected.
3. Which brings us to status. Intellectual deference probably follows normal patterns of status-based deference. So it probably helps to be either high status or arrogant. That’s a lot of effort, but you can have the experience of being high status or arrogant by talking to people who are relatively lower status or deferential, such as children.
4. It probably helps to be brought up in a situation where you learned to distrust the thinking of everyone around you. It’s probably ideal to be taught by your parents that everyone else around is an idiot, then to come to distrust your parents opinions also.
5. That’s hard to get later in life. But perhaps you can get something similar from experiencing apparently venerable intellects confidently asserting things, then later observing them to be false.
6. If you are in conversations where it seems like the other person isn’t making sense, try to assume that is what’s going on, rather than the potentially much more salient explanation that you are a fool.
7. Give esteem to people asking potentially silly questions. It can help to expose yourself to impressive people who do this.

Niels Bohr quotes are helpful (HT Wikiquote)

1. Refuse to ‘understand’ things unless they are very clear. I don’t really know how to do this, because I don’t know what the alternative is like—being steadfastly confused about things seems to come naturally to me and I don’t know how else to be, but maybe you have both affordances available here and could lean one way or the other.
2. Something something do real thinking versus fake thinking. Ironically, this point I am deferring on, because I haven’t finished listening to Joe’s (so far very interesting) post.
3. If you are going to pass on information that you don’t deeply understand, track that it is a different thing, for instance by saying ‘something something…’

Crossposted from world spirit sock puppet.

A common view around me seems to be that journalists are frequently dishonorable and dangerous, and talking to them is a risk to be avoided unless you have a very specific piece of information that you seek to publicize. Then you should carefully ensure that you are as off the record as practical, and prepare to aggressively pivot the topic back to your agenda.

My own attitude is different: journalists are to be talked to as much as possible, and ideally in a relaxed fashion. If a journalist wants to observe you in some unusual circumstance, say yes. Don’t have an agenda much more than in the rest of life; basically listen to their questions and say what you think. (Note: I don’t have strong reason to believe this is safe for others or even for me.)

As evidence of the commitment with which I act in this way, this New Yorker piece describes me as ‘an oversharer’, before detailing some of my incompetent and substance-involving preparations for a dinner party at my house. (To be clear, I consider that accurate and agreeable coverage.)

I’ve talked to a lot of journalists, so how do I survive such recklessness? Well, in my experience, journalists are usually delightful: smart, sincere people trying hard to understand important topics and convey them to the public. And I’m impressed by how well they do. When I meet them, they are usually spending a tiny number of days diving into a complicated and wild situation, and while their summary isn’t perfect, I think it’s pretty good!

And what they write about me rarely feels adversarial. I can think of maybe two cases where it felt unfair or unfriendly, though there might be more I’m forgetting. And meanwhile there’s also the upside chance of them writing whole articles explaining your ideas to a large audience.

How is it that I and other people can have such divergent views here? Policies for interacting with journalists are hardly abstract philosophy—the people with the other views presumably also get to repeatedly experience talking with journalists.

One theory is that we have different expectations. What seems to me like imperfect summarization maybe seems to others like lies. What seems to me like accurate summarization maybe seems to others like ‘making me look weird’. I’ve noticed some weird-to-me expectations about the nature and constraints of journalism—like, thinking it would be better if a New Yorker article was about a technical paper and didn’t contain personal anecdotes. These expectations seems wrong about where the value of such an article comes from.

Another possibility is that we have different risk tolerances. The same low (but real) risk of a journalist writing a hit piece or randomly otherwise misusing their powers might be a deal-breaker for someone else, while I’m not very emotionally troubled by it. (My feelings: “You’re saying they could lie about me? That seems like a them problem? Can’t they lie about me anyway? Is my input that helpful? I’ll just say they lied about me, and we can have a public disagreement about it, and then maybe other journalists will come to talk to me.”) If I had more information I really needed to hide, this might be different.

Another class of theories is that we are different people, and either journalists treat us differently, or we come in contact with different journalists.

For instance, maybe people trying to learn about expert surveys of AI researchers tend to be in a cooperative mindset. But I’ve talked to journalists about my dating life, AI destroying the world, and all manner of other AI-related questions (as well as about accidentally getting into a physical altercation with nationalistically enthusiastic protesters, and probably oleander trees, and I bet some other stuff, but it’s less clear I would remember if I didn’t like the coverage in those cases.)

Someone suggested the other day that maybe I’m just a likable person, or likable to journalists. Similarly, my demeanor might just discourage being an asshole to me somehow—perhaps I seem unusually cooperative or too naive to be taken advantage of without feeling bad.

Perhaps journalists are responding to these different attitudes themselves. Probably journalists are like everyone else: they can somewhat tell if you are on their side or seeing them as an enemy to be thwarted and steered and then escaped from unscathed. And perhaps seeing the latter causes them to suspect you and find it more likely that you are weird and troubling, and shouldn’t have your words quoted without qualification or your behavior described uncritically.

If I recall, I have basically always had the attitude I have here, and it has only been strengthened by my experience talking to maybe hundreds of journalists. I think I’m also attracted to this attitude beyond pragmatism. So if journalists were responding to the attitudes, I would have had a different experience from the start.

I don’t really know what to make of this. I feel reasonably good about my policies for myself, but I don’t know if I can recommend them, because I don’t understand why they go fine for me. I can still record what I’ve experienced here, and send it out for other people to see. Which is maybe what talking to journalists is all about.

Crossposted from world spirit sock puppet.

Suppose you and Bobby the car salesman are haggling over the price of a car. You could try saying that you won’t pay more than $3k, but Bobby can equally retort that he won’t sell it for less than $4k. If you guys manage to negotiate a sale, it will probably be at more than $3k (and involve revealing both of you as liars).

Now imagine the same situation, but you only have $3k and Bobby knows it. Now, if $3k is actually ok for him, you win and get your price.

Now imagine you are rich but you have a boyfriend at home who has only agreed to a $3k expenditure on a used car at this time, and thinks any more would be crazy. It’s shared money, so to pay more you would need to go away and get his permission, and it wouldn’t be easy. If Bobby believes you, then your situation is much like being poor again, and you win.

My guess is I read about this in Thomas Schelling’s The Strategy of Conflict when I was a teenager. The general observation is that being more constrained can often be helpful in a negotiation. Which is a bit shocking because it undermines the seeming truism that more power—more options, more resources—is always better for getting what you want.

A less general observation that also stuck with me about this is that you can trivially arrange to have such constraints through having an associate, such as a stubborn and spending-conscious boyfriend. (Ok, finding one of those is not trivial, especially if you have other desiderata.)

This is all background. The thing I want to point out is that being part of an organization rather than a free agent means creating and using this effect all over the place.

This is most obvious with timing and deadlines. I am a relatively free agent, and I am quite good at making deadlines for myself and then taking them seriously. But I feel like other people I casually negotiate with about how to spend time, aka my friends, often feel like deadlines I make are not very real, since I could just ignore them. Because it’s just an agreement with myself, it’s up for negotiation with myself. And if I insist on respecting these lines I drew myself that have no legible consequences, then it feels like I’m being weird and stubborn and unfriendly or perhaps charmingly neurodivergent. So I often don’t—once it’s a negotiation, then negotiating hard for my own goals, against my friends, doesn’t feel very friendly to anyone.

Now consider a friend working in an org. They can casually throw out that they have this thing due tomorrow, and everyone will take it as a hard constraint. I will take it as a hard constraint. I might even offer to help get it done, even though I have other things I want to do. Whereas if I had not only insisted on my imaginary deadline but hoped for any help in fulfilling it, I think that would often feel unreasonable of me.

The org believably cuts off the person’s options, like the boyfriend, and so the person implicitly wins many negotiations (or what would have been negotiations), all in the direction of doing more for the org, and without seeming unfriendly to their friends.

My own difficulties with this are partly a me problem—I’m probably not very good at ‘defending boundaries’. But my point is that if you are a solo human then there’s a whole skill-requiring task of ‘defending boundaries’ that just becomes trivially easy if you have an org around you to cut off certain possibilities. And also if your boundaries are ‘I am going to do this project tonight definitely regardless of if you want me to do something else’ then that will land a way with other people that reporting on your org’s boundary policing—‘I have to do this by tomorrow, alas’—will not.

I think this ‘service’ and making use of it is rarely intentional, but I’d guess it’s very effective, and is a dynamic that makes people more likely to join orgs rather than being solo. It just looks like ‘it’s harder to get things done on my own’ and a component of ‘it’s harder to structure my time’ and ‘I find I keep on doing stuff other than my work’.