If artificial intelligence was about to become ‘human-level’, do you think we (society) would get advance notice? Would artificial intelligence researchers have been talking about it for years? Would tell-tale precursor technologies have triggered the alert? Would it be completely unsurprising, because AI’s had been able to do almost everything that humans could do for a decade, and catching up at a steady pace?
Whether we would be surprised then seems to make a big difference to what we should do now. Suppose that there are things someone should do before human-level AI appears (a premise to most current efforts to mitigate AI impacts). If there will be a period in which many people anticipate human-level AI soon, then probably someone will do the highest priority things. If you try to do them now, you might replace them or just fail because it is hard to see what needs doing so far ahead of time. So if you think AI will not be surprising, then the best things to do regarding AI now will tend to be the high value things which require a longer lead time. This might include building better institutions and capabilities; shifting AI research trajectories; doing technical work that is hard to parallelize; or looking for ways to get the clear warning earlier.
Anders Sandberg has put some thought into warning signs for AI.
On the other hand, if the advent of human-level AI was very surprising, then only a small group of people will ever respond to the anticipation of human-level AI (including those who are already concerned about it). This makes it more likely that a person who anticipates human-level AI now – as a member of that small group – should work on the highest priority things that will need to be done about it ever. This might include object-level tools for controlling moderately powerful intelligences, or design of features that would lower any risks of those intelligences.
I have just argued that the best approach for dealing with concerns about human-level AI should be depend on how surprising we expect it to be. I also think there are relatively cheap ways to shed light on this question that (as far as I know) haven’t received much attention. For instance one could investigate:
- How well can practitioners in related areas usually predict upcoming developments? (especially for large developments, and for closely related fields)
- To what extent is progress in AI driven by conceptual progress, and to what extent is it driven by improvements in hardware?
- Do these happen in parallel for a given application, or e.g. does some level of hardware development prompt software development?
- Looking at other areas of technological development, what warnings have ever been visible of large otherwise surprising changes? What properties go along with surprising changes?
- What kinds of evidence of upcoming change motivate people to action, historically?
- What is the historical prevalence of discontinuous progress in analogous areas (e.g. technology in general, software, algorithms (I’ve investigated this a bit); very preliminary results suggest discontinuous progress is rare)
Whether brain emulations, brain-inspired AI, or more artificial AI come first is also relevant to this question, as are our expectations about the time until human-level AI appears. So investigations which shed light on those issues should also shed light on this one.
Several of the questions above might be substantially clarified with less than a person-year of effort. With that degree of sophistication I think they have a good chance of changing our best guess about the degree of warning to expect, and perhaps about what people concerned about AI risk should do now. Such a shift seem valuable.
I have claimed that the surprisingness of human-level AI makes quite a difference to what those concerned about AI risk should do now, and that learning more about this surprisingness is cheap and neglected. So it seems pretty plausible to me that investigating the likely surprisingness of human-level AI is a better deal at this point than acting on our current understanding.
I haven’t made a watertight case for the superiority of research into AI surprise, and don’t necessarily believe it. I have gestured at a case however. Do you think it is wrong? Do you know of work on these topics that I should know about?
Meteuphoric 
It’s a matter of a degree. AI is already here, more intelligent every passing day.
Is it above human level? Sometimes it is. More and more so, every day.
And yes, it will be surprise for anyone not directly involved in the most eye-opening AI project.
You’re using the word “surprising” but talking about the speed at which AI will be developed. Those aren’t the same at all. I found thinking about this very confusing while those two concepts were mixed in my mind.
We haven’t created software that self-modifies its programming without arbitrary restriction, and that capability seems necessary for an AI. I think general AI will follow soon after that capability, though not immediately. Because such software is necessary, most researchers will see AI coming, though broader society might not.
Slow development would be a case of ‘not surprising’, but isn’t the main thing I intend to talk about. For instance, it could be that AI will be developed quite fast, but will not be surprising because its development will follow a predictable trajectory from a way off.
What kind of ‘arbitrary restrictions’ are you talking about?
While I don’t have an opinion about this line of research, it is related to this thought experiment about AI risk:
If a group of people hold the emerging AI technology in secret the responsibility for the danger is on the human agents, because they acted carelessly or maliciously against their fellow humans. It would not matter if the danger was superintelligence that came to have responisble intent. The situation would be the same as if the group were to release a devastating biological or computer virus. There would be many human actions against which there would be no defense.
If the AI technology were more broadly known and a consensus formed that the big event would be in N years, consider what would be happening in N/2 years. There would be a race to put the useful precursors to the AI into wide use. This would lead to a significant reordering of human affairs, giving us a level of defense against the coming AI. Unless we presume that the AI would be able to recruit these tools away from us, we can imagine that the singularity will erase itself before it happens.
Variations on this idea are related to the rate of acceleration. It could be that the AI “just happens” and no one anticipates it. Or it could be that N is 1 year and there is no time to ramp up. But the premise is that prior to the event the work of inventing AI is a human endeavor at a human pace. Anyway, for these scenarios we need to prepare with an innovation that is quite low-tech, called the circuit breaker.
When you said ‘AI surprise research’, I thought you would talk about AI risk appraisal research, which prompted some thoughts about the latter:
Many effective altruists think that improving global rationality is important. Often, they say that the reasons that it is important relate to improving people’s perception of risks, and in particular, people’s detection of emerging technological risks like AI. The projects I’m thinking about are pretty young, like the Good Judgement Project, CFAR, LessWrong and Clearer Thinking but I wouldn’t be surprised if they or something like them had a large positive impact on the world. So why haven’t more projects been driven specifically to promote the rational appraisal of AI risks?
The closest things I can think of are:
– LessWrong
– CFAR’s SPARC
– AI risk surveys by FHI (and by Kruel)
– Biases in assessing risk (a paper by Yudkowsky)
What are the lowest hanging fruit in this area? Should we create a larger Good Judgement Project for AI?