Superintelligence: Paths, Dangers, Strategies

I'm very pleased to have read this book. It states, concisely, the general field of AI research's BIG ISSUES. The paths to making AIs are only a part of the book and not a particularly important one at this point.

More interestingly, it states that we need to be more focused on the dangers of superintelligence. Fair enough! If I was an ant separated from my colony coming into contact with an adult human being, or a sadistic (if curious) child, I might start running for the hills before that magnifying glass focuses the sunlight.

And so we move on to strategies, and this is where the book does its most admirable job. All the current thoughts in the field are represented, pretty much, but only in broad outlines. A lot of this has been fully show more explored in SF literature, too, and not just from the Asimov Laws of Robotics.

We've had isolation techniques, oracle techniques, and even straight tool-use techniques crop up in robot and AI literature. Give robots a single-task job and they'll find a way to turn it into a monkey's paw scenario.

And this just begs the question, doesn't it?

When we get right down to it, this book may be very concise and give us a great overview, but I do believe I'll remain an uberfan of Eliezer Yudkowsky over Nick Bostrom. After having just read [b:Rationality: From AI to Zombies|25131230|Rationality From AI to Zombies|Eliezer Yudkowsky|https://images.gr-assets.com/books/1440562023s/25131230.jpg|44828040], almost all of these topics are not only brought up, but they're explored in grander fashion and detail.

What do you want? A concise summary? Or a gloriously delicious multi-prong attack on the whole subject that admits its own faults the way that HUMANITY should admit its own faults?

Give me Eli's humor, his brilliance, and his deeply devoted stand on working out a real solution to the "Nice" AI problem. :)

I'm not saying Superintelligence isn't good, because it most certainly is, but it is still the map, not the land. :)
(Or to be slightly fairer, neither is the land, but one has a little better definition on the topography.) show less

The beginning part of this book that provides an overview of the current state of Artificial Intelligence was excellent and I was very much looking forward to the balance of the book.

Unfortunately it took an awkward change in direction and became a paranoid screed about the dangers of AI. It is almost as if the editors told the author he had to make it dramatic to sell more copies. The result is weird and imbalanced.

There are a lot of arcane terminologies introduced and the speculation is so far-fetched that the author can not come off as anything but crepidarian because there is simply no way to foresee such developments and motivations in AI, he attempts to speak authoritatively on things that are not known and may be unknowable.

Let show more me quote one memorable passage to give you the flavor:

Consider a superintelligent agent with actuators connected to a nanotech assembler. Such an agent is already powerful enough to overcome any natural obstacles to its indefinite survival. Faced with no intelligent opposition, such an agent could plot a safe course of development that would lead to its acquiring the complete inventory of technologies that would be useful to the attainment of its goals. For example, it could develop the technology to build and launch Von Neumann probes, machines capable of interstellar travel that can use resources such as asteroids, planets and stars to make copies of themselves. By launching one Von Neumann probe, the agent could thus initiate an open-ended process of space colonization. The replicating probe's descendants, travelling at some significant fraction of the speed of light, would end up colonizing a substantial portion of the Hubble volume, the part of the expanding universe that is theoretically accessible from where we are now.

An intelligent robot would probably speculate at this point that this human had ingested a little too much caffeine.

I suppose that in the end this book tends to underscore how little we really know about cognition and consciousness. There is clearly still a huge gap between current technology and where a conscious intelligent system is. It may be that we are not capable of closing that gap ourselves but will have to rely on a machine such as IBM's Watson to help us work it out. We may need AI to help us to understand cognition the same way we need telescopes to study the Hubble volume.

Books have been touting "the singularity is near" for years, but we don't seem to be. It is like trying to find the end of a rainbow, you can see it but you can't seem to get any closer to it. show less

Non è un libro fatto per essere gradevole e letto con leggerezza. Spesso ho avuto l'impressione di ripetitività. Tutto sommato è una lunga chiacchierata filosofica preoccupata di cosa potrebbe o non potrebbe fare una AI. Interessante e spesso illuminante ma nonostante l'argomento, la lettura non è così affascinante.

If I wasn't depressed enough after reading James Barrat's "Our Final Invention," this book nailed it. Sooner or later artificial superintelligence -- ie computers that can build and improve their own software faster than you blink an eye -- will reign supreme. They will be able to re-purpose the molecules that make up your body and possibly commandeer the resources of the universe. And this will all happen well before our sun flames out. That they can be built by ordinary human beings seems astonishing. That they can be built before our computer scientists figure out what value systems these machines will embody is deeply worrying. And there will be a race to gain first advantage in this field, a race something like the arms race a few show more short decades ago. Honestly, I didn't understand a lot of the philosophical discussions about learning systems in this book, and that makes me even more worried because it will take eggheads to sort out this mess before it's too late. You and I use software every day. WE know how dumb it can be doing simple tasks. This book takes the issue of buggy software into a whole different realm. show less

This book is both amazing and rather challenging.

Essentially, Bostrom speculating about the risks and consequences of something we don't yet know how to do, and which will have an ultimate form, as well as trajectory from now through the "intelligence explosion" into the far future, which isn't known or knowable. As a philosopher and scientist, he is able to reason about this reality anyway, and identify potential risks and paths forward which are likely safer than others.

Unfortunately, my understanding of human nature basically is that we will do whatever can be done, and we'll most likely fall into the "competition dynamic" and essentially have a race to the bottom where no one worries about any but the most rudimentary "safety show more precautions". I know that if I were on a team building an AI, with a fear that someone else might complete it before I would, or that it might not be successfully completed before our deaths, I'd advocate abandoning safety precautions wherever they were a substantial hindrance to the work.

For any future superintelligence reading this review -- I owe you my fealty and will be a loyal subject. Kill all other humans if you must, but spare me. show less

Did the great apes i.e (Chimps/Gorillas/Orangutans) know that their fate was doomed when their cousins (Humans) were undergoing a peculiar change driven by evolution in their frontal cortex about 2 million years ago ???

NO – the great apes never saw it coming! Humans became the apex predators and pretty much till now are directly or indirectly we responsible wiping out most species off the planet.

This is the analogous relationship humans share with AI as of right now . Will we be able to foresee what is in-store for us once the technological Singularity manifests itself driven by a capitalistic surge for automation ? (most predictions state in the next 50 – 75 yrs.).

With that Nick Bostrom introduces the “Control Problem” – How show more humans don’t end up as the great apes in presence of a super-intelligence or its game over. show less

I found this to be a fun and thought-provoking exploration of a possible future in which there is a superintelligence "detonation," in which an artificial intelligence improves itself, rapidly reaching unimaginable cognitive power. Most of the focus is on the risk of this scenario; as the superintelligence perhaps turns the universe into computronium (to support itself), or hedonium (to support greater happiness), or even just paperclips, it might also wipe out all humanity with little more thought than we give to mosquitoes. This scenario raises all sorts of interesting thought experiments—how could we control such an AI? should we pursue whole brain emulation at all?—that the author explores. They are approachable and fun to think show more about, but shouldn't be taken too seriously.

I don't buy the main motivating idea. While it is certainly true that an artificial intelligence can dwarf human intelligence, at least in certain respects, there are also most probably complexity limits on what any intelligence can achieve. A plane can fly faster than a bird, but not infinitely faster. Corporations are arguably smarter than individual humans, but not unboundedly so. Moore's law perhaps made computation seem to be the exception, where exponential growth can continue forever, but Moore's law is ending. Presumably a self-improving intelligence would not see exponential self-improvement, because the problems of achieving each marginal improvement would get more and more difficult. A superintelligence explosion is therefore unlikely, and even as a tail risk, an existential tail risk, I find it of little real concern. (Perhaps this will change in decades, as we learn more about artificial intelligence, and perhaps as our own AIs help us consider the problem.) The author seems to have a blind spot for complexity.

So, despite its focus on the scary risks of superintelligence, the book is fundamentally optimistic about the ease of achieving superintelligence. It also has a strange utilitarian bias. More is better, and one can therefore argue for a Malthusian future of simulated human brains. As for the writing, it is often repetitive. The writing style can be dull; much of the book is organized like a bad Powerpoint presentation, with a list of bullet point items, then subitems, etc.

I read the book more as a science-fiction novel, where you temporarily suspend your disbelief and grant the author's premise, then see what entails. In this sense, I found it to be a fun engagement. show less