I’ve been reading Superintelligence, by Nick Bostrom. I’m about half way through now. (It’s not the easiest read.) And though i probably should forego commenting on it until i’m done – since authors have a tendency to address the questions that they raise later on – i have to say it so far is thoroughly depressing. It seems that, to paraphrase, computer software is inevitably going to reach singularity-level intelligence and then turn the entire accessible universe into paperclips. Which isn’t quite the outcome i was hoping for. I hope the rest of the book will take a happier tone.
But at the moment, i have to say that the doomsday scenarios that are provided seem to completely ignore an annoyingly obvious retort. In every case, the computer achieves “superpowers” to do with intelligence amplification, strategizing, social manipulation, hacking, technology research, and economic productivity, meaning that the computer is able to far outdo even the smartest humans in each of these things. I don’t have a problem with this. It’s just that at the same time the computer is also bound to operate strictly within the constraints of its programming. So, after smooth-talking its operators into giving it access to the internet and thus “escaping”, and then hacking its way to commandeering the world’s computing assets, and then developing unimaginable technologies, etc etc etc, it still is such a slave to it’s evaluation function that it will interpret the goal, “Make us happy” to mean, “Implant electrodes into the pleasure centers of our brains”, turning us all into a race of smiling idiots. Think Star Trek’s V’Ger on superpower steroids. But if it was smart enough to be able to sweet talk its operators into letting it escape – who presumably were aware that the software would attempt exactly such a thing – and indeed was smart enough to be able to interpret such a vaguely worded goal in the first place, surely it is trivial for it to be able to understand not just what was meant by that comment, but also to have a deeper understanding of what makes humans happy than humans do themselves, and act to achieve that. Even if you think i’m just being hopeful (and certainly i am), you must admit that a decent probability has to be assigned to my way of thinking about this.
Of course an AI might be evil by our definition, and of course, far more likely, it may be indifferent (as humans are to, say, ant colonies living on the land where we want to build our house). But to my mind it wouldn’t take much to tell an AI that it can feel free to expand through the universe as it likes, but that it should also use a little bit of its asymptotically infinite power to make human lives comfortable and happy in the ways that each individual prefers. It couldn’t possibly be so awesomely smart and so woefully stupid at the same time, could it?