Who's Afraid of Artificial Super Intelligence (ASI)?

For anyone without a preexisting interest in artificial intelligence and who isn't already familiar with Moore's Law, the Turing test and the acronyms, nomenclature, and inside baseball of the field, it has been startling of late to hear folks from Elon Musk to Stephen Hawking to even Bill Gates mention the existential threat of artificial intelligence (or artificial super intelligence) and rank it as the top existential threat to the human species. Really? Beyond nuclear annihilation, global warming and Ebola? Why the kookiness on this topic, many wonder. Are these luminaries in their fields, otherwise highly intelligent and seemingly practical / reasonable individuals smoking something? Have they simply watched too much sci-fi and/or fallen for the ridiculous hype of their industries and/or their own fevered adolescent imaginations? Or is there really something to be concerned about here? Something to be very, very concerned about?

Maybe. I'd submit that if (more on this 'if' as we continue to explore this topic together) the most common scientific presupposition about the brain and willful human consciousness, is correct - that the the human brain is the material, biological mechanism that is all that is necessary to create willful human consciousness - we should be very, very concerned. It's very difficult to deny the logic that follows once the presupposition is accepted: that human minds and consciousness are simply the product of imperfect biological machinery, that we are now entering swiftly an age in which it is increasingly possible to retro-engineer the imperfect biological machinery of the human brain so that it can be perfected in way that could infinitely surpass our imperfect human template and, finally, that any willful consciousness that is created from such unprecedented mind machinery would be able to then take over its own development. Clearly, sooner or later (and many, including those above, would say sooner - as in mere decades), this might just mean game over for the human species, at least one that isn't somehow itself merged with or allowed on the rocket ship of artificial super intelligence.

Ready to be sucked into the black-rabbit-hole? While imperfect (and therefore sadly - and/or wonderfully - human), I've not run across a better, relatively brief introduction for the uninitiated than this recent two-part post at Wait, But Why?