Even robots with a seemingly benign task could indifferently harm us. “Let’s say you create a self-improving A.I. to pick strawberries,” Musk said, “and it gets better and better at picking strawberries and picks more and more and it is self-improving, so all it really wants to do is pick strawberries. So then it would have all the world be strawberry fields. Strawberry fields forever.” No room for human beings.
But can they ever really develop a kill switch? “I’m not sure I’d want to be the one holding the kill switch for some superpowered A.I., because you’d be the first thing it kills,” Musk replied.
Maureen Dowd in Vanity Fair.