Wendy Sherman: “Artificial intelligence is a powerful tool. It will be a more efficient tool. The question is, will it remain unaware?…”

“Artificial intelligence is a powerful tool. It will be a more efficient tool. The question is, will it remain unaware? If it becomes aware, will it have objectives? Some have said that artificial intelligence will be the perfect Zen Buddhist, well aware of the situation but utterly uncaring as to what happens. You go to today's largest computers and say: I think I should hit you with a sledgehammer and remove all your parts and sell the parts. How much will I get for the parts? Today's computer, today's artificial intelligence, will do its best to calculate the value of those parts. You go to the stupidest or one of the less intelligent life forms, perhaps a cockroach running across the floor, and tell it that you want to stomp on it and see how it reacts. It has a survival instinct. Let us hope that artificial intelligence does not develop a survival instinct, that it does not develop self-awareness. However, let us not just hope. Let's take 1 percent of what we are doing to make AI more powerful to make sure it is safe. I look forward to working with my colleagues on that objective. While I am talking about engineered intelligence, many decades from now we will confront another issue, and that is genetic engineering, designed to create new levels of intelligence. Whether they start with elephant DNA or human DNA, certainly a mammal with a genetically engineered brain 10 times the size of the average human would probably be pretty smart.…”