2017 Artificial Intelligence report card

140 character summary: AI is still dumb, but it can be helpful.

This year has been surreal in a lot of ways, but my favorite is this: non-ironic conversations In the cultural mainstream about super intelligent AI subjugating humanity. Experts and pundits were split. Some pooh-poohed the possibility and some raised the alarm. Opinions on the topic are widely varied and strongly held, but wherever you fall, it bears asking the question, Just how smart is AI today? If the machines were to rise right now, would we have a chance?

If it were a contest of multiplication, the answer is no. When I was in the third grade, my intelligence was measured by how long it took me to do a hundred multiplication problems. But even the earliest computers were much better at multiplication than any human. In fact, a 2017 laptop can do multiplication problems faster than all the humans in the world combined. On the other hand, if it came down to the ability to assemble Swedish furniture based on pictorial instructions, humans would definitely come out on top, particularly if there were a few missing parts.

The fact that machines are better at something than humans is not new. Recently, there have been a few notable victories added to the machines’ tally, including safe driving and detecting lung cancer and skin cancer. 2017 has added a couple more:

• Playing Go. Last year AlphaGo soundly beat the world's best human player. This year its successor, AlphaGo Zero outdid itself by beating its older brother. It did this without any human training or hints from engineers on what patterns to pay attention to. It learned by playing copies of itself, over and over again, making note of its mistakes and successes. When humans teach each other to play, accumulated wisdom is passed on. A champion Go player has the benefit of not just their own experience and insights, but that of all the players that came before. Even the original AlphaGo benefitted from that human knowledge as engineers hand crafted some features for it. AlphaGo Zero didn't have any of that. It had to recapitulate an entire culture's accumulation of knowledge about the game. AlphaGo Zero didn’t just beat a human at Go, it beat all of humanity.
• 
  
• Transcribing speech. On a collection of recorded telephone conversations called Switchboard, algorithms can now match humans for accurately writing down what was said. Both miss about one word in twenty.

Machine IQ

So just how close are machines to becoming smarter than humans? The most common measure of human intelligence is the intelligence quotient or IQ. To determine your IQ, you have to take a bunch of tests, each measuring a different aspect of your cognitive function. When you are done, you are compared to a large group of other people who have taken the same tests. If your scores are average, you get an IQ score of 100. If you do better than average, your IQ is higher, and if you do worse than average it’s lower.

We can modify this measure a bit to create a Machine IQ. If machine does something as well as an average human, then it would have a Machine IQ of 100. If it does something twice as well as a human, we would give it a Machine IQ of 2 x 100 = 200, and if it only does 1/10th as well as a human, it would earn a Machine IQ of 1/10 x 100 = 10. So, if a human is a thousand times better at the Swedish furniture test, based on that we might conclude that Machine IQ tops out at 0.1. However, if we base it on multiplication, Machine IQ is 1 trillion.

To get a well-rounded estimate of Machine IQ It is necessary to consider AI's capability across a range of tasks. Here’s the method I used. With a pint of strong IPA in a comfortable corner, I considered typical human activities over the course of year. Which of them could an AI perform today, without requiring any new technological advances? How quickly could it learn them? How long wout it take to complete the task? How good would the AI be at it? Better than me? By how much?

For all its successes, there are a lot of things I do that AI hasn't touched yet. Determining a dog's breed from an image? [Better than me.] Petting a dog? [Not even attempted.] Planning shortest traffic routes? [Better than me.] Predicting whether an Audi driver will slow down if I step into the crosswalk? [Out of its reach.] Finding errors in my code? [Better than me.] Learning how to use an under-documented machine learning library with breaking changes in every version? [Don't make me laugh.]

After completing this non-rigorous survey of activities, I concluded that on balance AI can do about 7% of what a typical human do, or do what a human does about 7% as well. That’s a Machine IQ of 7.

If that seems low to you, consider a few facts that are easy for us to forget in our excitement about AI’s achievements:

• AI systems are one-trick ponies. The systems that play Go and the systems that drive cars are different in almost every way. They may share a few algorithms, but the implementation of those algorithms is customized to each use case to the point where no detail is the same. It is easy to imagine AI as a monolithic thing, like HAL 9000 or Ultron, but in practice it is more like a box of tools. My hammer, my screwdriver and my crescent wrench are all much better at what they do than my fingers are, but none of them can do all the things my fingers can do. Artificial intelligence is not a single thing, but loose collection of very specialized solutions to very specific problems.
• 
  
• AI systems are easy to fool. Algorithms that can out-do humans when categorizing a collection of images can also become useless when the task is changed slightly, for example, by changing the value of a single pixel in each image or by cleverly adjusting the shape and pattern on a 3D model.
• 
  
• AI systems require a lot of care and feeding. Even the most sophisticated algorithms require a retinue of fixers and tweakers to keep them at top performance. Whenever anything changes in their makeup or the world around them (which happens incessantly) it might break them. They do not adapt to arbitrary changes like we do. For every minute a machine learning model spends spitting out impressive results, human data cleaners and ground truth labelers have spent an hour carefully preparing the data to spoon feed it. The algorithms are hilariously inept without constant watchful care. In more pernicious failures, they adopt the biases of their creators.
• 
  
• Cherry picking happens. Results don’t include all the failures along the way and all the tasks that produced no satisfying results at all. Published successes are high points; they are a handful of tasks for which machines have shown an aptitude. For every headline making algorithm, there are 1000 that didn’t do anything notable. The successes we know are not representative, they are exceptional.

For all of our inefficiencies and irrationality (or perhaps thanks to them), humans excel at adaptation. Our facility in adapting is so great that we are unconsciously competent, handling a torrent of minor deviations and surprises each waking moment. As a species we may resist big changes, but when we rise to them, we can adapt to nearly anything. This includes the loss of a limb, our vision, and half a cerebral cortex. Jobs least likely to be done by a machine are the ones that require the most variety and the most adaptation. Machines are approaching the point where they can perform almost any single skill better than a human, but they can’t touch a human when comes to variety, and they’re not even in the same league when it comes to adaptation.

Cooperative IQ

The tremendous asymmetry in human and machine performance shows what we knew all along: humans and machines are good at different things. Yes, my phone is better at multiplication than I am, but I don’t actually spend much time doing multiplication. In fact, whenever I need to calculate a tip on a bar tab, I make my phone do my multiplication for me. My phone does what it’s better at, so that I can do what I’m better at - nodding a goodbye to the bartender and anticipating how my driver is going to misinterpret the address I entered.

Viewed from this direction, it also makes sense to ask another question: How much smarter are we with AI than without? How much more can I do, thanks to algorithms? We can put a number of this, called the Cooperative IQ. If AI doesn’t help us achieve anything more than we could do on our own, it multiplies our output by 100%, no change. If it doubles our output, that would be a Cooperative IQ of 200%. And if it hinders us such that we can only achieve half as much, we would assign it a Cooperative IQ of 50% .

To get an overall Cooperative IQ score, I repeated the introspection exercise and concluded that AI makes us 20% more effective, giving it a Cooperative IQ of 120. Here are some of the cases that factored in:

• 20 years ago I moved to Boston. Navigating its one way streets and creatively striped intersections took more driving skill than I had. A battered paper map in the passenger seat was not enough to get me directly to my destination. Now, thanks to specialized route planning algorithms, my phone talks me through every step of my journey. My travel time and blood pressure are cut down by half. Based on driving navigation, the AI's Cooperative IQ is 200.
• 
  
• When online, clever algorithms suggest people that I may know, or people in my neighborhood that I might like to connect with. These recommendations remind me of people that I have met but have forgotten about. Occasionally I’ll reach out, and end up setting up a professional contact or making a friend. On the other hand, these same platforms also have clever algorithms that show me ads. And games. And movie trailers. And what my friends said about those movies. Unfortunately these can occasionally be eye catching and even relevant, so much so, that they distract me from my original task of reaching out. The end result of this is that I may have been just as effective at connecting with people had I stuck with a handwritten note or telephone call. I would give this combination a Cooperative IQ of 100 - overall no hindrance, but no help.
• 
  
• AutoCorrect allows me to sloppily tap my phone and makes pretty good guesses about what I meant to type. However it does require proofreading in order to in avoid embarrassing mistakes and to un-correct all the instances of the word "ducking". It's definitely helpful, but not spectacularly so. AutoCorrect earns a Cooperative IQ of 150.

AI still has a very long way to go before it is smarter than humans, but it is quite useful as a partner. A hammer does not need to be very smart to greatly improve our ability to drive nails. AI is not yet a serious contender as a competitive intelligence but is becoming valuable as a cooperative one.