E-mail updates
Follow on Twitter
Subscribe to RSS
IBM via AFP - Getty Images
IBM's Watson computer is made up of a cluster of 90 servers with a total of 2,880 processor cores.
IBM's Watson supercomputer looks like the clear favorite to win this week's man-vs.-machine match on the "Jeopardy" TV game show in the wake of today's action. Right now, the score totals are $35,734 for Watson, vs. $10,400 and $4,800 for the game's two human champions. But even if by some miracle Watson doesn't take the million-dollar top prize, computer scientists say its performance will be judged a triumph for artificial intelligence.
"Watson is clearly playing at a championship level," inventor/futurist Ray Kurzweil, who predicts that A.I. will match human intelligence by the year 2029, told me today in an e-mail. "Note that it's only going to keep getting better. We cannot say that for unaided human intelligence."
Kurzweil said Watson merits the high praise he bestowed upon the machine after seeing its performance in last month's public practice round. In his essay on KurzweilAI.net, he said computers had "not shown an ability to deal with the subtlety and complexity of language" ... until Watson came onto the scene.
"Watson is a stunning example of the growing ability of computers to successfully invade this supposedly unique attribute of human intelligence," Kurzweil wrote. He said that level of language understanding, combined with a well-programmed aptitude for pattern recognition, would make Watson's descendants "far superior to a human."
Alien intelligence
Boris Katz, a computer scientist at the Massachusetts Institute of Technology who pioneered the development of natural-language question-answering systems, agrees that Watson is a wonder. "IBM did a fantastic job," he told me. But he said Watson's foibles also show that a computer's brand of intelligence is still alien to us.
When Watson is good, it's very, very good. But when it is bad, it's horrid. For example, one of the clues dropped during a practice round was: "This trusted friend is the first non-dairy powdered creamer." The correct answer was "Coffee-mate," but Watson gave a nonsensical non-non-dairy reply: "What is milk?"
Another example: On Monday, "Jeopardy" rival Ken Jennings gave a wrong answer for the decade when Oreo cookies were introduced (the '20s), and Watson followed up with what was basically the same answer. ("What is 1920s?") It was left to the third contestant, Brad Rutter, to come up with the right answer (the 1910s). Expert observers assume that Watson flubbed the answer because it didn't catch the fact that the '20s and the 1920s were just two different ways to refer to the same decade.
"When you look at the blunders, you realize that they did not build a machine that thinks like us," Katz said. "The success of Watson does not bring us closer to the understanding of human intelligence. When we observe it making these mistakes, that should remind all of us that this problem is still with us, and it's waiting to be solved."
Overconfident computer?
Watson draws upon 15 trillion bytes of information in its memory banks, the equivalent of 200 million pages of text, and ranks the potential answers for a given clue using 2,880 parallel processor cores in its 90 computer servers. If the highest-scoring answer exceeds its built-in "confidence threshold," it'll buzz in. If no answer scores high enough to reach the threshold, Watson will keep mum. At least theoretically.
"We're seeing already that there are times when Watson really doesn't have enough information to have a good answer, but has the 'confidence' to give an answer anyway," said Eric Nyberg, a computer scientist at Carnegie Mellon University who helped program the supercomputer.
Despite Watson's occasional missteps, Nyberg is proud of the computer's overall prowess, as well as the speed with which it's answering the "Jeopardy" questions. "I was pleasantly surprised that Watson was able to buzz in against Ken [Jennings], because in all of 'Jeopardy,' he's the guy with the fastest trigger finger," he told me.
Today, during an interview on MSNBC, Jennings acknowledged that Watson has "an edge on that buzzer that human reflexes have a hard time keeping up with." He also acknowledged that the pressure was on, big time, going into the final round. (Jennings actually knows who won, since the three shows were taped last month under tight security.)
"The computer can't get stage fright, it can't get discouraged or frustrated. It's like 'Terminator,' it's just going to keep coming," Jennings said. "And so the human race is going to have to play probably aggressively here — big bets where necessary, play recklessly to win."
On Wednesday, TV viewers will find out how this particular man-vs.-machine match ends. But the computer scientists emphasized that this is just the beginning for Watson and its successors. "The fact that it's this fast, and this accurate, and its abilities allow it to do this well at 'Jeopardy' means that question-answering technology is really ready for prime time," Nyberg said.
Watson was built to serve up quiz-show knowledge, but those question-answering capabilities would probably be most valuable in specialized fields such as medicine and law. Watson's kin could help us puny humans sift through millions of possibilities and come up with the five or six best medical diagnoses, or legal precedents, or chemical configurations, or ... well, you name it.
"We're not thinking about applications where there isn't a human in the loop," Nyberg said. "We're definitely talking about an intelligent information agent that's working with a human."
What do you think? Will Watson win this week's showdown? Will question-answering machines become our most reliable advisers? Or will this turn into a replay of "Terminator 3: Rise of the Machines"? Feel free to weigh in with your comments below.
Correction for 12:20 a.m. ET Feb. 16: Error! Error! I've fixed the humans' totals at the end of the first game, and have corrected The Associated Press' figures in the referenced story as well.
More human-vs.-machine matches:
- Chess computer beats world's best player
- Checkers computer becomes invincible
- Poker-playing robot beats human pros
- New Scientist: Computer beats human at Japanese chess
Join the Cosmic Log community by hitting the "like" button on the blog's Facebook page or following b0yle on Twitter.
Leave a Comment:
You have a mistake in the article. Watson has no voice recognition, so it can't hear other contestants' answers. That's why it didn't have a problem with giving the "1920s" answer after the first contestant said "'20s".
Watson cannot hear, but he sees the other contestants' responses in text. This was a modification during development, when it was observed that Watson would lose points by giving the same incorrect answer as one just given. That problem can still happen ('20s and 1920s) but this is another input that Watson can absorb and use for adjustments. The human contestants also use this type of knowlege to learn about the categories - sometimes it takes a couple of examples to catch on to the tricky concepts.
Mailman8 is correct. Watson does receive the input of other contestants' answers in text. Its software was also modified to parse out things like '20s and 1920s in its analysis of possible answers. But Katz said Watson apparently ran into a snag with the combination of those two phenomena (figuring out the alternate context for numbers when they are input as text). All I know about this is what Katz told me.
Chris Welty of the IBM team told the audience at RPI (the broadcast is being shown on in our theater) on Monday that Watson does not receive the answers given by other contestants.
The whole SPECTACLE was "bizarre" at best, and simply a commercial for IBM at worst.
The Jeopardy producers chose to sink to a new LOW in syndication programming.........UGH.
What are the rules? The standard Jeopardy rule is that you cannot "Buzz in" until the questions has been read. This makes some sense -- it make contests wait until the whole question is out so as to prevent early "buzz ins" from ruining the concept of the show by making wild guesses as to where the question is going. I suspect the computer gets the question zapped in as soon as it appears on the board. That is a heavy three second advantage to the computer's not to mention that the human reflex to hit the buzzer, after the question is read, is at least one second. The computer does not have that reflex problem, which explains why it's always "buzzing in" first. The computer obviously knows what it is talking about, but to put it up as a contest of SPEED and knowledge is a sham.
I agree that this is nothing more than an add for IBM
Doesn't the question appear on the board in full as soon as Alex starts reading it? It's possible to read faster than he speaks, and you could have the right answer (not just a "wild guess") before he finishes reading.
Does Watson take into account that he has a length of time after he buzzes in order to formulate and vocalize his response? It seems that there is a small period of time where Watson can buzz in with a couple "guesses" and then spend an extra second or two to narrow it down further.
mKlRivOwner........Humans do that too !
Alan, that's weird. The IBM engineers who are here with us at Univ. of Texas at Austin to watch the show and answer our questions about Watson told us that nobody was typing the answers of other contestants to be provided to Watson. They've told us that Watson simply doesn't receive the wrong answers of other contestants. It only receives the right answers, since those are already in the system (no need to type those).
I agree with DoobieDo. It's a big ad for IBM and yes, ABC is at a new low.....In addition, it's been very boring.....
Do I sense a lot of Microsoft, Google and HP fans here? No matter how poorly you understand how Watson plays the game (and it does it as fairly as it is possible), the fact that it can "compete" with humans in answering these questions with very complex formulation is absolutely impressive. An ad? These people are completely missing the point!
To those who say its just an ad for IBM are correct....but this ad is amazing...when have we ever seen a machine that is NOT connected to the internet, but thinks on its own in recognised human language....a total breakthrough in technology which is going to benefit mankind in the near future.
This is an ad for IBM. However it was entirely fair to pit Watson against humans in a contest of speed. Only now at the end of the contest, do we know how unfair it was.
The test was to see how long it took Watson to find the correct answer, and how often Watson would answer correctly. Before now no computer has been able to answer random questions. (Or provide a question for a random answer).
Even being able to find the answer takes time. Your home computer would take days to find the answer. Watson may not even find the answer, or it may find the wrong answer. It is clear that Watson cannot think or reason. Watson is simply Google 2.0.
I would say that while there was some moderate success, I would call this test a failure. It does show progress. The problem is that Watson is too confident in answers that are wrong. The one about non-dairy creamers, and Watson's response being "Milk" is a serious error. I would hate for professionals within any field to rely on a computer that has such poor judgement.
@MM, that was kind of my point.
The machine can certainly recall facts. But will it ever have wisdom or moral judgement? Will it be able to paint like Van Gogh or compose like Beethoven? Will it be able to make us laugh?
So far, the human brain is the more complex and mysterious thinker upper. Hope it always stays that way.
You mean I think, will AI ever have imagination? It, and it alone is essential in just about every human
endeavour.
teacher trish, it can learn moral judgment just like we do. Moral judgment is not an inherited trait or in our DNA sequencing, it's a learned responses for the most part. And honestly, do humans have wisdom and moral judgment? Some do, but a lot do not!
I completely agree! He also has no common sense and answers incorrectly for things that you can't neccessarily look up.
But how can you teach morality, a innately human concept, to an inhuman machine? How could something that is not human value human life? These are the questions that our species will have to answer before the machines get smart enough to answer it on their own (we might not like the answer we get).
Even more basic than issues of moral judgement, is the question as to whether Watson still needed to be programmed to play a specific game (Jeapordy), as opposed to possessing the capability to adapt to *any* game given a set of rules. The latter is a sign of intelligence that, while innate to the human mind, may still be elusive to computer intelligence. Given a new game, or even a variation on the previous game, you and I can learn new rules and adapt to the new game. Can Watson?
Without the ability to assimilate rules of different games, without specific programming, Watson is still just a very sophisticated (and powerful) pattern matching and information retrieval machine in the end. Real intelligence is still more than that.
I'm not impressed. Watson is nothing more than a powerful search engine with a sophisticated algorithm--a "Google Jeopardy" if you will. Calling this artificial intelligence is over-stating the matter. Watson is unable to reason or grow in ways beyond the recalling of facts. With Watson the old axiom of "garbage in-garbage out" still holds.
actually, Watson CAN learn and grow during the game. If you saw the NOVA documentary that covered some of the practice games, you saw a situation where Watson completely misunderstood a category. The category called for a response naming a month of the year - Watson was throwing out nonsense. After the human contestants answered a few examples correctly, Watson caught on and answered correctly for the highest value response.
Obviously there is further room for improvement. On Final Jeopardy of the first game, the category was US Cities. Watson named a Canadian city (maybe he was kissing up to Trebec.)
Only reason that the computer is winning is because it can signal faster. That is almost if not more important as how smart you are. I imagine that the two guys could probably answer the same questions, except that they cannot beat it to the buzzer.
There's no question that the technology is an advancement but still has significant limitations in comparison to human reasoning. Regardless, the game is clearly unfair. First, the computer can process the input in an infinitesimal amount of time compared to a human. Second, the computer can "buzz in" instantaneously after it is allowed to (electronics and computers are designed for this sort of thing). Therefore, Watson will always have an advantage in this game and clearly controlled the board by beating the contestants to the buzzer. With three humans, the result is more random.
Call it A.I. or don't, but imagine how cool it will be to have this computational power on your cell phone in 5-10 years. Being able to ask a question in plain English (or by then perhaps just think the question) and get an answer back with 99% + accuracy. In 5-10 years it will most likely be that accurate, and that technology will most likely be feasable on a cell phone.
Jeff-3061280.....I have that now. I just call my ex-wife, she knows everything !
"
A.I. will match human intelligence by the year 2029" I don't know where he pulls this number from, but I call bs. Watson's only real triumph is its ability to decipher complex language, which is a pretty simple task as far as human abilities go. AI is still not even close to humans' creativity, problem solving, emotional intelligence, ect. Computers will be better than humans at many things by 2029, but to say they will have the same intelligence is ridiculous.
Clearly you haven't talked to many other humans lately. Most people are beaten handily by Watson. Most people would be beaten handily by a Speak N Say in an intelligence test.
I have to wonder if, given the task to design a system smarter than he is, what would Watson's descendants be like? Would it be something that cures cancer and tells us stories or would it decide that the best way to deal with the human problem would be to send a cyborg back in time to murder Sarah Connor?
It's an amazing time to be alive right now that we would have to ask these questions.
Ask Watson about the "Trickle Down" Theory of economics.
Touche Bill!
You made me smile! I don't think a computer can be made to explain trickle down economics!
Stupid. Thanks for lowering the discussion level.
ooooh, and thank YOU for contributing so much!
I do have a question for Watson: Watson, where is Osama bin Laden?
Toronto ?
Watson, identify the locations of the weapons of mass destruction of Saddam Hussein so that the hundreds of thousands of people who died in Iraq, Iraqis and Americans, shall not have died in vain.
Kurzweil's point about the machines inevitably improving while human intelligence remains the same is well founded. Watson will get better, I wouldn't be surprised to see something like it challenge the Turing Test within the next five to ten years. Once they can be tasked with seriously improving their own performance though, it will be over for us.
You are so right Dan. The day a computer can exam its own code, re-write it to improve itself, and then upload its new code to itself, will be a turning point in the world. They even have a name for this event. http://en.wikipedia.org/wiki/Technological_singularity
http://en.wikipedia.org/wiki/Seed_AI
I consider these computers to be developing children. They are learning to adapt in the world we've brought them up in. To write them off now because of the limitations in their current capability would be giving up too soon.
Also, one day, this "garbage in-garbage out" machine will be able to act autonomously, and within its innumerable memory banks, recall the time that you belittled it, pinpoint your location using its search database, and alter a dying satellite's trajectory to crush your house.
Watson, estimate the number of years before the United States destroys itself if the economic policies of the Republican Party is put in place.
or
Do you want to play a game?... Did you say MAD?
The scores in the first paragraph of the article are not correct.
Watson's score after first game is right - 35,734
Brad is at 10,400 after Final Jeopardy (not 5,400)
Ken is at 4,800 after Final Jeopardy (not 2,400)
Aha, I believe you're right. I picked those numbers up from AP, which apparently didn't account fully for the Final Jeopardy results. I'm fixing that now. Thanks, Mailman8, you definitely know your "Jeopardy."
Here's a Trivia Question: What do you call it when 18 Saudia Arabians hijack planes and crash it into the World Trade Center and then the US sends its troops to another country instead, and then leave the rest behind in the US considering them as suspects?
Answer: False Flag Operation.
The Jeopardy setup includes a lockout device that prevents a contestant from ringing in until the clue is complete. So how did Watson consistently beat Brad and Ken to the buzzer, assuming all three knew mosat of the same answers. Maybe the issue is simply Watson's superior reaction time.
I am more interested in what the programming of Watson says about English grammar. Does someone out there know if Chomsky's work was involved? Watson has to parse English even when puns, circumlocutions and metaphors are involved. How did the programmers pull this off? In the meantime I am not worried about Watson overtaking the human race until he can walk off the stage like Data.
This technology is going to have a big impact on the job market:
Check out the book "The Lights in the Tunnel: Automation, Accelerating Technology and the Economy of the Future"
This book tell how technologies like Watson are going to create unemployment and even more concentration of income and wealth.
Steve,
I fear you are correct and it does cause me concern. I am 60 years old. In my lifetime I have seen things change dramatically. In the 1950's, if you had a High School Diploma you could get a good job, buy a house, own a car and raise your family. By the 1960's you really needed some college and preferably a bachelor's degree to really succeed and by the late 1980's/early 1990's you had to have Master's Degree or a professional degree like Engineer, or Doctor or Lawyer to guarantee success in life.
Oh, that's not to say that folks can't find work and feed and support their families but it's becoming harder and harder without an advanced degree to achieve the American dream.
I remember in Jr. High in 1962 a teacher telling us that someday we'd have the ability to play movies in our own homes without having to buy a projector, screen and sound system to run film. We'd be able to play them right there on our television sets. It seemed like science fiction, but by the middle 70's it was true. The same with computers, cellphones and on and on. I've seen such quantum changes in my lifetime.
But it does make me wonder what the future will bring and what problems mere mortals will face in the coming days as computers and AI begins to outpace human development.
I have to echo Skip's comment here (hiya Skip!)
I was born in 1981 and as I approach my 30th birthday I too am really starting to examine my lifetime and the changes that I've witnessed. My father is a structural engineer and thanks to him I am now a drafter and engineer technician 2 (fancy talk for someone who works with engineers). I don't have that higher education degree but the pressure certainly is mounting and so I have plans to go back to school and become an engineer myself.
I don't remember exactly when my family got our first computer but I think it was in the mid 80's. They put it in my bedroom, which I wasn't a fan of. My two brothers and one sister all had their own rooms but I was forced to convert my room into a den? Well, lets just say that doesn't sit well with someone who about 6 years old.
Then there came along the Nintendo. We didn't get a nintendo right when they first came out but when we did get one it came with Super Mario, Duck Hunt, and that Olympic game with the pad that you run on. It was all so futuristic to me and I was only a kid!
Then there was the Internet. I still remember playing Doom over the modem with friends from my Jr. High school. Back then it wasn't quite as streamlined as playing computer games online today. The massive multiplayer games still manage to boggle my mind. Anyway, I remember the first time we plugged the computer into the phone jack and it seems like a century ago now that I have high speed cable internet with wi/fi doo-dads and all that jazz.
My parents are both over 60 now and they both have blackberries. I have a much simpler phone because I'm somewhat of a technophobe when it comes to cell phones. Out of all the devices we use regularly the cellular device has really developed at a pace that seemed to quick for even my taste. But I think I'm alone in my fear of the phone.
Technology all over is developing at such an incredible rate. It's amazing what is possible now that was barely a dream a few years ago. But on that same note it is concerning that automation will replace thousands and more probably millions of people in their jobs. People will need to focus much more on their education if we are not to fall far behind. And even then who can truly say what the technology will be able to do in 5, 10, 20 years time? Quantum computing and new materials are making new possibilities into reality before many people even know there was a need for such technology.
I look to the future with a great hope and a small fear. I hope humanity will be able to ride the wave of technology and I fear what will happen if and when we fall off our surfboard of mental superiority.
I think what we have here is less a prototype Data from Star Trek -- the Next Generation and more a prototype Doctor from Star Trek -- Voyager, insofar as what we have is a mainframe or server farm with an avatar.
Please state the nature of your trivia emergency.
I think what we have here is less a prototype Data from Star Trek -- the Next Generation and more a prototype Doctor from Star Trek -- Voyager, insofar as what we have is a mainframe or server farm with an avatar.
This has been interesting - especially as we have been able to watch the competition in "real" (though delayed) time. IBM partially promoted this as being a "Grand Challenge" - I cannot but help wondering what will be the next "Grand Challenge". Perhaps something involving robotics - e.g. baseball or soccer? Or some attempt at beating a Turing Test with additional human subjects coming from a variety of fields - engineering, medicine, social science, etc. How about some other ideas for the next "Grand Challenge".
It's hardly a "fair" contest to pit one man against a supercomputer and the collective efforts of thousands of first-rate computer coders and electronics engineers. "Watson" is nothing more than a very, very fast machine, programmed by a team of human experts. Supercomputers DON'T design and build themselves. Until they do, I'm not impressed.
I found watching Watson's replies fascinating. While more lofty pursuits may be medicine and law, what I actually see happening is the replacements of customers service representatives worldwide. Something I think has already happened, but poorly and with much consumer backlash. It seems like at least 50% of the time whenever I email customer service I get a form answer than usually does not apply to the question asked, and it takes 2 or 3 emails to actually get a reply that a human looks at. I see Watson not as a hope for the future, but as the harbinger of more and more consumer frustration and rising human unemployment as companies attempt to replace every human job possible with computers. And we thought talking to Indians with thick accents was bad! Just wait til we are talking to cheap Watson-style " service reps".
The whole man vs machine scenario was boring.
Stanley Kubrick would be proud.
After all, the initials H, A and L were picked as each letter preceded I, B and M.
10 years late ... oh, well.
I can't believe I did not know that. Thanks, that's very interesting, is it true?
Very Interesting forum! It's got it all....digital explosion...entertainment...desocialization....I would like a 'mini' Watson on my Blackberry to burn through all the BS and minutia of my job....technology is evolving in its natural order...throw it all in the mix....genetic engineering...In the Year 2525,,,,,,why not get friendly with Watson....we already are "intimate" with our Blackberry; laptop;GPS;Ipod;PC;......I'll stop meandering....and keep my traditional preferences...nature...friends...adventure.....keep the "box" in the laboratory/data center!
Apparently the producers spread one complete game over two days. I didn't watch Monday's show, but Tuesday's consisted of the Double Jeopardy round and a Final Jeopardy question plus a lot of IBM puffery. I suspect Monday's show was the first Round and I don't know if there was a Final Jeopardy question but the totals at the start of Tuesday's show indicate there wasn't one.
I'm looking awaiting Wednesday's show and I suspect it will consist of 2 round plus Final Jeopardy.