E-mail updates
Follow on Twitter
Subscribe to RSS
Seth Wenig / AP
"Jeopardy" champions Ken Jennings, left, and Brad Rutter, right, look on as an IBM computer called "Watson" beats them to the buzzer to answer a question during a practice round of the "Jeopardy!" quiz show in Yorktown Heights, N.Y., Jan. 13.
How smart is your favorite search engine? If the game show "Jeopardy" is a guide, it's just about as smart as the average human.
Computer scientist Stephen Wolfram, the brains behind WolframAlpha, tested how often the correct answers to "Jeopardy" questions appear in the title or text snippets of the results page on Google, Bing, and a handful of other search engines. He didn't include WolframAlpha, his own search engine, because it uses a different type of technology.
Google displayed the right answer on its result page 69 percent of the time. Ask.com's page had the correct answer 68 percent of the time. Bing registered a 63 percent success rate, and Yandex came in at 62 percent. Blekko (58 percent) and Wikipedia search (23 percent) performed worse than the average human, who gets 60 percent of Jeopardy questions correct.
WolframAlpha
This chart shows the comparative success of several search engines at answering "Jeopardy" questions.
Of course Ken Jennings, the all-time winning champ of the game show in which players buzz in to provide questions that go with answers displayed on a screen, gets 79 percent correct, meaning that basic search engines have a way to go beat the best in the game.
That's where the IBM Watson supercomputer comes in. Next month, "Jeopardy" will air a series of shows in which the question-answering machine goes head-to-head against Jennings and Brad Rutter, another champ, for a $1 million prize. We already know that Watson bested the two Jeopardy whizzes in a test run this month, and the tournament shows have already been taped. Any bets on who's the winner?
Ken Jennings, Watson and Brad Rutter in a practice round.
The buzz over the human-vs.-machine match inspired Wolfram to conduct the search engine test as part of a thought exercise comparing his WolframAlpha technology, which is built on a different paradigm, to Watson.
He says IBM's machine is great for answering questions from unstructured data. This has potential real-world applications such as mining medical documents or patents, and doing discovery in litigation, he notes in a blog post about his test.
WolframAlpha technology, on the other hand, can be used to "investigate structured data in completely free-form unstructured ways," he writes. He goes on to explain:
"One asks a question in natural language, and a custom version of WolframAlpha built from particular corporate data can use its computational knowledge and algorithms to compute an answer based on the data — and in fact generate a whole report about the answer."
So where does Wolfram stand on the human-vs.-machine battle? The last line of Wolfram's blog post provides a pretty big hint about where his sympathies lie: "Good luck on 'Jeopardy'! I'll be rooting for you, Watson."
More stories on Watson and Jeopardy:
- Computer beats 'Jeopardy' champs in test round
- Supercomputer plays 'Jeopardy'
- IBM computer taking on 'Jeopardy' champs for $1M
- 'Jeopardy' streak comes to an end
John Roach is a contributing writer for msnbc.com. Connect with the Cosmic Log community by hitting the "like" button on the Cosmic Log Facebook page or following msnbc.com's science editor, Alan Boyle, on Twitter (@b0yle).
Leave a Comment:
The standard quoted for the search engine to answer the question ("that the answer be in the search results") is just not comparable to what Watson is doing. A more accurate comparison is to look at the answer from the first search result ("I'm feeling lucky", or equivalent). It would be interesting to see what the numbers are in that case.
Even more interesting, what will happen this Valentine's day. I attended one of Kasparov-DeepBlue matches, can't wait for this one.
Of course none of these search engines nor Wolfram could play Jeopardy. It's one thing to try to come up with the exact specific answer that Jeopardy demands. You also have to have a good sense of when you know the correct answer and when you don't so you know when to try to buzz in. If you buzz in on every question and only know half of them, you will be slaughtered at Jeopardy.
Just wait until this technology becomes widely available and affordable. The potential to automate all kinds of jobs in the service sector, including very high paying jobs that require substantial education, is likely to be significant.
Check out these videos and think about the implications:
This comment won't matter, since not many people probably read this blog, but your title is very misleading. Watson is fed information and learns based on experience how the human language is structured. It understands puns and jokes, etc. It knows when it gets the wrong answer, and then learns why, as to not repeat itself. Like Stan and Jose have eluded to, there is still a human element involved in searching with a search engine. It doesn't matter how you word a question, or even if you ask the question in the form of a pun. Watson can answer it regardless.