Computer to play Jeopardy against past champions

[flash9286]

Pretty cool. Look forward to seeing Watson in action.

IBM (NYSE: IBM) and America's Favorite Quiz show® Jeopardy! today announced that an IBM computing system named "Watson" will compete on Jeopardy! against the show's two most successful and celebrated contestants -- Ken Jennings and Brad Rutter.
The first-ever man vs. machine Jeopardy! competition will air on February 14, 15 and 16, 2011, with two matches being played over three consecutive days.
View video: http://tinyurl.com/295wpoq
Watson, named after IBM founder Thomas J. Watson, was built by a team of IBM scientists who set out to accomplish a grand challenge – build a computing system that rivals a human's ability to answer questions posed in natural language with speed, accuracy and confidence. The Jeopardy! format provides the ultimate challenge because the game's clues involve analyzing subtle meaning, irony, riddles, and other complexities in which humans excel and computers traditionally do not.
Competing against Watson will be two of the most celebrated players ever to appear on Jeopardy! Ken Jennings broke the Jeopardy! record for the most consecutive games played by winning 74 games in a row during the 2004-2005 season, resulting in winnings of more than $2.5 million. Brad Rutter won the highest cumulative amount ever by a single Jeopardy! player, earning $3,255,102. The total amount is a combination of Rutter's original appearance in 2002, plus three Tournament wins: the "Tournament of Champions" and the "Million Dollar Masters Tournament" in 2002 and the "Ultimate Tournament of Champions" in 2005.
The grand prize for this competition will be $1 million with second place earning $300,000 and third place $200,000. Rutter and Jennings will donate 50 percent of their winnings to charity and IBM will donate 100 percent of its winnings to charity.
"After four years, our scientific team believes that Watson is ready for this challenge based on its ability to rapidly comprehend what the Jeopardy! clue is asking, analyze the information it has access to, come up with precise answers, and develop an accurate confidence in its response," said Dr. David Ferrucci, the scientist leading the IBM Research team that has created Watson. "Beyond our excitement for the match itself, our team is very motivated by the possibilities that Watson's breakthrough computing capabilities hold for building a smarter planet and helping people in their business tasks and personal lives."
"We're thrilled that Jeopardy! is considered a benchmark of ultimate knowledge," said Harry Friedman, Executive Producer of Jeopardy!. "Performing well on Jeopardy! requires a combination of skills, and it will be fascinating to see whether a computer can compete against arguably the two best Jeopardy! players ever."
Prepping a Machine to Play a Human
This fall, Watson played more than 50 "sparring games" against former Jeopardy! Tournament of Champions contestants in final preparation for its television debut. In addition, Watson has taken and passed the same Jeopardy! contestant test that humans take to qualify to play on the show, giving Jeopardy! producers confidence that the match will be both entertaining and competitive.
Highlights of the sparring matches can be viewed and tracked over the next few weeks at www.ibmwatson.com.
Real World Applications for Watson's Technology
Beyond Jeopardy!, the technology behind Watson can be adapted to solve problems and drive progress in various fields. The computer has the ability to sift through vast amounts of data and return precise answers, ranking its confidence in its answers. The technology could be applied in areas such as healthcare, to help accurately diagnose patients, to improve online self-service help desks, to provide tourists and citizens with specific information regarding cities, prompt customer support via phone, and much more.
What is Watson?
Watson is a breakthrough human achievement in the scientific field of Question and Answering, also known as "QA." The Watson software is powered by an IBM POWER7 server optimized to handle the massive number of tasks that Watson must perform at rapid speeds to analyze complex language and deliver correct responses to Jeopardy! clues. The system incorporates a number of proprietary technologies for the specialized demands of processing an enormous number of concurrent tasks and data while analyzing information in real time.

http://www.prnewswire.com/news-relea...111818999.html
Here is a video of it in action.

[Lorizael]

Saw this article earlier today. Awesome.

[SlowwHand]

Ken Jennings will kick ass. He's a nerd that no machine can stand against.

[Main_Brain]

Bump as this is airing. It is quite interesting

Spoiler:

(Ken Jennings didnt kick ass)

[rah]

It was quite interesting to watch. It all depended on the type of question. If it was straight up facts, Watson was untouchable, but some questions posed a big problem.
The second day's final question was the perfect example.

Spoiler:

The question was what major US city's largest airport was named after a world war II hero and it's second airport was named after a famous world war II battle. Jennings and the other player had no problems with knowing the answers was Chicago for O'hare and Midway but Watson couldn't make the multiple connection and Answered with TORONTO ???? (how it picked a Canadian city is beyond me)

But what was most interesting was that on every question it showed watson's top three picks along with it's confidence level. SOme of the secondary answers were hilarious. And when watson would get a daily double, the amounts he bet were quite bizarre.

[Lorizael]

The amounts Watson bets are actually specifically calculated based on its confidence in the category, its position relative to the other players, the state of the game, etc.

[rah]

Yes, of course there was an algorithm involved, I'm just saying that it resulted in some bizarre results. Even Alex poked fun at one of them.

[Lorizael]

The thing is, I bet those weird bets are all a lot "better" choices than the very round bets humans usually provide. AI may just now be able to do natural language processing at the level Watson is capable of, but number crunching and game theory have been AI's forte for a lot, lot longer.

[rah]

If you had seen his wager on the final question yesterday you might think differently.

[Lorizael]

I did. Given that it had a commanding lead, it didn't need to bet a lot (or anything) to maintain its position. But it obviously had some confidence in its knowledge of that category, which is why it bet some money.

[Lorizael]

Btw...

Watson’s wagering strategies
Editor’s note: This guest post from IBM Researcher Dr. Gerald Tesauro is the third article in a three-part series about how Watson plays America’s favorite quiz show®.

Daily Doubles and Final Jeopardy! are often the most critical junctures of a Jeopardy! game; the amount wagered can make a big difference in a player’s overall chances to win. How does Watson decide on the amount?

Daily Double wagering

In principle, to compute the best Daily Double (DD) bet, a player must answer two basic questions:

(1) How likely am I to answer the DD clue correctly?

(2) How much will a given bet increase or decrease my winning chances when I get the DD right or wrong?

Match Play

The Watson-Jeopardy Challenge is spread over two games, with combined totals determining the winner. This style of play requires different strategies than a typical game. Final Jeopardy! of game one is analogous to “half time,” so requires different strategies by all competitors, compared to when game two is the last chance to win.

Humans are at best only able to make crude estimates of these quantities. By contrast, Watson uses advanced mathematical models that can answer both questions with far greater precision than humans can achieve.

To address the first question, Watson uses an “in-category DD confidence” model. Based on thousands of tests on historical Jeopardy! categories containing DDs, the model estimates Watson’s DD accuracy, given the number of previously seen clues in the category that Watson got right and wrong.

Watson tackles the second question by using a Game State Evaluator (GSE), a complex regression model that estimates Watson’s winning chances at any stage of the game, given the information set that describes the current game state (for example, the scores of the three players, the number of remaining clues, the value of remaining clues, and the number of remaining DDs).

The GSE was trained over the course of millions of simulated Jeopardy! contests pitting Watson vs. two simulated human opponents. The human opponent models in these simulations capture important statistical profiles of human contestants, such as how often contestants attempt to buzz in; how often they are right when they win the buzz; their accuracy on DDs and Final Jeopardy!.

Optimal wagering

By combining the GSE with the in-category DD confidence, Watson can compute an overall expected chance to win the game for any given DD bet. This analysis runs for every legal betting amount – from the $5 DD minimum, to its entire bankroll for a True Daily Double – to come up with an optimal amount. The calculation also uses risk analytics to trade off expected winning chances against the risk of a particular bet.

Watson’s resulting bet might seem unusual, in that it frequently may be far more aggressive, or far more conservative, than typical human bets. The amount may also take on non-round values (i.e., not an exact multiple of $100). Such values may make the arithmetic a little more challenging for the humans when computing their bets.

Final Jeopardy! wagering

In calculating a Final Jeopardy! (FJ) wager, Watson first needs to know if it is playing a single game or a two-game match [see Call out box: Match Play]. In the latter case, Watson will use very different strategies for game one and game two. The analysis for game one is similar to Daily Double analysis: Watson uses a statistical model of likely human bets, human FJ accuracy, and Watson’s FJ accuracy to calculate its expected winning chances for every legal bet. It then selects the bet giving the best risk-adjusted chance to win the match.

While there are no previously revealed clues in the FJ round, Watson does obtain evidence of its likely FJ accuracy from the category title. Given the title, Watson first computes several salient features via Natural Language Processing analysis. It then consults a “FJ prior accuracy” regression model, based on Watson’s performance on thousands of historical FJ categories, to predict Watson’s accuracy given the category features.

Wagering in game two of a match is similar to FJ in ordinary games. The predominant consideration is score positioning (first, second or third place). In some cases, the contestants may need to use strategic reasoning as in games like Rock-Paper-Scissors – predict the opponents’ bets, while taking into account the fact that the opponents are also trying to predict their bets.

Watson has been programmed with a library of known FJ strategy rules, such as Two-Thirds Betting and Shore’s Conjecture. The research team also added novel rules for some special situations which we discovered.[1]

Depending on the situation, Watson will either bet according to a suitable strategy rule, or it will run a real-time simulation to calculate the best bet, among all legal bets. For the match with Ken and Brad, Watson will also take into account the prize values for second place ($300,000) and third place ($200,000), leading to a different objective than simply trying to win the match.

--------------------------------------------------------------------------------

[1] One such rule in ordinary FJ applies when the leader’s score exactly equals the sum of the other two players’ scores, for example, if Watson has $20,000 and the two humans have $13,000 and $7,000. Watson would normally bet $6,001, to win by $1 when the second place player doubles her score. However, in this case Watson will bet $6,000 to tie for first place. The reason is that if Watson bets $6,001 and is wrong, it gives the third place player a chance to win by $1 ($14,000 to $13,999) if the second place player is wrong.

http://ibmresearchnews.blogspot.com/2011/02/watsons-wagering-strategies.html

[rah]

I disagree. Betting as little as he did was not a good game decision. With that type of lead, it should have more aggressive and put the game out of reach, especially if it had some confidence in that category. Betting less than 3% of your cash when you're leading by that much is really pretty wussy. I'm sure it would have been much different in a one day game. (that's a pretty straight forward calculation) My point is that it's game theory programming was too conservative for my tastes. Watson had the opportunity to put it out of reach but still hold a sizable lead even if he failed.

[Jon Miller]

It is basing it's betting on what will win, not on what will 'win big'. It doesn't value money as a player would.

JM

[Lorizael]

I disagree. Betting as little as he did was not a good game decision. With that type of lead, it should have more aggressive and put the game out of reach, especially if it had some confidence in that category. Betting less than 3% of your cash when you're leading by that much is really pretty wussy. I'm sure it would have been much different in a one day game. (that's a pretty straight forward calculation) My point is that it's game theory programming was too conservative for my tastes. Watson had the opportunity to put it out of reach but still hold a sizable lead even if he failed.

I'm sorry that your tastes are stupid.

[rah]

In a two day match a huge lead can make it virtually impossible for either opponent to catch him on day 2. And if the machine was truly taking into account

Watson tackles the second question by using a Game State Evaluator (GSE), a complex regression model that estimates Watson’s winning chances at any stage of the game, given the information set that describes the current game state (for example, the scores of the three players, the number of remaining clues, the value of remaining clues, and the number of remaining DDs).

Even an idiot could see that he was wiping the floor with these champions and there would not be much risk in more aggressive betting.