Computer to play Jeopardy against past champions

[rah]

I'm sorry that your tastes are stupid.

So much for a civil debate. But I shouldn't have expected anything different.

[Lorizael]

But if he's wiping the floor with them then there's no need to be aggressive. If playing conservatively will produce victory, why risk losing by betting a large amount?

[rah]

It's called the killer instinct in gaming. And based on what I observed Watson wasn't programmed with it correctly. My opinion.

Too conservative. I expected different.

[Lorizael]

There's a point at which more money negligibly affects your chances of winning in Jeopardy. At that point, large bets are stupid.

[Asher]

It was quite interesting to watch. It all depended on the type of question. If it was straight up facts, Watson was untouchable, but some questions posed a big problem.
The second day's final question was the perfect example.

Spoiler:

The question was what major US city's largest airport was named after a world war II hero and it's second airport was named after a famous world war II battle. Jennings and the other player had no problems with knowing the answers was Chicago for O'hare and Midway but Watson couldn't make the multiple connection and Answered with TORONTO ???? (how it picked a Canadian city is beyond me)

But what was most interesting was that on every question it showed watson's top three picks along with it's confidence level. SOme of the secondary answers were hilarious. And when watson would get a daily double, the amounts he bet were quite bizarre.

It's been explained why it didn't get the right answer there. Chicago was actually a very, very close second to Toronto in its guesses.

It was trained to not give much weight to the category name, as historically they've proven to be slightly misleading to the computer. If the question was "This US city..." it'd have gotten it right. It also got confused because Toronto has an American League baseball team, and there is a Toronto in the USA as well. Toronto also has an airport named after a World War hero (Billy Bishop).

Could still use some refinement, but the guess wasn't as far-out there as many people seem to expect when all the computer can do is correlate data.

[Lorizael]

Neat. What's your source for this information, Asher?

[rah]

Actually Asher, I was just pointing out the type of question that was more challenging. The straight forward knowledge questions were a snap for it. It took multiple combinations to stretch it.

And Lori, you're wrong. A larger amount of money on lead significantly reduces the % of someone catching up, all the way to the point where there isn't enough money left in play to catch up. The programmers said that they took that into consideration.

But again, you're satisfied with a conservative game plan here, and I would prefer a more aggressive game plan.

[Asher]

http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-day-two-the-confusion-over-an-airport-clue.html

Well, Watson beat the human champions in the first game of the Jeopardy! face off between man and machine, with a score of $35,734 to $10,400 for Brad Rutter and $4,800 for Ken Jennings. But Watson’s developers were puzzled by his flub in the Final Jeopardy! segment. The category was US Cities, and the answer was: “Its largest airport was named for a World War II hero; its second largest, for a World War II battle.” The two human contestants wrote “What is Chicago?” for its O’Hare and Midway, but Watson’s response was a lame “What is Toronto???”

How could the machine have been so wrong? David Ferrucci, the manager of the Watson project at IBM Research, explained during a viewing of the show on Monday morning that several things probably confused Watson. First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Watson, in his training phase, learned that categories only weakly suggest the kind of answer that is expected, and, therefore, the machine downgrades their significance. The way the language was parsed provided an advantage for the humans and a disadvantage for Watson, as well. “What US city” wasn’t in the question. If it had been, Watson would have given US cities much more weight as it searched for the answer. Adding to the confusion for Watson, there are cities named Toronto in the United States and the Toronto in Canada has an American League baseball team. It probably picked up those facts from the written material it has digested. Also, the machine didn’t find much evidence to connect either city’s airport to World War II. (Chicago was a very close second on Watson’s list of possible answers.) So this is just one of those situations that’s a snap for a reasonably knowledgeable human but a true brain teaser for the machine.

The mistake actually encouraged Ferrucci. “It’s goodness,” he said. Watson knew it did not know that right answer with any confidence. Its confidence level was about 30%. So it was right about that. Moreover, Watson has learned how the categories work in Jeopardy! It understands some of the subtleties of the game, and it doesn’t make simplistic assumptions. Think about how Watson could be used in medicine, as a diagnostic aid. A patient may describe to a doctor a certain symptom or a high level of pain, which, on the surface, may seem to be an important clue to the cause of the ailment. But Watson may know from looking at a lot of data that that symptom or pain isn’t the key piece of evidence, and could alert the doctor to be aware of other factors.

(By the way, there are many fields where Watson could help out. IBM general counsel Robert Weber describes how Watson might be used in the legal profession in a guest blog posting on The National Law Journal Web site. Anne K. Altman, general manager, Global Public Sector, talks about how Watson could be helpful to government in a posting on Government Technology magazine’s blog.)

Another encouraging sign: Watson bet intelligently, just $947, so it still won the game by a wide margin. “That’s smart,” Ferrucci said. “You’re in the middle of the contest. Hold onto your money. Why take a risk?”

Watson may not have much of a sense of humor, but Ferrucci sure does. He wore a Toronto Blue Jays jacket to the Jeopardy! viewing.

–

Here are some explanations of how Watson plays J

Here’s how Watson knows what it knows from Jon Lenchner:

http://ibmresearchnews.blogspot.com/2011/02/knowing-what-it-knows-selected-nuances.html

Here’s a post on Watson’s wagering strategies from Gerald Tesauro:

http://ibmresearchnews.blogspot.com/2011/02/watsons-wagering-strategies.html

Here’s some info on how Watson sees, hears and speaks from Dave Gondek:

http://ibmresearchnews.blogspot.com/2010/12/how-watson-sees-hears-and-speaks-to.html

[rah]

It's been explained why it didn't get the right answer there. Chicago was actually a very, very close second to Toronto in its guesses.

I don't remember them showing the alternative answers and %s like they did during the regular portion of the game. Did I miss it or did you see this somewhere else. What were the percentages?
I did like that his answer was followed by ?????? showing it's confidence level to be quite low.

X-post

[Lorizael]

...all the way to the point where there isn't enough money left in play to catch up.

Exactly. If I have $100,000, and the next closest person has $200, and there's only $5,000 in clues left, then more money has a negligible (zero) impact on my chances of winning. This situation isn't as extreme, but what it means is that Watson still believed it was at the level where more money has a negligible impact on winning. Given that it knows the value of all the clues in the next game and given that it knows the historical chances of winning based on various bets, my guess is that it is a better gauge of analyzing its winning percentage than you are.

[Provost Harrison]

It's called the killer instinct in gaming. And based on what I observed Watson wasn't programmed with it correctly. My opinion.

Too conservative. I expected different.

It's a slippery slope. Now you give it killer instinct. Next it is launching your nuclear arsenal on everyone and marching machines around that have more than a passing resemblance to a certain former Californian governor

[rah]

my guess is that it is a better gauge of analyzing its winning percentage than you are.

No, it is based on the factors the programmers took into account. The machine is only as good as it programming. CIV V anyone?

and 30K is not an insurmountable lead.

[Lorizael]

No, it is based on the factors the programmers took into account.

Yes... which includes its odds of winning the game with particular amounts of money.

and 30K is not an insurmountable lead.

And neither is 60k.

It has to weigh the risk of losing if it bets a good chunk of its money and is wrong against the increased chance of winning if it bets a lot of money and is right. If the increase in winning percentage from having $60,000 is not significantly higher than the winning percentage from having $30,000, then it's not worth the risk.

Consider that its chances of winning with 30k may be 90%, and with 60k, 95%, but with 0k (or some figure roughly in the same area as the other two competitors), 60%. (These numbers are pulled out of my ass.) If that's the case, why risk reducing your odds of winning by 30% for the chance of increasing them by 5%? (And I mean increasing and decreasing percentage points, not actual fractions.)

[rah]

As you said, you pulled some numbers out of your ass, making it your opinion. My point is that judging whether it's worth any risk to upgrade your chance of winning from 90 to 95 is still a judgement call. Or even from 50 to 55. The programmers provide an algorithm for the computer to weigh it. In my opinion the machine was too conservative and I would have programmed it to be a touch more aggressive. It's a judgement call based on what I observed. I would be willing to bet that generally you're more conservative than I am and that helps to feed our impressions of us watching the same event.

[Ogie Oglethorpe]

It's called the killer instinct in gaming. And based on what I observed Watson wasn't programmed with it correctly. My opinion.

Too conservative. I expected different.

Who are 3 people that have never been in my kitchen?

22000 big ones.

Cliff Clavin.