Surprise attack bonus

[-Jrabbit]

This looks interesting. Thanks for starting this thread, AH.

* -Jrabbit posting his +1 just so it'll show up on "my threads" *

[DrSpike]

Even if you only want to establish the existence or not of the bonus my method is far superior. Ask your staff. But never mind......we have exhausted potential discussion I think.

I don't think the textbook gibe is fair........formulating the test procedure above is not trivial........so I posted it......not least to show you how it would be done by someone who actually has experience in this area, and that I wasn't criticising your formulation without proposing an alternative. (because it p1sses me off when people do that )

I admit your claiming experience in the area whilst disparaging mine annoyed me and made my responses prickly......I hope it is all in the past now.

Happy testing.

[rah]

yep, I'm not a fan when people stick equations under my face and say, see I'm right. I have books full of equations and I'm never right So sorry, if I jumped on you there.

But since you do have an interest and the skill, I'll post the raw data results here, and you can take it from there. (while I'm boasting that there is a surprise bonus while I'm waiting for the final analysis, since that's what market research firms are good at.)

RAH

[DrSpike]

Originally posted by rah

But since you do have an interest and the skill, I'll post the raw data results here, and you can take it from there.
RAH

Agreed.......though for the basic test all I need do is supply the decision rule, since we don't like arbitrary ones

I just did some back of the envelope calculations (I used the shortcut so they are at present a touch off, but not much). They confirm the need to analyse tests properly. It turns out that with exactly 50 repititions that 32 or more wins is sufficient to reject the null at the 5% level. This is the same as saying you would only observe more than 32 wins with no bonus 5% of the time.

18 (from 25) is *much* too strict a decision rule, since 36 or more wins from 50 repititions is only observed a fraction of 1% of the time. Consequently the probability of accepting the null when it is false (precisely what you wish to avoid) is needlessly large. I should have done this earlier......it makes my points far better than anything else possibly could. Hope you are convinced now.

[rah]

Yes the 18 was more based on playing experience and guessing how much 50% bonus would impact it.

Your example that 32 is necessary only to prove that there is some type of bonus, not taking into account how much. The bigger the bonus, the more wins will be necessary to prove it. (and this is assuming a 50-50 norm human vs human, which has yet to be verified even though i'm sure it will)

My stat guy asked me how much of a bonus I was trying to prove.

[Bloody Monk]

All this technical stuff makes my hair hurt

But, I'm sure all of us await , with great fascination, your conclusions.

On another note..."Let the word go forth from this time and place, to all the known worlds, that rah and DrSpike are two very smart guys."

Monk

[Scouse Gits]

Glad the flames have died down guys...

I guess it just shows the difference of approach between the academic and (dare I say it) the real worlds. We both need each other - but sometimes we resent it!

SG[1] - an academic

[DrSpike]

Originally posted by rah

My stat guy asked me how much of a bonus I was trying to prove.

Having returned from the local weekly gathering of the......uh....strategic gaming society I hope I can retain my usual level of clarity.

Yeah, if we knew the bonus exactly we could formulate the test the other way around, with the null as "there is a bonus". Since we don't the way around I suggested is natural.......alternative hypotheses are rarely exactly specified since it would lead to the test having very bad properties if the specification is even slightly wrong. Plus it is more natural to ask the data to prove the existence of the bonus rather than disprove it.

Hence what your guy was after was a measure of the power of the test (related to the probability of accepting the null when it is false).......which does depend on the specification for the bonus. Incidentally the power of my formulation has almost twice the power of your original formulation.......I checked that earlier in case I needed it to settle any arguments.

Any results yet?

[DrSpike]

Originally posted by Scouse Gits
Glad the flames have died down guys...

I guess it just shows the difference of approach between the academic and (dare I say it) the real worlds. We both need each other - but sometimes we resent it!

SG[1] - an academic

There weren't any flames......I can't usually be assed to insult people........I just bridled a little when Rah assumed I was bullsh1tting and didn't know what I was talking about......when in fact I do.

I admit when Ming made his post about Rah being a professional and hence knowing better that my keyboard came close to extinction. All will be pleased to know it is safe and sound.

If I am perfectly honest (and I am ) Rah could carry out this test by brute force (the way most stats that 'real world' people do is) with little loss of precision.......it would just take longer. And if the result happens to be semi-conclusive that is when an academic with a more rigorous procedure will win every time.

[Six Thousand Year Old Man]

I hate to say this, but - why use Warriors?

Warriors are A:1/D:1 - if there is a bonus, it's going to be pretty hard to see it, if rounding is used by the Civ program. For example, a 20% bonus is significant if you're attacking a Riflemen with a Dragoons - but the .2 bonus may be rounded to nothingness when using Warriors to test for it.

Sorry if that point has already been considered I hate to get involved in a high-level statistical discussion, anyway - I just use stats to confuse my clients

[rah]

warriors, for two reasons.
One, it takes faster to set up on a random start, I'll just do it on a small world and it won't take war and i more than 10 minutes to line em up, and i won't have to use a scenario that could bias the results.
Two, my initial intent was just to prove there was a bonus. so any equal units would do. IF there is one, we'll work on the amount.

With Spike, we're fine, we were just feeling out each others backgrounds, and I took early offense to a few of his choice of words, and comments about not needing a control group. I overreacted and fueled the flames. My initial thought was a simple test, but it's kinda gotten bigger. I believe we're fine and will share on the discovery. I'm glad there's a statistician available since I really don't like using my staff for my personal concepts(even though I know they'd be happy to do it) This way we can keep the entire thing in house.

RAH

And everyone showed up early to play so I didn't get a chance to test it, War and I will do it tomorrow before we play.

[rjmatsleepers]

Originally posted by DrSpike


Sorry but it isn't.

Having demonstrated my statistical naivity once , I probably shouldn't post again on this subject but ...

Surely we only need to test the hypothesis that there is a "sneak attack bonus" that is large enough to increase the attack strength by 1 (against the null hypothesis that the bonus is not big enough to increase the attack strength). If you're using warriors to run the test, that would mean changing a 50% win chance into something pretty close to 100%. I think either Dr Spike or Rah's stat guys could show that the sample size needed for that test is pretty damn small.

(RJM retires behind SDI shield in case of nuclear attack from any direction)

RJM at Sleepers

[DrSpike]

DrSpike uses nuclear weapons!

SDI defence blocks attack!

Bugger.

Hehe, um, as to your query I don't understand why we should test an increase in attack strength of one.

The way hypothesis testing is done is to have 2 hypotheses you wish to test, and derive the distribution for your statistic (in this case its the proportion of wins). Then you can calculate whether or not the value from your sample (that Rah is going to provide) could reasonably have come from the postulated distribution. Your hypotheses are a little rough.....they look tricky to test to me.

Of course it is natural to want (as Rah did) the test to be stricter the bigger you believe the bonus to be....but unfortunately this is the wrong reaction........all that a bigger bonus does it make the job easier by increasing the power of the test.....since the probability of rejecting the null when it is true is fixed by the researcher anyway. This also shows why large sample sizes are not necessary for this test if the test procedure is well formulated.......though they are if there is no formal test procedure.

As to sample size all we really need is to be sure that the statistic has a normal distribution.......and for that we need in excess of 30. I am perpetually paranoid so I said 50.

[SCG]

Originally posted by Scouse Gits
Glad the flames have died down guys...

I guess it just shows the difference of approach between the academic and (dare I say it) the real worlds. We both need each other - but sometimes we resent it!

SG[1] - an academic

You know - Maquis de Sodaq never did run for trade advisor

*ponders whether it would still be worth while trying to set up a ticket stand for the 'heated debate'*





this joke brought to you by the new Democracy Game Trade department

[rah]

Originally posted by SCG
this joke brought to you by the new Democracy Game Trade department

Don't you guys get enough posts in that forum?