Surprise attack bonus

[DrSpike]

Oh and as for the independence query........I take your point, because I expressed myself badly. Check out the info: combat thread for the real formula, and to see why the results from any one combat isn't bog-standard binomial.

[rah]

I figure the test will include 4 combats a turn.
Human vs. Human.

4 warriors against 4 warriors all on neutral terrain.

1. warrior 1 surprise attacks enemy warrior1
2. warrior 2 attacks enemy warrior2.

make peace.

3. warrior 3 surprise attacks enemy warrior 3
4. warrior 4 attacks enemy warrior 4.

reload, and repeat all 4 attacks for a minimum of 25 times.

if the results aren't obvious in 25 rounds I'll be shocked.
I'll bet, on the first surprise attack, that the attacking unit will win 18 or more of the 25 encounters, and 22 or 23 won't surprise me. I'll be real interested to see if the results of attack 3 match attack 1 or if they're consistent with 2 and 4.

RAH

[DrSpike]

Originally posted by rah

if the results aren't obvious in 25 rounds I'll be shocked.
I'll bet, on the first surprise attack, that the attacking unit will win 18 or more of the 25 encounters, and 22 or 23 won't surprise me.

RAH

The point is you have picked 18 completely arbitrarily, with no idea what the probability of observing 18 would be even without a bonus.

[DrSpike]

Oh and there is absoutely no point to providing an attack whilst at war for control purposes.......since we have perfect knowledge in that state anyway.

[rah]

No to both posts.
18 was not quite so arbitrary. I'm assuming a similar bonus as barbs attacking humans at raging (50%)

I using a control because I'm betting that EVERY test previous was not done Human vs Human, but Human vs AI. I'm not expecting any difference, but why take any chances?

RAH

[DrSpike]

18 is arbitrary, unless you have calculated the effect of the number of repititions on the variance in the proportion of wins within the sample......which you haven't.

And the second point was the probability of a warrior beating another with no bonus can be derived exactly from first principles......there is no need to sample this probability.

But whatever......it's your test, do it whatever way you feel is best.

[rah]

18 was based on assumptions and years of experience so it's more of a swag. Researchers find it hard not to anticipate/reason a possible result prior to the test.
Why don't you make a guess and we'll see who's closer.

WHAT PRINCIPLES. Those have all been developed and tested human vs AI. Those principles have never been proven to be valid for human vs human. NO ONE has ever published a test on human vs human. Again, my guess is that there will be no difference, but until it's tested, it's just another assumption.

RAH

[DrSpike]

Originally posted by rah


18 was based on assumptions and years of experience so it's more of a swag. Researchers find it hard not to anticipate/reason a possible result prior to the test.
Why don't you make a guess and we'll see who's closer.

RAH

Your missing the point. Say the warrior wins 17 - what will your conclusion be? And what about 19? And the absolute critical point is that the number you choose as your cutoff point for evidence of a bonus effect must depend on the number of repititions.

Somehow I thought you'd be more open to others' expertise in this area Rah.

[rah]

I'll wait for the result before I crunch the statistics.
The 18 was an educated guess. Previous experience is the entire reason why I'm doing the test. People have said there isn't one, and experience tells me there is. There's a big difference between arbitrary and educated guess.

I've worked in Market research for years. I have programmed and analyzed a considerable statistical runs. Your comments lead me to believe that you don't have similar experience in this area. (based on your comments about a control not being necessary)
I'll be more that willing to listen to a statistical expert when one shows up.

[Ming]

Considering that rah does research for a living... and that he's a professional at it.. I think he's the one with some expertise

[rah]

And, if it's 17, i'll probably test more cases. If it still remains 17 after that, it will still prove my theory that there is a suprise bonus. It just not be the same bonus that I guessed. And no, I didn't run the numbers to see how many observations were necessary to get it into the 95% confidence level. But I did ask a more qualified statistician here at work to run it and let me know what it would be. When he gets back to me, I'll post it. But I'm going to start testing prior to his answer because my curiosity is getting the better of me.


RAH

[DrSpike]

I build and estimate cutting edge econometric models for a living. I have a ph.d in econometrics (statistical economics). However, screw that, your comment about my expertise is not what bothers me. What bothers me is your reaction to my suggestions........it really doesn't seem in keeping with the picture I have built up of you.

Still, because I respect you, one last try:

The problem with the control you propose here is that it itself is noisy from your proposed sample.........a far better control would just be to use the formula we know holds (and the formula for bog standard comabt is the same for MP, we can be virtually certain of this - it is just the bonus modifiers that may be different against a human rather than an AI).

As to the arbitrary 18, the point is not that your educated guess is a bad one, but that you need a concrete decision rule. Your statement:

"I'll bet, on the first surprise attack, that the attacking unit will win 18 or more of the 25 encounters, and 22 or 23 won't surprise me. I'll be real interested to see if the results of attack 3 match attack 1 or if they're consistent with 2 and 4"

Now 'consistent' is the key bit........how much of a divergence are you going to allow before it affects your conclusions? Do you know the probability of the warrior winning more than 18 even if a bonus does not exist. No, you don't. Do you know the probability of observing 17 when a bonus does exist? No, you don't. Both of these are ways your decision rule fails.......now potential failure in hypothesis testing is endemic, but you have to know the associated probabilities or results are just glorified junk.

It's funny, I always stay out of these stats debates, because it is far easier to let the person do it by brute force, with a badly constructed experiment but lots of repititions than try and explain a more cogent testing structure. But you said you were going to talk to stats people about confidence, so I chirped up, and thought you would be receptive.

Still, happy testing........hope it goes well.

[rah]

I guess I overreacted to your belief that human vs humanl combat need not be tested and that previous ai vs human results would be valid despite never having been tested. Hence my desire to have a control. (but I do agree that they will probably be the same, but paranoia when testing is never a bad thing)

And your comment that 18 was arbitrary. It was not.
An educated guess is quite different than arbitrary.

And to your nitpicking on 17 or 19. WE DON"T KNOW WHAT IT"S GOING TO BE, that's why were testing.

The initial hyposthesis is that there is a difference.

And I'll do the statistical run, AFTER the test. Having data has been known to make it easier.

Since the testing will consume some time, I wanted a statistician to give me a ballpark of number of observations needed so I wouldn't not get enough or waste my time doing hundrends.

If I overreacted, I apologize, but I recommend that in the future, if you don't want people to get defensive, don't start by attacking there initial position distorting what they said. And your use of abritrary and misinturperting my hypothesis (there is a difference, the amount of difference was just a guess) was exactly that.

RAH

[DrSpike]

Originally posted by rah

And your comment that 18 was arbitrary. It was not.
An educated guess is quite different than arbitrary.
RAH

Again, sure you used knowledge to select 18, but you don't know any of the properties of that decision rule or you would have posted them to shut me up by now. Maybe arbitrary is a strong word, but it's not far off.

I doubt anyone really cares, but this is how I would do it. Far from being clueless in this area as you suggested I do this (well this is analogous to the most basic econometrics I would do) every day, and lecture other people at undergraduate and postgraduate level on how to do it.

I would use 50 repitions (or 25*2 as you suggest is the same, as long as making and breaking peace gives you the same state as the initial one before the civs have met). I would derive the proportion of expected wins for the attacking warrior without a bonus. Your null hypothesis is then.

H0: p hat = derived number

with

H1: p hat is not derived number.

Then you just derive the variance of phat in your sample of 50......it is just {p(1-p)}/n, where n is the number of repititions Under the null the distribution of phat is normal (from various statistical theorems) with a mean of 'derived number' and a variance as above. You then choose a level of significance, which also fixes the probability of you rejecting the null when it is true.

You then compare the standard normalised value for the proportion of wins you observed in your testing with the associated critical value from the normal distribution, which tests the null at the 5% level, the level you said you wanted.

Easy. Concrete. And as the icing you can derive the power of the test (related to the probability of accepting the null when it is false) under the assumption of a bonus of 50%. YOU CAN EVEN DERIVE THE MOST LIKELY SPECIFICATION FOR THE BONUS GIVEN THE DATA.

Please feel free to show this post to your stats guys if you doubt my expertise as you said above. Any competent statistician would carry out the test the same way.

[rah]

Originally posted by DrSpike
I would use 50 repitions (or 25*2 as you suggest is the same, as long as making and breaking peace gives you the same state as the initial one before the civs have met). I would derive the proportion of expected wins for the attacking warrior without a bonus. Your null hypothesis is then.

I'm doing 25*2 to test if the different states make any difference. (can you get a second suprise bonus) But I'm glad you mentioned that because I was going to just line up 4 warriors against 4 warriors, now after reading this, yes the first combat must be run right after the initial contact notice, So I'll start with the four warriors 1 square away as the intial base point (i probably would have remembered that when I set it up but I might have wasted time ).

As to your statitiscal analysis. Yeah I could have copied crap from my SAS book, but I have statisticians working for me and their expertise there is greater than mine. I have never once claimed expertise there, just understanding. My expertise is the test methodology and data collection/manipulation. Analysis comes after the data is collected. And if you want, I'll give you the raw data and you can save my staff some effort

But, if the result is > 18, I'll claim there is a bonus prior to the stat run, but I'll wait for the analysis to tell me how much and how confident it is. My initial intention was too disprove all of those that claim that the sneak attack bonus is a myth. How much has always been secondary.

And the difference in win percentage is all MP players are really interested in so they can develop some rule of thumb guidelines for on the fly attacks. When the clock is ticking all you really have time for is hmmm 4attack vs 2def, good, or 4att vs 4 def, I'd better have more units attacking than he has defending. or 4att vs 6def, I'd better have more than twice the units attacking.

RAH