CTP Ratings (03-FEB-2001)

[Franses]

First, I would like to mention that I and probably everyone else here very much appreciate the work that quinns and Solver are doing. So, I think, quinns that we are not trying to wine to you at all but the effect the last ratings had on Birdman are well... impressive.

To the discussion I want to add that developing a rating system that is accurate (in the sense that the best player gets the highest rating) for CTP, using the power chart every 9th turn, is almost impossible. I do not know how things are in your case but when I play deity I am always one of the last on the power chart for at least the first 100 turns. And I always win if I want to. This means that the power chart just is not a good measure of the real position of the player in the game.

I know why the current system (using intermediate results) has been developed and I agree with the reasons behind it. However, expecting that this will result in the best players getting the highest ranking is, in my opinion, a false expectation.
Changing the current system now is, I think, to early. Perhaps we should just gather more data and decide then what to do. One thing that is plain clear for me is that we will have to alter the effect of wins and losses at the end of a game. Perhaps we should come to something like: during the game you can win or lose x rating points max while winning the game gives 3x rating points for the winner. 2x for the runner up and so on.

Franses

[This message has been edited by Franses (edited February 04, 2001).]

[quinns]

Franses,

Please refer to the Elimination Rule and Alien Life Project rule in the first thread of this forum. This already covers the "end of the game" that you mention. As far as the Power Graph goes... no one ever said it was completely accurate. It does not take into account Advances or the number of Cities a nation has. It only reflects CONSTRUCTED (and still living) BUILDINGS, WONDERS, AND UNITS -- that's it -- that's why we are always behind in the beginning against the A/I. You are losing points by building cities with your settlers and not gaining anything from your advances from science via gold, nor from population growth. (Refer to the CTP Multiplayer thread entitled POWER GRAPH INTERPRETED.)

We had a very long discussion about this in the beginning of the rating system and we all decided that THE POWER GRAPH WAS THE ONLY THING WE HAD. If you can think of a SIMPLE, ACCURATE, and FAIR method for rating someone in a game, please let us know. I really don't mind changing it to a good system, but I haven't read anything solid that challenges this existing system yet. Criticisms are easy... it is coming up with a real working system to replace it that is hard.

(edit: ... and thanks, Mobius, for the encouraging words!)

Quinns

[This message has been edited by quinns (edited February 05, 2001).]

[Keygen]

Man, you have fallen 4 points in one week!
This takes you from the 1st position on the 20th in one week!!!
And imagine that I am currently playing in four games where only in one game I am the last and this is the only rated game where I play in.
This could take me straigh down to the bottom of the charts in a few weeks!

I think the system itself does not forgive any of the low charts in a game.
If someone is winning and someone is losing the difference between them gets extremely large!
Think of two players playing in a rated game where the one player is slightly behind the other even if he wins the game he won't beat him in the charts .

Such big ups and downs to the positions looks anoying to me.

I have to suggest two things to the rating system:

1. Players only wins points. They do not lose any. This way players will only surpass others and not drop down to the bottom like a tomato.
2. By wining a game should significantly rise winner's points.

Two more issues for discussion:

1. Whether a player should get points on how many opponents is beating on the CTP charts or just the average of these "wins". This would eliminate the fact that some players who play in 2-3 rated games where they have high scores would beat the players who play in 1-2 rated games and have high scores too. I know that a player who is involved in many games and has high scores that has more chances to be truly the best but there is a chance a player involved in less games to be better than him.

2. The only time that a player would lose points should be according to the Elimination Rule only.

[quinns]

Keygen, Birdman didn't fall 4 points in one week. He fell 4 points in 4 months (since November 2000). Remember, Dutcheese was BACK RATED from November for THREE eight player game results, where he was last in each of them. That is a huge number of losses... 21! Don't jump to conclusions on changing a working system because of a single example of back dated adjustments. The Birdman was right, he shouldn't have agreed to Dutcheese being rated for him if it was an unfair position. I've already agreed to adjust his rating if Slamp agrees.

Quinns

[Keygen]

Quinns is right!
It is a hard job and he is doing it for us!

Don't break his... you know .

[quinns]

Thanks Keygen! I appreciate your support!!

[Keygen]

OK Quinns, it's your show.
Maybe I am wrong and I wish I am but just keep them in mind.
The next two or three months will be a test for the rating system.
And by choosing on which games we will be rated and which not will surely not show the true ability of a player. Except if he has inherited a nation in a bad situation as I guess BirdMan has.
Accually in that case the player shouldn't get rated at all! Even if the inherited nation is in a very good shape but the game is in a late stage.

[Franses]

LOL

I just wanted to support you!! I understand that you interpreted it differently. I wrote sentences as: "I appreciate your work", "I know why the current system (using intermediate results) has been developed and I agree with the reasons behind it" and "Changing the current system now is to early". We do not have sufficient data. So, I am almost completely behind what you said in your last post. Sorry that I have not been clear enough.

The only part that I deviate a bit was that I believe that we might have to think again later about the rating of the relation between intermediate- and end-results. This to overcome the effects the current system had on Birdman. How? I do not know yet. We need much more data before we can define a solution and perhaps we even won't be able to. Therefore my implicit statement was not to change the current system.

Hope this more clarifies my support for you.

Franses
[This message has been edited by Franses (edited February 05, 2001).]

[TheBirdMan]

I have been looking "behind" the ratings (one of the links) and it seems fair.

But I do have a proposal: NEVER-NEVER again rate a game backwards.

I will climb up again, be sure of that even if Quinns gives me a penalty for changing status in Monkey 2 to unrated.

As for turn 119 in DC and the actual position in turn 139, I'll wait to Slamp is back. Now I think back, I was only sub for Sophix from turn 119 to 124-125, so I really didn't change anything in the setup of the english those turns.
[This message has been edited by TheBirdMan (edited February 05, 2001).]

[Swissy]

Ok, sign me up.

I have beaten the game in single player at Diety. I am in Monkey1 and want that rated.

[quinns]

Sorry Franses. I guess I was in a defensive mood when I answered your thread. Thanks for your support also! I suppose I was getting tired of convincing everyone of the benefits of the system. I didn't mean for it to sound so harsh. Your points are well taken. I am always open to improvements to the system. But I am also careful of CHANGING a system in mid stream (without a very good reason) as this causes confidence to be lost in the system, (similar to the United States Legislature ).

Birdman, I have a tendancy to agree with you on this "back rated" problem that happened. If others agree with this "no back rated games" policy, from now on, then we will make that one of our rules. I'm sure you'll climb back to the top soon enough. Maybe this was a very good lesson for us all to be very careful about reporting results and becoming rated in games that are hopelessly lost.

Quinns

[Solver]

Franses,

Actually I think that developing an all-around good and completely fair ratings system is impossible for any game - is it CtP, football or chess. The only real measure to see if A is better then B - let them play a couple of games each vs. other and see it then.
Chess - even though Kramnik is now the World Champion, many and many people still consider Kasparov being the best chess player.
Football - France are World and Europe champions, but Brazilians still have a higher rating.
CtP - someone is behind on the powergraph, but in a very good position.

I have an excellent example of this in my game of R2_Islands, where I am the last on the powergraph. I have lost 2 units to barbarian hordes, and that might be what sent me down, but I am sending explorers and Settlers to another Island, while I'm sure there's someone who hasn't yet even built a ship.

The reason of developing this system, is, of course fun. I don't think one should be really upset if his rating goes down. TheBirdMan was in an excellent winning position on the ratings table, now he's much more lower. Does he cry? No. He keeps joking about it.

And, you say it might be possible to rate the game at it's end. Go to the PBEM Ladders thread and see the first post there - that was the origianal idea, but then the ratings would remain static for periods of at least two month, while now each week we are seeing changes. Static ratings would completely kill all the system, as it is.

However, I will greatly appreciate suggestions on how to improve the system, of course. Just my comments on the subject.

------------------
Solver - http://www.aok.20m.com

[TheBirdMan]

I agree

[Franses]

OK quinns. You were not the only one who misunderstood me I am just not much of a writer, I suppose. My conclusion is that we all agree so I will rest my case except for one major item.

Solver, I agree with you regarding the difficulty of developing a fair and accurate rating system BUT... you have gone down in my appreciation. Everybody knows that the Dutch have the best soccerteam. Why don't you?

Let's continue having fun,

Franses

[quinns]

I agree with Solver in his comments.

Swissy, you are now in the ratings... welcome! I will add you to the list this Saturday or Sunday.

Due to the high response (and understandable complaints) of the "Birdman Drop in Ratings" that surfaced, I believe the most fair way to handle this is to allow a one week period (until next Sunday) where anyone can switch to unrated status in any game WITHOUT PENALTY . If enough people agree with this (please respond here), then I will post a new thread stating this "reprieve".

Quinns

P.S. I just noticed that BILHQ has not been rated, yet was considered rated in CONQUER!!! Someone please contact BILHQ to get his initial rating. In the meantime, he has been changed to "unrated" in CONQUER.


[This message has been edited by quinns (edited February 05, 2001).]