Tweet
The Column 36
By Andrew Livings
April 10, 1999
The purpose of a review is to give an indication to readers of how good something is, highlighting both good and bad. But there is a tendency of reviewers to summarize their review with a score, or rating, as a lazy means of expressing approval or disapproval. This is standard practice now, and even expected behaviour, but we should always be aware that a score is an accessory of a review and not the purpose of a review. This is the point that is often lost on people who are driven to award "scores" for the sake of scoring, and it is this practice that I wish to discourage.
Many Civilization 2 sites - excepting this one - do this for all submissions and especially for links to other sites, usually by awarding little stars like we used to get in junior school. Site, scenarios, mod-packs, maps; all must be labelled, perhaps because originally they were carefully considered judgements, but later? They would need a label simply to fit in with previously labelled items. But who asked these people to decide that label, which is passing a judgement on that work that they have no right to do. Unless, in very small print at the bottom, you read the disclaimer: "the scores here are the personal opinion of the web-master and should not be considered definitive", which in fairness should be at the top in quite large print.
Commercial products are a different category to the world of Civ2, because companies submit their products for review by magazines as both a means of publicity and endorsement. But with Civ2 we are all producing add-ons in our own time for no reward. It is in our interests to publicize and receive warm words of thanks or congratulations if they're any good, but it is unfair for these add-ons to be submitted to the "scoring" method that passes for review on many sites.
An exception to this practice is of course the Scenario League, whose members will undertake to review your submission with the purpose of constructive feedback. And how do they present their findings? You guessed it: by a "score". In fairness they have established a code for reviewing, and the "score" is derived from an objective assessment of four areas. But the reviewers' assessment of these criteria is still highly subjective, indicated by the variances in the "score". And is the purpose of the League to review or to rate? If you submit for review, you then also submit for rating; they are inseparable where this is concerned.
To summarize:
We may also wish to consider the other journalistic principle of "right of reply". If designers submissions are scored without their prior approval, or even with, they should have the chance to counter an unfavourable review.
By Andrew Livings
April 10, 1999
The purpose of a review is to give an indication to readers of how good something is, highlighting both good and bad. But there is a tendency of reviewers to summarize their review with a score, or rating, as a lazy means of expressing approval or disapproval. This is standard practice now, and even expected behaviour, but we should always be aware that a score is an accessory of a review and not the purpose of a review. This is the point that is often lost on people who are driven to award "scores" for the sake of scoring, and it is this practice that I wish to discourage.
Many Civilization 2 sites - excepting this one - do this for all submissions and especially for links to other sites, usually by awarding little stars like we used to get in junior school. Site, scenarios, mod-packs, maps; all must be labelled, perhaps because originally they were carefully considered judgements, but later? They would need a label simply to fit in with previously labelled items. But who asked these people to decide that label, which is passing a judgement on that work that they have no right to do. Unless, in very small print at the bottom, you read the disclaimer: "the scores here are the personal opinion of the web-master and should not be considered definitive", which in fairness should be at the top in quite large print.
Commercial products are a different category to the world of Civ2, because companies submit their products for review by magazines as both a means of publicity and endorsement. But with Civ2 we are all producing add-ons in our own time for no reward. It is in our interests to publicize and receive warm words of thanks or congratulations if they're any good, but it is unfair for these add-ons to be submitted to the "scoring" method that passes for review on many sites.
An exception to this practice is of course the Scenario League, whose members will undertake to review your submission with the purpose of constructive feedback. And how do they present their findings? You guessed it: by a "score". In fairness they have established a code for reviewing, and the "score" is derived from an objective assessment of four areas. But the reviewers' assessment of these criteria is still highly subjective, indicated by the variances in the "score". And is the purpose of the League to review or to rate? If you submit for review, you then also submit for rating; they are inseparable where this is concerned.
To summarize:
- Ratings are labels. To repeat: as a summary, they are an accessory to a review. They should not take the place of a review. It is one thing to "recommend" a product, or even award it the title "game of the month", but it is another to award it a score that endures for much longer than that month.
- Ratings are relative. Scores are relative to other products also available at that time, which become dated later when design techniques become more advanced and lead to better products. But the scores are never revised.
- Ratings are judgemental. Scoring itself is a judgemental process, which leads in turn to rankings and hierarchy: Product A scores higher than product B and is therefore better. How many people will then look twice at product B?
- Ratings are subjective. Reviewers cannot be free from a personal opinion is their assessment of how good something is, which may be influenced from first impressions and presentation.
- Ratings are fixed. Products can be submitted over a period of time, and their score will compete with previous submissions in the artificial hierarchy that has been created. We all what know the level of add-on building was like before Fantastic Worlds and how they would be received if submitted now. We may be tempted then to dismiss all Civ2 add-ons before mid-1997 as a waste of time, but there were some gems then which we'd dismiss as well. A balance cannot be drawn so long as they have their "score" carried alongside.
We may also wish to consider the other journalistic principle of "right of reply". If designers submissions are scored without their prior approval, or even with, they should have the chance to counter an unfavourable review.
Comment