I like the way they handle this, and I thought it might be of interest to other people, since most of us wind up judging at one point or another (IFComp, Spring Thing, etc.)
Not a bad rating system - I’m not quite sure what I think off-hand about the ‘design, execution or basic functionality’ trifecta, but in general it comes pretty close to what I’d use.
I’ve been writing up my own comp-rating guide, largely because there are communities of practice in which anything below an 8 is an insult and 10s are routine, and I want to avoid people thinking that a 5 means ‘I hated your game’.
Hmm! I think I would like to adapt (with attribution) both your ten-to-one and Polygon’s as “Here are two examples of well-thought-out and defensible scoring rubrics” for a forthcoming “Guidelines for Judges” IFComp.org page, if that’s all right with you…
I wonder if putting something like this on the official page will act as a deterrent to some potential judges. Even if you put it up with the wording you have here, some people might think they have to come up with something equally well-thought-out and defensible, and then just decide to not bother. Has there been some problem in the past with people being assumed to judge based on something less well-thought-out? Would putting this on the official page actually deter anyone who is out to just vote 1 for every non-parser game, for example? I assume the goal is to have as many judges as possible, and this could result in fewer judges.
I agree it’s a possibility. I’d suggest something like “You don’t need any particular credentials to judge games. You don’t need to have a degree, and you won’t have to defend your score before a panel, or worry if your method is as good as the other judges’. How you judge and score things is private. Don’t worry about if your scores average to 5.5, fit a bell curve, or are uniformly distributed from 1 to 10. Don’t worry if you react badly/emotionally to one game, and don’t feel you have to judge all games. However, first-time judges or people looking to add a degree of rigor may find the following guidelines helpful:”
(and then allow the player to open up collapsible HTML)
Basically, if I don’t really like a game - like, 7-8 territory - I’ll probably end up spending more time talking about its flaws than its strengths. Often this is because the things it succeeded at are straightforward enough that they don’t bear much discussion - things like a lack of bugs, competent implementation and inobtrusively smooth prose are all good things that improve game experience, but it’s hard to write more than a sentence about them in a review.
But there’s a lot of reaction space in between strong enthusiasm and hatred.
For what it’s worth, in the poetry slam community, for whom scoring is a big deal (albeit handled very differently), they don’t require judges to be calibrated to a similar scale so long as they are internally consistent within their personal systems. (One night I saw an old friend entrusted with judging duties and found him giving extraordinarily low but, in my mind, fair scores. I asked him what was his secret, and he told me: “Every poem starts with a full score of 10, and then I deduct a portion of a point every time I hear the words ‘I’, ‘me’ or ‘my’.”)