So I didn’t see anything that looked like groupthink or bias in this year’s comp - can you elaborate on what you see as a problem here? How do you think it affected the comp results this year?
It looks to me like the top 20 this year was a good mix of new and old faces, so I’m not sure what the problem is here?
And I’m confused by your confusion. If the blurb says eg it’s a dating sim I may know it’s something I don’t like; if I have no idea what kind of game it is from the blurb I can probably figure it out after a few minutes of playing it.
And none of this is the “implicit reason” why I think the judging period is too long. I’m skeptical that either a longer or shorter period would have any significant effect on the fairness of the ratings.
I think the current guidelines already cover both situations quite well:
If you are unable or unwilling to play a certain IFComp for any reason – it’s Windows-only and you’re on a Mac, for example, or you simply ran out of time before you had a chance to get to it, or the game advertises itself as a heartfelt paean to fish sticks and you really don’t like fish sticks – please don’t rate that game at all. Just leave its rating on the ballot screen as None.
(If you try in good faith to play an entry but find it unplayable for some reason inherent to the entry itself, then that’s a different matter, and one you can choose to reflect in the score you assign that game.)
There were several games this competition that I started but didn’t finish. Some of those, I didn’t finish because I raqequit them over bugs or horrible spelling/grammar or other signs of poor craftsmanship, and I gave those bad ratings. Others, I didn’t finish because they just weren’t my cup of tea (including the 3rd-place finisher, Forsaken Denizen) and those I left at None.
I definitely would not want a shorter judging period. I haven’t really been aiming for completionism since back when the number of entrants per year was like 20, but I played about half the games this year and still failed to get to multiple games that I was actively interested in and think I will probably enjoy once I finally play them. People have health issues and busy lives and so on and so forth, so it’s really not just the people trying to play every single game who benefit from the extra time.
Just to take a stab at this, if part of the concern is that a long-time author with an audience of folks who enjoy his games won, it’s maybe worth pointing out that the last time Chandler Groover entered the Comp he came 59th 
Yes, I think this is the best counter argument. My argument is basically just that I personally find it kind of annoying.
I’ve definitely refrained from rating games that I just don’t understand, or that trigger me negatively in some very personal way that I know is unfair to the author. I’m a good judge of whether or not I’m being fair to the author in whether or not I should rate the game. But that’s my metric and I can’t impose it on anyone else.
For the record, I hate the idea of a smaller judging period. I also hate the idea of enforced pseudonyms. Why would we punish successful authors? I’m living proof that you can do really well in IFComp without any authorial track record, and a new author placed in the top 10 this year. If that’s not evidence that the community is generally judging on merit, I don’t know what is.
Thank you for this framing! I agree it feels premature to debate solutions without identifying the problem they’re meant to solve, and if a problem is real it should be visible in the results by now. (If folks are worried about raining on someone’s parade, I don’t think anything relevant to the issues under discussion have changed since last year’s results, say.)
Two other quick thoughts:
As someone who just kind of showed up out of nowhere to participate in reviewing, I definitely would not have gone to the effort of playing and reviewing the games without the community forum aspect. I found it a really engaging social and creative space that in my opinion sharpened, rather than dulled, my ability to evaluate the submissions.
I’ve placed 2nd, and I wouldn’t even re-enter if I was forced to use a pseudonym Not that I’m planning re-entry in the near future (“Oh, Matron!..”).
-Wade
I think it’s a problem whenever a judge goes into a game with preconceptions that aren’t a function of the game’s content. By which I mean, for example, that if the game’s blurb categorizes it in a particular genre and the judge goes in with foreknowledge of the conventions of that genre, that sort of thing is fine; but I don’t want it to ever enter my mind when I’m judging a game that its author is well-known or that I’ve seen a lot of buzz about it. It’s bad whenever this happens, regardless of whether that badness evidences itself in the competition results. But since you’ve asked me to point to such evidence, I’d point out that several games are outliers for having gotten many more reviews than others. These outliers include both first and last place, so clearly this attention isn’t pushing their scores in any one consistent direction, but it’s obviously having an effect. Ideally, there wouldn’t be very much variation in how many reviews each game gets, and whatever variation exists would be mostly explicable through the game’s blurb and metadata (e.g. maybe there’s more interest in some genres, or shorter games can be reviewed with less effort).
I have no prior experience seeing the innards of a competition of any kind, but it seems to me that the best way to overcome individual bias would be to get as many judges as possible, and anything which makes the process harder or less communal/fun to engage in might be counter-productive on that front.
Note that the last-place entry was submitted under a pseudonym, so there was no name recognition going on there.
Honestly, I think if it had been submitted under a pseudonym it would have done exactly as well. In a world where most people don’t have time to play all 67 games (I know I sure don’t) word of mouth is going to be a big part of how people select things to play. I don’t think the number of ratings The Bat got was “hey, Chandler Groover!” so much as “hey, this is a great game!” Like yes, I’m sure it influenced how quickly The Bat took off, but word of mouth would have boosted it regardless.
First and foremost, judges and reviewers are here to have fun. You can rules lawyer everything to be perfectly fair if you want, but I think what you’re suggesting would do more to turn authors and judges away than it would be to make anything more “fair”.
Worth noting that this is actually how IFComp used to work. I definitely think it’s more fun now that we can talk about the games while we’re playing them. Maybe it’s not the most scrupulously fair way of doing things that anyone could ever conceive, but this isn’t, like, the Nobel Prize for Literature, it’s a competition for incredibly niche amateur games. It’s not that serious.
Edit: Which, to be clear, is not meant as, like, “haha, why do IFComp results matter to anyone?” (which would be a little ungrateful under the circumstances), just, in this kind of hobbyist context I think whether a competition is a fun experience for the community matters more than whether it’s carefully engineered to remove all possibility of bias.
I don’t want it to ever enter my mind when I’m judging a game that its author is well-known or that I’ve seen a lot of buzz about it. It’s bad whenever this happens, regardless of whether that badness evidences itself in the competition results.
I’m not convinced that’s a major problem. When a game gets a lot of buzz because a lot of people liked it, it’s not surprising when more people are encouraged to give it a look and also like it. From what I’ve seen, the author’s celebrity status is mostly ancillary to the results. Maybe you can point to a counterexample, but I haven’t seen IFComp votes fall victim to predisposition purely based on the author’s renown.If that was a problem, we’d probably see a lot more top 3 winners also getting the banana of discord.
Basically what I’m saying, this sounds like a problem that doesn’t warrant new rules to fix.
Yeah, there are people who have spent years building up a fanbase, and they’re going to get more people playing. That’s why Stephen King sells a lot of books and why Tom Cruise makes the big bucks. We don’t make them do their work anonymously, and new authors and actors still get opportunities to play with the big kids, just like new folks do here. And place high, just like they do here.
But it’s not an indicator of the quality of those games, and I know for a fact that we hold established authors to a really high standard, because I’ve seen the many comments on that very thing. No one here is just giving established authors a pass, and you can see that in the results.
If that was a problem, we’d probably see a lot more top 3 winners also getting the banana of discord.
Just mathematically, it’s pretty hard for that to happen since the rating scale is bounded at both ends. A game which averages an 8.5 can’t possibly have a stddev higher than 3.21, while a game which averages a 5.5 can have a stddev up to 4.5. Of course if the 8.5 game really did hit that max it would probably still get the banana, but nothing is ever really going to come out that extreme.
I don’t support the ‘pseudonym for past entrants’ rule (which I think only one person is really asking for?).
If there was significant pressure for that sort of thing, I would support dividing the comp into past entrant and new entrant categories, meaning the comp would have no true first place winner.
I really don’t see any need for this though.
I filled out the survey! <3