Minimum ratings in IFComp (from Relaxing IFC's "hush rul

hard pass on an increase. we had a record number of contributors and judges last year, and i don’t know of any problems that can be attributed to the amount of time the judges spent playing, or to having too many insufficiently-burdened judges. put it this way - this means that there’s a chance of having 20 hours of work required of a judge over the course of six weeks. that’s a huge amount of unpaid work, especially as a bare minimum.

I’ve expressed interest in raising the minimum to 10 games.

In the “hush rule” thread, others pointed out that raising the minimum means effectively raising the barrier to vote, but only for honest participants. Dishonest participants can just give fake ratings.

Here’s a related question: when, if ever, would it be reasonable of me to blog about IFComp on the Choice of Games blog, in light of prior events? … al-attack/

My hope is that by raising the barrier to vote, it can be safe to publicize the competition to large fan bases outside the IF community; that it can make eliminating the hush rule safer.

But specifically I want to talk about IFComp with the Choice of Games community, which I really want to do, but not if it will ruin the competition. I want to integrate the ChoiceScript community with the broader IF community, but history shows that no matter what I say, at least some members of our community will absolutely not vote in keeping with the spirit of the competition.

My hope is that if we raise the minimum number of games, most dishonest participants can be detected, look-ee-loo participants would shrug and move on, and we’d get a relatively large number of new honest voters. But maybe I’m being naive?

I especially liked zarf’s take on this: … bout-them/

For what it’s worth, I agree that asking people to play a minimum of 10 games is too much. I feel like 5 is fine for the purposes of demonstrating that you’re not just a drive-by or a shill. Requiring more is likely to lead more people to vote on games they haven’t given a fair shake just to satisfy the requirement (or just not vote, which is arguably even worse).

Are you saying that people will cheat (undetectably) with the current minimum of five, but they won’t with ten? I’m not sure what’s different about 5 vs 10 for the problem you’re worried about. (Unless there are 5 ChoiceScript games in the comp?)

What if we just split the difference and remove the minimum, but after the voting is done, we divide the votes into two pools: One of votes for 10 or more games, and the rest; then we average those two pools out. That is, we make it so the final score is half from the heavily invested voters who took the time to review lots of games, and half from everyone else. This would allow us to accommodate votes from more casual users while still keeping invested community members as a deciding factor in the standings.

No. To clarify, I’ll describe two possible worlds in which we blog about IFComp, World 5 and World 10.

I claim that World 10 will have fewer voters than World 5, but specifically the voters who would vote in World 10 but wouldn’t vote in World 5 would be voters who care more about IFComp.

I think many of the folks in the CoG community don’t (yet) care about IFComp itself, and so are willing to click a few buttons to promote ChoiceScript authors, but aren’t willing to go to any real effort to ruin IFComp.

Thus, I claim, the voters from our community who would have voted in World 5 but wouldn’t vote in World 10 are mostly voters who wouldn’t follow the spirit of the competition.

I don’t claim that it would be easier to detect the dishonest voters in World 10, but probably many/most of them will be easy to detect regardless: they’ll navigate to the IFComp via, they’ll give CS games a 10 and a handful of other games a 1. Those votes can/should be nerfed and ignore.

I don’t think caring has that much to do with it honestly. It just means less people will vote due to time/resource/patience constraints.

I guess my counterclaim (based on not much more than a gut feeling I admit) is that “voters from the ChoiceScript community [who wouldn’t follow the spirit of the competition] who would have voted in World 5 but wouldn’t vote in World 10” is a pretty small number, and that the World 5 - World 10 difference is much smaller than the World 2 - World 5 difference. (SpringThing is a World 2 comp, but I guess it has no ChoiceScript games this year so we couldn’t run that experiment…)

I have to confess that I didn’t even make it to 5 games last year. I was completely overwhelmed and exhausted from work.

Increasing the minimum number of games required to vote might result in people prioritising playing short games - or accidentally penalising people who just happened to start by playing a number of longer games. (Not to mention penalising the authors of longer games, as more of their ratings might be eliminated from the final count.)

I really don’t see why we would want to make it harder to be a judge.

Am I just being too paranoid? Should I just blog about IFComp in 2016 and let the organizers sort out the mess? (There will be mess.)

And if I’m not being too paranoid, then I have to ask: what rule change could make it possible for me to blog about IFComp without messing up the competition?

In a sentence: because otherwise it will be too easy to flood the competition.

Do you disagree with that? Do you think it would be hard to flood the competition? Or is flooding the competition not as important as it seems?

If the goal is to prevent flooding, would it help to give an incentive to the voter? E.g. if you (voter) do this, (your vote will be thrown out and) you will be disqualified from judging in future IF Comps.

Some relevant backstory that I’m not sure is equally in everyone’s mind is that in 2012 the ChoiceScript community did flood the 2011 XYZZY awards (unintentionally) after a blog post from Dan brought in a ton of voters, and I believe Dan has avoided posting about the XYZZYs on Choice of Games ever since.

bg: It seems to me that the problem isn’t so much intentional misconduct by individual voters but something that brings in a ton of voters from one source that overwhelms everyone else’s voice. Like, I always wind up playing mostly short games (and usually breaking the ten-game threshold!) because I dive in enthusiastically and polish off a few games that I can get done quickly and then two weeks into the playing window I get clobbered by midterms and have no energy or time to do anything demanding for the next month. Eventually I mark the games I’ve played. There’s nothing wrong with that, I hope!

Now if I post about IFComp on the huge-readership “Easily distracted academics who are going to get clobbered by midterms” board and wind up sending hundreds of people who behave exactly like me to the IFComp, the short games are going to get many more votes than the long games. That seems unfair. But it isn’t because any of the individual voters did anything ban-worthy, I don’t think; it’s just an unfortunate collective action problem.

At least, that’s what I see as a major concern–not so much deliberate trolling as uncoordinated pile-ons.

Maybe I misunderstood then. I was reading the concern as “dishonest voters who give 10s to COG and 1s to other games without given them a fair chance.”

I don’t anticipate that increasing the requirement from 5 games to 10 games will weed out dishonest judges who score games without playing them. Once a person decides to vote falsely, there’s a negligible difference between “vote 4 times falsely” and “vote 9 times falsely”.

Additionally, with regards to the 2011 COG XYZZY debacle: I think we’re doing a disservice to the CoG fanbase.

Yes, there were CoG fans who voted a pure CoG slate - but we don’t ask or expect XYZZY judges to play all the games. I expect those voters were being completely honest when they said “yes, this is my favorite IF game of 2011”. They just hadn’t played many others, if any.

As far as I’m aware (and please, correct me if I’m wrong) - the only clear indication of CoG fan dishonesty is that a puzzleless game received votes for its great puzzles… and, well, that wasn’t cool, and I was pretty irate about it at the time. But today, I’m not even sure that stemmed from dishonesty. Most of the people on this forum have a very specific concept of what a puzzle looks like, because the majority of the people on this forum started out in parser IF. But it’s not like we give a definition on the IFComp page - and if you interpret “best puzzles” as “most difficult to solve successfully” (which is not entirely unreasonable) then it’s even an understandable error. And either way, that’s a pretty far cry from claiming to play games and then sinking them with 1’s without even booting them up.

We should stop assuming the worst of the CoG fanbase. They’re IF fans, just like we are, and they deserve the benefit of the doubt.

I wasn’t trying to assume the worst–I was going by what dfabulich said:

So either you disagree with dfabulich’s assessment of the situation, or I’m misunderstanding something (or both).

My example of 10s and 1 was meant to be an example of the type of thing I thought was the concern, because I was trying to get some clarity about the purpose of the discussion. I’m by no means trying to claim that 10s and 1s are exactly the form “not vot[ing] in keeping with the spirit of the competition” necessarily takes. I don’t know what voting practices dfabulich thinks might happen in IFComp.

I didn’t meant to suggest that CoG fans had done anything dishonest with the XYZZYs, or could be expected to do anything dishonest with IFComp! Quite the opposite–I was thinking that, even if every CoG fan votes perfectly honestly, and most of them honestly prefer ChoiceScript games, they would still be likely to overwhelm the vote.

Which might be OK, because it means that the games that win are the kind that most of the voters like! But it still means that the winners would be the ones whose fanbase mobilized most effectively, which is I think the main thing that people defending the muzzle rule want to avoid.

Yes, I think it would be hard to flood the competition. I think the differences between the XYZZYs and the IF Comp are significant here: for one thing, the 5-minimum vote rule is precisely there to avoid drive-by flooding by people who aren’t engaged with the comp as a whole. Also, with the XYZZY Awards, it’s about pre-existing games, and if you’re already a fan of a game, it seems natural to want to go and vote for that game, to give it your support and help it get recognized as the best game of the year. But, all the games in the Comp are brand new, and (usually) specifically created for the Comp itself, so it’s a different dynamic. Of course, people might be fans of particular authors, or fans of ChoiceScript games as a genre, and might think it’s natural to want to go and “root for their team” in the IF Comp. But the voting ballot page still makes it pretty clear that this would be against the spirit of the competition (both from the text at the top and from the fact that they’re putting numerical ratings on the games rather than just picking a favorite), and that they should go and actually play a few games and make an good-faith effort to judge the games against each other. (I assume your blog post would say the same thing, Dan.) So unless they really are intentionally going against the spirit of the comp, I don’t think the flooding would be a bad thing (and hey, maybe they’ll try out some non-ChoiceScript games and realize they like some of those too!). And if they are in fact there to try to game the vote, well, a 10-vote minimum isn’t going to stop them, and hopefully the vote-checking heuristics will catch them anyway.

I didn’t mean to misrepresent the perspectives of people in this thread. For years, I’ve heard (and generally supported) the perspective that CoG fans shouldn’t be allowed to “overwhelm” IFComp. But the more I think about it, the more I think those concerns may not be applicable to IFComp. I wanted to speak to that specific issue, rather than addressing or rebutting specific posts in this thread so far.

Of course, dfabulich has a better perspective on the CoG fanbase than anyone else - and if there has been a history of CoG fans skewing IFComp, he would be better-positioned to know than I would.