Growing number of comp entries and reviewer selection bias

[Post removed by author]

5 Likes

Well, I always have been a little stupid about numbers :slight_smile: To me I see a 95% chance of a judge avoiding my – or your – games, and as I told the fellow in the “grah grah grah i wish there were another parser competition and twine people shouldn’t enter the competition expecting to win” thread, I didn’t enter the competition expecting to win.

But for the record, there are only 102 games in the competition this year :slight_smile:

2 Likes

[Post removed by author]

5 Likes

I’ve been thinking about this. There are a lot of entries in the competition, and with it paying out money, it’s only going to grow. So I was wondering if the Comp could be focused. It would be a split, but the playtime that’s split. Since the Comp is now two months long, couldn’t you release Choice-Based having that played and voted on in October, and then release Parser having that played and voted on in November? Afterward, releasing the full results. No split there.

People might hate the idea, but it’s something I’ve been thinking about - D

2 Likes

[Post removed by author]

2 Likes

I dunno, I’m one of those crazy people who tries to play every game in the Comp, and part of what makes that doable (and to my mind actually easier than in the early parser-only games) is the variety. It’s a lot harder to motivate oneself through a dozen lightly-comic puzzlefests with XYZZY in-jokes and obtuse guess the-verb-puzzles in a row (to unfairly stereotype parser games), and similarly could be a bit of a slog to wade through a similar number of navel-gazing puzzleless choice games about mental health issues (to return the favor). Using the randomizer and getting served up a list that’s highly varied in terms of genre, length, mode of interaction, etc. helps create more novelty and momentum to play more games.

Anyway, personally I’m not all that concerned about reviewer biases – the stakes for victory, as others have pointed out, are generally fairly low, both in immediate returns and in legacy. The good games do tend to have a way of being noticed, even if they don’t win. And I think there’s an under-observed dynamic, where the growing number of entries also bespeaks a growing number of authors. While they obviously don’t (and shouldn’t) count as regular judges, someone who’s an author is probably more invested in playing games, writing reviews, and participating in the community (and of course voting for the Miss Congeniality awards, which you could see as a sort of shadow-jury helping to ID if something seems to be going awry with the main voting – so far I haven’t seen that happen).

As I said in the other thread, it’s potentially a small problem that in whole and on average, a specific type of game tends to take top honors in the Comp, and maybe it’s worth having a conversation about that – but less, I think, in terms of structural changes and more about how judges are approaching their work. Like, if I recall correctly, last year I ranked Zozzled and Turandot the same – they’re both really really good! – but Turandot has stuck with me to a much greater degree, and if I were to rate them in retrospect I’d definitely give it the edge. So I’m trying to bear that in mind as I evaluate games this year.

8 Likes

[Post removed by author]

6 Likes

I’m another who’d hate that idea. Part of what keeps me going through as many IFComp games as possible is the variety. Including often alternating web/choice with parser. If I couldn’t do that, and had to play all of one type I’d not play so much. Also parser games tend to be longer, so would be more of a struggle to get through if they had half the weeks allotted.

But I don’t think it’s something the organisers would approve anyway, irrespective of it reducing how much I’d judge.

In 2010, there were an average of 93 votes per entry, and the top ten got 101 votes per entry. (Least voted entry got 50 votes.)

In 2019, there were an average of 43 votes per entry, and the top ten got 51 votes per entry. (Least voted entry got 9 votes.)

I think this matters to the extent that an individual author is really in it for the feedback. I may just be an attention hound, but I would (in all honesty) prefer getting more votes and more feedback. (Of course, those stats don’t measure feedback in general.)

I don’t mean to say there’s a “problem”, and I don’t have any changes to propose, but I think there’s a downside as well as an upside to having a larger field.

6 Likes

As another viewpoint on the numbers issue I do feel concerned that I’m likely to miss out on some great games that I can’t play. All a result of the competition entry numbers growing so much. I used to be able to play most of IFComp. Even with my progressive neurological illness. Now the illness has advanced more that wouldn’t be possible for most years of the competition, but with 104 entries this year it’s certainly impossible.

Part of what keeps me playing IFComp - and I’ve been judging since the very start, back in 1995, yes I can remember the split comp era! - is finding great games. Variety is another huge factor keeping me going. But ultimately I want to play to find some great games. And as the number of entries grows so much there’s less chance of me seeing all - or even many - of the greatest games entered. At least during the competition period itself. I can catch up later.

This might not seem that important, but I think it can be an important motivation factor for judges. We are not paid, we are not rewarded for our time and effort, beyond getting a good gaming experience. Yes we only have to play 5 games each year, but are those 5 going to be the ones that make you want to play more? I’ve been doing a lot of promotion of IFComp this year on Facebook, Twitter, Reddit and web forums, trying to drum up new voters. Always saying “You just need to rate 5 games for your votes to count.” But I’m having doubts how many will reach even that. And ultimately the competition rests in a way on the judges, albeit perhaps not as much on the competitors.

Just some alternative thoughts. I also feel that there can be a bit of an author/judge split in views on this issue. Where most authors are probably “Yay, more entries!” Whereas some judges like me look at the field of 104 and go “Wibble …”

3 Likes

Well, it’s true to form. The genre was basically invented when a bunch of programmers decided, hey, you know what, our work and educational lives don’t involve enough situations where we need to find proper syntax, untangle complex systems, and sort disparate information, we need to make more. :slight_smile:

1 Like

More so than if 100 games of this quality/size were released over the course of a year, absent a competition event?

1 Like

In a different way, yes. I may miss games throughout the year because of less promotion / the reduced coverage of IFComp. But I’m personally more worried about missing some gems in IFComp. Which is probably daft and all, but as I say it’s a motivation issue for me to keep going in a very intense few weeks of judging.

2 Likes

The condorcet method of voting might address many of your concerns if adopted. It was last discussed here: https://intfiction.org/t/is-the-condorcet-method-suitable-for-ifcomp/44160/6

1 Like

I could see an argument for trying to make the SpringThing more relevant, but otherwise I can’t see good things coming of intentionally splintering up the one big unifying event of the community. The sheer bewildering amount of variety is one of the things I love about IFComp games, and it’s the only time IF communities that generally keep to themselves come together and see what other authors are up to. (And this is not even getting into how much interest there’d be in multiple smaller comps with the promise of significantly less prize money and significantly fewer players for each of them…)

SpringThing originally charged money just to enter, and then later it stopped being a competition at all. If it never really took off that seems like largely its own doing.

I’m surprised that when everyone was so against the CYOA vs parser split (and is every single year) you’ve all just turned around and started entertaining the idea of breaking the comp apart in other ways.

I’m not convinced there’s an actual problem to begin with. Yes there are a lot of games, but this thing is designed with the idea that most people aren’t going to be playing all of them, and that many voters will be new or casual players. That’s worked up until now, how about at least waiting to see if more games somehow lead to problems with the judging before hitting the panic button and trying to change it all. Really all I’m seeing here is people freaking out because their hobby is a little bit popular now.

And I’m pretty confused as to what that tag was even about but thx @mother u 2 bb.

Disclaimer: I’m not remotely planning to play every game. But if it turns out I missed a good one even with recommendations from friends etc then I’ll just check it out after the comp. I’m sure the authors appreciate feedback at other times of the year too.

It’s probably not doing authors any favors either to get votes from people forcing themselves to play things they already know they’ll hate…nor can the vast majority of voters ever be expected to do this. If there was pressure to push out casual players and people with limited time who are already doing the authors a favor by playing/voting at all, pretty sure that a significant number would be happy to sit this phase out altogether and let other, more masochistic judges sift out the chaff for them. And then you’d really start seeing issues to be concerned about with the author/player ratio.

And this concludes this all-over-the-place thread’s most recent text wall of long winded Opinions that no one else really cares about and most people won’t bother to read.

5 Likes

Well, good night, I have already voted a dozen games and write a comentary about each one.
As this is my first time in judging this contest, I dont know if I should public my comentaries.
As far as I can see public comments enhance that some games had been more played than other ones.
On the other hand if anyone can read those, experience can be more sattisfactory.

Best regards.

9 Likes

Dear Jade, welcome! I would certainly make them public – many of us judges love to compare notes, so to say, and check out other people’s view when we’ve played a game ourselves. :slight_smile:

9 Likes

Well for the record, don’t make any decisions based on any comments I’ve made :slight_smile:

1 Like

SpringThing was set up as an alternative and counterbalance to IFComp… after IFComp hit a shocking and record-setting high of 51 entries.

In the alternative universe where Spring Thing did take off, and ran at ~50 entries per year, I can’t see anybody arguing to merge it back into IFComp on the grounds that one big event would be better. One big event would clearly be worse, I’d think. You’d need to pick one set of rules, for a start. And what the heck would people play when it wasn’t October-November?

(I always worry about one big anything. What if it goes wrong?)

(Yes, I think about this a lot in IFTF planning.)

To answer a relevant question from the other thread:

Maybe we could divide the Colossal Fund between the two events?

If the event planners decide to do that, then yes, that’s easy to do.

This would be, let’s be clear, a big change in the way Spring Thing plays out. Part of the point of Spring Thing is to work differently from IFComp, and that includes not having large voting-based prizes. Should that change? Are there alternatives? Big discussion there.

4 Likes

Second this. Reading other reviews often deepens my understanding of a game I’ve already played.

Plus more reviews contributes to the overall discussion of how to design games, which results in better games in the long run.

2 Likes