IFComp 2022 Review Spreadsheet

Competitions IFComp
41

Update: I noticed that the mean total # of public or private reviews is now more than 1 above the median, which means the mean will probably be more than the median for the rest of the comp. For a while it bounced from the median being more than the mean to the less.

I thought it would take longer than it did to get here, especially because people are taking time to spread their reviews among those entries that haven’t gotten much yet. So that’s a really good sign – and in general I’ve been pleasantly surprised how, even during weekdays, I come back to the forum and see a few new reviews up in various threads.

For public reviews we have 3.6 versus 3. That means it’s pretty close. In fact, we would need to review 14 games that currently have 3 public reviews to move it to 4, but we have 27 reviews until we hit a mean of 4.0. Given there are 21 entries with 3 reviews, we would expect the 27 reviews chosen at random to have 27*21/71~8 reviews bumped from 3 to 4, so this is a fait accompli, especially since there are 28 entries with 2 reviews, and there’s a deliberate effort to review gaems with less reviews!

thoughts on my own currently-private reviews

I also think that various reviewers trying to emphasize reviewing longer games has helped me get to them earlier. I didn’t last year, and I had a backlog at the end.

I forget if I posted my pacing document to get through all the reviews publicly last year, but in 2021, I didn’t account for game length, and this year, I did. So this is a good thing that goes beyond the basic “how many games did we look at” numbers. There are a lot of fun sounding games with snappy titles and short playing times (Applebee’s, Gum E. Bear, To Endure) and this combination will get more reviews just because. (Also, I’d like to note it isn’t just a gimmicky title. I enjoyed all three, and it looks like reviewers did, too. And people without a lot of time deserve to be pointed to something they’ll probably just flat out enjoy.)

But it looks like a lot of reviewers have planned ahead to try to tackle the tougher/longer games, and that’s great to see, and I think it encouraged me, too.

Final note: the entries lagging behind are Chinese Room (fewest public reviews) and Lost Coastlines (fewest private). I haven’t reviewed either, but I am looking forward to playing both! And I’m glad one entry doesn’t have the fewest private AND public votes.

1 Like
42

What I’ve noticed is that the mode and median are both 4. That is, the most common number of reviews is 4 and over 50% of games have 4 or more reviews. That seems great: four is a decent number of reviews.

At a basic level, it’s also good to see that no games have zero reviews; no games have just a single review; and very few have only 2 reviews.

It’s hard to describe what an optimal situation would be, right? Perhaps it’s that every game has more than one good, constructive review. That’s hard to tell from the numbers, but if most games have 4 reviews, there’s a good chance that two of those are constructive.

All in all, this seems a good situation to be in.

(The other thing I’d note is that the distribution is heavily skewed. Perhaps this is inevitable? There are always a few games - the short ones, the fun-looking ones? - that get a lot of attention.)

1 Like
43

I’d guess so. This is above my statistical pay grade, but running a Python simulation like so:

import random
from collections import defaultdict

totals = defaultdict(int)

def print_rand_range(amt, times):
	freq_dict = defaultdict(int)
	my_ary = [0] * 71
	for x in range(0, times):
		my_ary[int(random.random() * 71)] += 1
	for x in range(0, amt):
		freq_dict[my_ary[x]] += 1
	out_string = ''
	total_sum = 0
	for x in sorted(freq_dict):
		total_sum += freq_dict[x]
		out_string += ' / ' + '{}={}/{}'.format(x, freq_dict[x], total_sum)
	print(out_string[3:])
	print(max(freq_dict), min(freq_dict))
	totals[max(freq_dict) + min(freq_dict)] += 1

for x in range(0, 100): print_rand_range(71, 370)

for t in sorted(totals): print(t, totals[t])

Seems to create, on average, a curve that skews right but not too much, with everything picked randomly. But entries aren’t picked randomly, of course. People look for short fun stuff.

And yeah it’s tough to balance review quality versus quality. I’m of the “quantity will eventually beget quality, with minimal self-awareness” school so I don’t mind seeing more.

2 Likes
44

Does anyone have an idea about the ratio of reviews to total number of votes in past years?

1 Like
45

Last year’s spreadsheet is here:

There were about 800 total reviews, and 565 public ones. I’m sure there are varying ways to slice and dice the numbers, but to get things started, the most-reviewed game had 23 reviews, and got 82 votes in the Comp (that’s the winner, And Then You Come to a House…), while the least-reviewed game had 4 reviews and 11 votes.

And if anyone wants to use 2020 data, here’s that year’s spreadsheet:

(Though note that authors only started being able to vote in the main Comp in 2021, and 2020 had a lot of authors, so comparisons should probably be drawn with caution).

1 Like
46

This spreadsheet contains review/rating counts for 2018-2021 - see the “All past results” tab (I added a plot showing the rating count vs number of public reviews). I used this spreadsheet to predict results last year, and probably will this year as well, once there are more reviews.

5 Likes
47

Emily Short posted about The Short Game podcast. They have started talking about the IF Comp. Their first episode is up, and they talked about:

  1. The Grown-Up Detective Agency
  2. Esther’s
  3. The Thirty Nine Steps
  4. Nose Bleed
  5. Use Your Psychic Powers at Applebee’s
  6. Glimmer
  7. HOURS
7 Likes
48

Just a shout-out to whoever it was who, within minutes of me posting my reviews, already had them added to the spreadsheet. That’s quick work! I’m super impressed.

3 Likes
49

Nearing 500 reviews total!

1 Like
50

In a fortuitous bit of arithmetic, being close to 500 means we’re also nearing 7 total reviews per entry at the same time–497. And, more equitably, we are at 2 public reviews for each entry! It looks like getting to 3 will be a foregone conclusion, as Chinese Room and Lost Coastlines are the only ones with 2.

(Note: this is motivation for me to review Chinese Room, since I’ve reviewed Lost Coastlines, and I’m making my reviews public post-comp, so that’d put us sort-of kind-of already at 3.)

These milestones (500, 7 per, 2 minimum) are all worth celebrating, maybe not lavishly, but to say – hey, neat!

And while this isn’t the place for a macro-economic political discussion of whether a rising tide lifts all boats, I think the strategy has worked very well here.

11 Likes
51

The Short Game has posted episode 340, the second one to cover the IF Comp. They talked about another seven games.

  1. Into The Sun
  2. The Absence of Miriam Lane
  3. Admiration Point
  4. Star Tripper
  5. To Persist/Exist/Endure, Press 1
  6. The Only Possible Prom Dress
  7. Trouble in Sector 471
9 Likes
53

A milestone: we have officially surpassed 2021 in public reviews (although a significant part of that is one reviewer moving from the private to the public category). We are also on track to surpass 2021 in total reviews, if the pace keeps up.

11 Likes
54

Hey, first time listener, first time caller. I noticed one of my reviews (for US Route 160) was not linked. Is that something I am empowered to edit/fix, or is there a sheet-master that manages that?

UPDATE: lol, and then someone beat me to it. Thank you spreadsheet gnomes!
UPDATE2: nope, I am an idiot I misread it. Fixed myself!

As it appears running the table is potentially in reach for me, I am particularly aggrieved that my game computer (linux laptop) is blocked by ipv6 and windows executables. It’s going to be a fraught final weekend of workarounds I fear.

Poe had his Telltale Heart. My heart of madness is nearly-but-not complete lists.

1 Like
55

I’ve been editing it myself.

1 Like
56

I’m not sure they’re robust enough to include in the spreadsheet, but there are quick reactions to at least a dozen games in this SomethingAwful thread.

7 Likes
57

Oh hi @MoyTW

58

Hello hello

59

@jjmcc I hope you make it! I think half of the reviews is a really impressive haul. If/when I’m not in IFComp, I don’t know if I’ll have the motivation. I want to review because otherwise I’d feel like a potential black hole. Also it seems every year we have someone drop in with a bunch of unexpected reviews and this is you, so thanks for that. Last year it was Lava_Ghost & I’d be glad to see them back but I know life gets in the way.

I’d be interested if long games are getting reviewed more this year. It feels like it. It’s interesting that last year the mean was at 12 but this year it’s 10 and very close to 11. (The next 3 reviews of entries with 10 reviews will push it over.) It looked like a lot had 13-14 reviews and a lot of entries with odd formats or presentation were less reviewed, for example, Windwalker or Cygnet Committee or Silicon and Cells.

My author-forum reviews should help push us over the 2021 mark and 800, as I have a few lined up. If I don’t make it, someone else will.

I may just miss running the table this year as I had other projects – I’ve barely been able to check this board for reviews of entries I’ve been interested in! – but it’d be neat to help with a final push.

If anyone wants to push the mean to 12 for whatever reason, it might not be too bad to look for ones with 11 reviews. This may seem silly, but I often find narrowing down my choice of entries to review next, even arbitrarily, helps me push forward. (I’d repeat what I said earlier, which is to try to give less-reviewed entries the most air if possible. I’ve really enjoyed a lot of them and am looking forward to a few others.)

2 Likes
60

So, for my final “look at the pretty numbers” update: we’re currently at 632 public reviews. That’s 7 away from the nice round number of 639, which doesn’t seem that round until one notes 71 * 9 = 639!

While it’s entirely possible The Short Game may drop 7 reviews in their podcast (which would be way cool!) I thought I’d just lay this out there for anyone who was on the fence about writing one more review to say, hey, why not, I could be the one to help us hit that average!

Thanks to all the reviewers so far. I’ve barely been able to keep up, with my own reviews and also my projects elsewhere. I believe I have some reviews to respond to.

4 Likes