IFComp 2021 Review Spreadsheet

Hello, folks.

It is okay for authors to discuss games, and to post reviews, in public. They may not feel comfortable doing so, but it is allowed. Some authors have asked about this, and I confirmed with them that this is the case, but given the discussion here, I wanted to jump in.

In 2021, authors are also judges. This will be re-evaluated after this year’s competition, but this means that, at least for 2021, the rules about being a judge also apply to authors, including the rule that judges "may discuss the games during the judging period."

What has not changed is that authors “may not encourage competition judges to violate the rules that pertain to them,” including the fact that judges "must make a good-faith effort to play, as intended, every game that they submit ratings for."

There is a difference between a review that says, “I loved this game, it was amazing, I hope it does really well and I hope everyone checks it out” vs “please vote a 10 for this game that I loved.”

There is a difference between a review that says, “I was not a big fan of this game because I really feel it needed more work” vs “no one should give this game anything above a 3.”

I appreciate that authors may still desire to keep their reviews private until after the competition, perhaps out of a sense of collegiality, a wish to avoid awkwardness, etc. That’s still their option. But so long as the reviews that authors post are within the IFComp code of conduct, and are not encouraging other judges to disregard judging rules, authors may post reviews and publicly discuss games.

Again, we will reevaluate this approach after this year’s competition. Judges (including judges who were also authors) will be encouraged to fill out the post-competition survey.

9 Likes

This is my second ifcomp as judge, so I am a newbie but want to give altruistly my opinion.
I think that “corralito” author’s forum is a very good, very useful for these authors so they can talk about programming, authoring, etc. More than that, I sincerely think that author’s forum should become public sometime after the comp. All people needs to learn from actual authors.

The question: Can authors vote? I think they should vote anonymously. They have to be, as every judge smart, polite and sincere. After all they are the top masters on IF and it’s supposed to be the more adequate people to emit an objective veredict.
I want to remember here the big different ranking shown between ifcomp and ifdb as time go by, after every comp.

  • Jade
3 Likes

“Should” is a tricky word. I know I’d very much like to see what happens there, but I also know there’s some privacy involved. A lot of authors take stuff from the forum with permission, and a lot of us wind up posting our reviews to IFDB, and I’d like to see more of that.

But some people enjoy being able to discuss things because they may have a concern they don’t want to go public. It might be something as simple as “I think reviewer X doesn’t like me but I don’t want to start a fight. Am I out of line?”

Some private stuff should stay private. But there’s a lot of stuff that was written in the author’s forum that made its way to the public eye.

8 Likes

Out of curiosity, why are a bunch of the entries red now instead of green in the spreadsheet?

Games in red have fewer public reviews than the median. As new reviews are added the median slowly goes up.

5 Likes

Ah okay, thanks!

I just added a column for myself, but I had a (temporary) problem with the formatting. It should have turned gold.

I think what happened was that I edited cell AO1 from “Andrew Schultz Author” to “Andrew Schultz Author’s Forum” or something and then I pasted the link in while editing, or right after, and there may’ve been a race condition.

I cut/pasted another of my review links over and then changed the URL, and the formatting showed up. So just a tip for other people who might have this happen.

Also

  1. Is there anything tricky we need to do to add a new column? It seemed quite straightforward, which is nice
  2. Is there any objection to adding, or should we add, red or orange cells for if an author wrote the game, or if a reviewer beta-tested it? (a full disclosure thing, I guess)
1 Like

Nope, just pull the Total formula over at the bottom and add the border and you should be set.

1 Like

It looks like the row with mean number of reviews has disappeared. Was that deliberate?

My game has been stuck at three reviews for what feels like weeks, but I was reasured as the mean was not much more than that. I do not feel the median is quite the same somehow.

1 Like

I moved the mean to the new Stats sheet.
I thought it would be cleaner that way but feel free to add it back to the Reviews sheet if you disagree.

2 Likes

I had not noticed there was a new Stats Sheet. As long as I know where to find it, it is fine.

1 Like

First, a small aside to say congrats to adibianca for being the first person to 10 reviews and bjbest for being the first person to 10 public reviews!

I missed those stats being moved, too, but I agree they’re better on a separate stat sheet. Because we could add one stat, then another, and so forth… and it would get messy. Here it’s clear and simple how to add a new reviewer or reviews, and we know what entries could use more reviews.

Are there any guidelines/restrictions on what sort of stats to add? I have to admit, it’d be neat to see how “median” the median is (e.g. if it takes #36 through #60 of the 71, it is about to go up.) I like those sorts of numbers, but it might just be fluff to most people.

The reviews per game bar graph is clearly not!

Also I’d love to make a bar graph of reviews per reviewer – I’m not sure how we’d lump ranges together. Perhaps 1-5 up to 61-65 and 66-71. Not for competitive purposes of course but just to see how things fall into place.

4 Likes

No, please feel free to add / change anything you want. While I started this year’s spreadsheet it belongs to the community and everybody should feel empowered to improve it without asking for permission.

3 Likes

I can’t say that that would be helpful.

I’d like to encourage more reviewers, and I’d like to encourage more formats of reviews – right now we’ve got Twitch streamers, we’ve got people sharing thoughts on Twitter, and we’ve got written reviews in different venues (here, the IFDB database, and personal sites).

It’s difficult to tell what kind of behind-the-scenes effort was necessary to create some of these reviews, which dismisses the work of people who spent a lot of time on a smaller number of published comments.

When you start charting the number of reviews per reviewer, it suggests that a lot of their work is interchangeable. I don’t think that’s the case.

5 Likes

Quick note for those adding reviews (thank you!): there’s no need to manually adjust the review count, the sheet will do that automatically.

2 Likes

One side point: it may seem like it doesn’t automatically, but that’s because it takes time to load. (I’ve had that happen and wondered what was up.)

Yes, good point. It’s tricky–we want to promote more reviews, because I really do think quantity (eventually) begets quality. But people who already have a certain level of quality shouldn’t feel forced to go back to quantity.

And just as people who compete in IFComp the first time do a lot better the second time, I think first-time reviewers learn a lot and come back and do even better later, and it’s okay if their reviews are a bit generic, because their observations add up, so they shouldn’t be ashamed if they can’t be profound–too often I’ve let that block me, where I know I don’t have something big to say, so I wait until I do, and I get backlogged. And I know sometimes a throwaway observation helped me add something important.

Yet at the same time, yeah, we want to encourage people who say “OK, I’m only going to look at games in genre X or length Y, but I’m going to look in-depth.” I’m not sure how to balance that.

That said, I do enjoy looking at stats, and even if the # of reviews per day tapers off, it looks like we’ll hit 10 reviews per game, which seems quite good. I’m just not sure what stats are most useful and productive.

5 Likes

I’d been curious in general, so I decided to track overall reviews per day, just to see if there are any trends. To do this, I took the public/private reviews on the last update of the day and put them into the data. I graphed total, private and public reviews with trendlines. As we’d expect, reviews per day go down with time. I was amused to see the authors’ forum had expected negative reviews on the final day of the comp! Maybe it is due to a small sample size, or maybe it is that the graph isn’t a line but something like y=a/x (or x^.2) or something.

Anyway, hadn’t seen this done before, hope it’s on-topic and interesting, if no-frills. I’ll be adding rows every day. The data may be useful to IFComp staff who are curious. If anyone wants to see other data, let me know.

6 Likes

The author’s forum ALWAYS expects negative reviews. /s

2 Likes

Especially from reviewers who clearly haven’t read the instructions.

1 Like