In 2017, as an IFComp author, I ran a competition awarding prizes to reviewers who reviewed every game.
In that year, there were 755 public reviews. The next year, after some discussion, I didn’t run it and there were 502 public reviews reviews. Last year, there were 599 public reviews, but 83 of those were mine, and I won’t be publicly reviewing during the comp this year.
Last time this competition was run, a few odd things came up (with someone doing a somewhat rushed series of videos reviewing all games), but nothing negative.
I’ve asked the current IFComp organizer if doing something like that would be feasible again this year. There was some concern before about authors giving anything of value to reviewers, but the rules are designed to make it difficult for reviewers to show any favoritism:
The Rules:
There is a prize pool with N entries (for some N).
At the end of the competition, the N reviewers that reviewed the most games are put into a pool. If there is a tie (such as multiple people reviewing all games), then the first person gets into the pool first. I can almost guarantee there will be more prizes than people who review all games.
Authors (or, if we open it up, anyone) votes for their favorite reviewers using the same scoring method as IFComp. The finalists then get to choose their prizes in order of their score.
Let me know if anyone’s interested in running this year, or if you have concerns. The record seems to show, though, that this resulted in a much higher chunk of reviews than usual.
Good idea! I think the proposed rules are fair. Reviewers are primarily ranked by how many reviews they write, which can be measured objectively. Popularity only affects the relative order in which they get to choose prizes.
In my opinion, it would look and feel better to allow public voting. Authors (as a group) might have a tendency to favour reviewers who produces a large number of very similar, insubstantial, but flattering reviews (“This is the best game ever!!!”). The public (as a group) would not have this particular bias, but maybe others. By opening up the vote, reviewers are incentivized to appeal to both groups, i.e. to write with substance and a positive vibe.
Allowing the public to vote would be a positive measure. It is not just the quantity of reviews but their quality that is important. I expect to review entries this year. I would like to complete quality reviews. I tend to like just about everything I play and feel reluctant to express negative impressions. I did find this thread that provides guidance on the review process to be helpful.
I am not personally motivated by awards that have significant intrinsic value as a reviewer or potentially as an entrant in the future. Recognition for quality of work is most important.
Allowing public vote would be good. Perhaps even go one step further and say that IFcomp entrants cannot vote. They could be biased by the reviews of their own game. I also agree that quality is most important, not quantity. Still, I think it could motivate reviewers to make more quality reviews, i.e. more reviews than they would have written otherwise.
I do think that one way or another we, as a community, should invest in increasing the amount and quality of reviews / analysis / and so on. Not just for IFComp, but in general. I don’t have a very strong feeling, either for or against, about the current proposal. Seems worth trying, at the very least.
I would, however, at least consider putting less emphasis on the amount of pieces reviewed. Speaking as an author – I’m not an author this year, by the way – one in-depth review is worth much more to me than five cursory reviews. And the present scheme might push people towards quantity over quality, which seems like a shame to me.
I appreciate everyone’s thoughts.It seems like going forward having non-authors do some or all of the voting seems like the better idea.
My reasoning for valuing quantity is that I believe focusing on quality could decrease the number of reviews as reviewers focus on crafting witty and in-depth reviews for a few games. While this could be a desirable goal, my intent in this competition is to increase reviews for ‘bad’ games. Low-scoring games are generally still things that require significant amounts of work, and to put them in the comp and see very few people react to it is very disheartening.
So, my modified proposal is to have two categories: one based on my method above, and one based purely on quality.
So here’s my revised proposal:
Everyone can assign a score to their favorite reviewers (no voting for yourself!) in 2 categories:
Best Coverage is for reviewers who you feel did a good job covering the whole competition. Best In-depth Reviews is for reviewers who you feel had high-quality or insightful reviews.
There would be no other restrictions on these votes, so if someone reviews all 80 games but does a bad job, you’re under no obligation to vote for them.
Strongly agree with the first sentence, mild disagreement with the second sentence.
I’m not convinced that there’s a large population of reviewers out there who would abandon their original plans of doing detailed reviews in an attempt to win a prize.
On the other hand, these rewards might bring out some new voices who are more comfortable being part of a larger group – some people could be intimidated if they only see detailed critiques published by “regulars” in the community, and that could be offset by a greater number of published reviews from a broader range of reviewers.
I think if we start seeing a larger quantity of reviews, it might motivate some authors to differentiate themselves by writing better quality reviews
Probably not possible for this comp, but would it help to have a central area where people can leave reviews on current comp games? (Even as a subforum here?) I have a feeling not everyone without a blog would want to set one up specifically for the purpose of reviewing ifcomp games. Otherwise I saw a few people left reviews on their own thread in the infiction forums or directly to IFDB which would also be existing options.
That seems more sensible, but the precedent from previous years seems to have been to post reviews directly in the IFComp category. Perhaps a moderator could weigh in on the preferred venue for reviews?
Either way is fine. There are tags you can add to messages ifcomp2020 (for general discussion) and ifcomp2020-game (for discussion of specific entries).
Most review traffic historically happens in the IFComp sub-category under competitions, but I believe most users live in the “Latest” tab where any new messages float to the top regardless of category.
Don’t sweat it - it’s an easy enough matter to move your own message between categories by clicking the pencil icon (in the original post) to edit subject, tags, and category; or Staff and Regular members at trust level 3 or higher can also recategorize if necessary.
Given the immense number of games, I think reviewers will definitely appreciate incentives. I’m planning on going forward with this, but not for a few days.
My concern is that if the goal is to play and review them all, very little time will be spent appreciating the game or writing decent reviews. The games will just feel like obstacles to get past as quickly as possible.
One thought I had, although I don’t know how this would be implemented, would be to have games to review assigned. Like you know how when you join a team based online game and it automatically assigns you to the team that needs more players? It’d be like that, but with reviews.
That’s why I’m doing two awards, and one is for quality.
If everyone uses their personal shuffle and goes off of that order for reviewing games, it will give even attention to games. But if not, I think people usually set up a big spreadsheet of most reviewed/least reviewed games, and some reviewers use that for ideas of what to try next!
Some people start their own thread specifically for their own reviews, while others post in a dedicated discussion thread on the game they’re reviewing (starting one if it doesn’t exist). I’ve always preferred the latter approach to facilitate discussion of specific games. Is either one of those acceptable for the Mathbrush Review Exposition?
I’m an author this year, so I’ll be posting reviews only in the author’s forum. I’ll probably do some broad overview post if I got through a big chunk before someone else does.
I recently sorted all topics on intfiction that I have access to by views. For me, the posts with the smallest number of views of all time were my individual topics I created for each game of the 2018 competition. No one was interested in the individual threads at all, so I’m going to make one big review thread in the authors forum for my reviews. I’m also going to use IFDB’s ‘embargo’ feature to write reviews on there and have them show up when the comp ends.
My counterpoint is that not every reviewer has the same motivation.
As @markcmarino noted elsewhere, deadlines are good for some writers. I’ll be using my best judgment to set deadlines for myself that strike a balance between “covering a large number of entries” and “making sure that my coverage is worthwhile.”
When I see everyone assuming that “large quantity = low quality,” it makes it difficult for me to meet those (self-imposed!) deadlines.
I don’t think that Brian’s plan to have two awards will hurt anything. And if I’m wrong about that, we can always learn from it and do things differently next year.
You’re totally right. I probably should have worded it different because I made it sound like it was something I think will definitely happen and it will happen for all reviewers, when really my concern was that it enabled that possibility, not that it was guaranteed. And it was more targeted towards those that wanted to speed run it in order to get the rewards, not those that were being honest about it.
That happened last time I ran this, where somebody released a two-hour video which had quick, fairly simple reviews of all games. But there were also more serious reviews, too. For the Best In-Depth reviews, make I should make it an award for the ‘Best In-depth Review’, where people nominate their favorite single reviews, so even if someone only reviews a small number of (or even one or two) games, they could still win it if their review is great.
