I also didn’t vote because I hadn’t read reviews for games I didn’t play due to spoilers, but also because my brain disease problems meant I quickly forgot the details of those reviews I did read! I very much enjoyed reading those reviews though.
I’m not sure about this competition going forward given the limited number of votes. But I think it helped encourage a strong reviewing community, not setting out to win the competition per se, but more as a community event. Which is great, and certainly a feeling I had, encouraging me to keep reviewing.
First and foremost, I really enjoyed the side-contest and would definitely like to see it run again next year (and that would be true even if I hadn’t won a prize). That said, I have a some miscellaneous thoughts.
One thing that struck me was how much voting in the review competition was conditioned in small ways by the small practices of the Comp itself, if that makes sense. For one thing, the “How do I decide what to vote? What are my criteria? What’s a five? What’s a nine? Who am I to make these determinations?” questions that went through my head while I was voting were a lot like the kind of questions that go through my head when I vote on Comp entries. I think that that may have gently discouraged more people from voting. One way to ameliorate that might be for people to do what some Comp reviewers have done, which is to publicly post their evaluation criteria for other people to mull over.
It might be helpful to have a breather in between the end of the Comp and the beginning of review voting: after two months of the Comp, it may be hard for people to rev up their enthusiasm for voting on reviews. Maybe a week for people to polish reviews up that they’ve written on Comp games and/or finish writing reviews for games they’ve played, while we wait for the official Comp results to be tallied and announced, would let people who just voted on Comp games relax a bit? On the other hand, maybe a break will cause people’s energy to dissipate and result in fewer votes because they wandered off and did the other things humans do in their human lives when their attention flags. It’s hard to predict.
I got some good, encouraging feedback from the results and would love to get more next year. One way to help accomplish this might be to have a “feedback for author” section under each voting entry, as in the Comp voting form. I don’t know how easy this would be to set up with the existing Google Forms-based setup, though.
I enjoyed this contest, as it gives reviewers a chance to give each other recognition.
I left the first question (about which reviewer provided the best coverage) unanswered, as I felt it was asking me to count how many reviews each reviewer posted and then choose the one with the highest number. And I didn’t want to do that homework!
I did answer the second question, but I only gave ratings to the reviewers I had read, which was only a few. (And the one I rated the highest, Anssi, didn’t win.). I’m not pointing out a flaw in the contest here, just sharing my experience.
If it were up to me, I would combine both questions into one, and add a second question to recognize one favorite review. So the questions would be like:
Rate the reviewers, taking into account both quantity of coverage and quality of criticism.
What’s the one review that really stood out to you? (There would be two drop-down menus here, one to choose the reviewer and one to choose the game.)
Let’s see, a couple of thoughts that may or may not be useful:
I definitely enjoyed the competition – I was trying to keep up with reviews as folks posted them but also tried not to read reviews for games I hadn’t played yet, so it was nice to have a prompt to go back over the ones I’d missed, since I found more than a few gems I’d missed. And having it run after the competition was over was nice too since it helped pass the time between the Comp closing and the results being announced. So I’d definitely support it coming back next year!
I also thought the split between coverage and individual review was a little challenging, though I think it was ultimately helpful since it allowed folks like Patrick to get just recognition for their contributions (FriendOfFred I think wrote only one review, but it was one of my favorites!) Maybe here’s somewhere where rubrics would help? Not saying my approach was great, but I tried to start with a 1-10 rating based on how many reviews a person wrote (using a nonlinear approach though since most folks wrote between 5 and 40 and I wanted to differentiate them). Then I added or subtracted a point based on the diversity of the games they reviewed (including parser vs. choice, but also less-reviewed games or those by less-well-known authors), and then added or subtracted a point based on the overall robustness of the coverage. It felt a little more interesting than mechanically assigning a score based on the number in the spreadsheet, at least.
Speaking of the spreadsheet, I’m not sure how many folks had easy access to it, since it seems like some, including reviewers, missed it. In retrospect, pinning it to the top of the forum, and/or maybe linking to it in reviewers’ threads, might be helpful for highlighting it, since having it available makes reviewing the reviewers much easier!
Getting more feedback for reviewers I think is a worthy goal, and hopefully this competition helped foster it. I’m doing an in-depth review of one of the reviewers as part of a prize I offered, which hopefully will be fun (I know I’m looking forward to it!) but that was actually inspired by Mike Sousa, who many years ago did his own review post of all the IF Comp reviewers. I think he might be considering doing something like that again this year, which would be awesome, but I think the competition was also good for pushing me to provide at least a couple sentences of feedback for the reviewers I was able to get to, which hopefully was interesting or rewarding for folks.
In terms of incentivizing more participants, I wonder whether taking a page from the Comp and explicitly asking folks to rate at least 5 (or however many) would help? That might help folks with less time to be comprehensive feel like their participation is invited and valuable too.
I’m sure others will have great ideas about better technological solutions for counting the votes and compiling the results, but that’s above my pay grade. I’m definitely willing to help out with any of the administration tasks or double-checking things if that’s helpful, though. If I don’t volunteer next year, feel free to recruit me while reminding me of this post
Thanks again for running the competition, and to all the reviewers and voters!
I just got mine in a PM from @mathbrush here on the forums. Mine boiled down to “good reviews, wish there were more!” which is about the most encouraging thing I can think of.
Another idea: Voting for reviews instead of reviewers. That way, it should be easy to vote for at least five. Coverage could be quantified without votes.
First, I’d like to publicly thank @mathbrush for running the review competition this year. I appreciate his hard work not only writing reviews himself but also encouraging other people to write them.
Regarding the competition itself, I’d like a better understanding of our collective goal.
I figure that everyone wants more public discussion of IFcomp, and
we’re assuming that more public reviews will encourage more public discussion
The review spreadsheet works to encourage discussion. When I was an author, it was fantastic to have a single place to look for updates. As a reviewer, putting links on the spreadsheet drove a substantial amount of traffic to my blog.
The competition encouraged people to keep the spreadsheet updated, which was good (even if it wasn’t perfect).
But does the review competition encourage more discussion and reviews? That’s a difficult question to answer.
I think the reviewer competition is a great idea. And getting more reviews for IFcomp is a good thing. But the other competitions could also need some more reviews.
In lack of a better measure, and in order to compare the competitions, I took data from IFDB:
IFComp 2020 IFDB reviews so far: 3.5 reviews per game (many authors are still adding some) Ectocomp 2020 IFDB reviews: 3.0 reviews per game Spring Thing 2020 IFDB reviews: 1.8 per game
Add to this that almost no blogs(?) review Ectocomp and Spring Thing, we must conclude that especially Spring Thing games get much less attention.
First, I was thinking, that if the reviewer competition covered both IFComp games and Ectocomp games (would be practical as they overlap) this would help Ectocomp get more reviews, which would be good, even though it doesn’t help Spring Thing.
Not sure what to do with Spring Thing though? It could have its own reviewer competition perhaps if anyone would like to run it. A XYZZY-award for best reviewer could also be helpful.
There’s a difference, though: In the comp, all games are released at the same time, so if everybody picks five games at random, we still get a fair vote. Reviews trickle in over time, so if everybody reads five reviews (spread out over the comp period), the early reviews will get the bulk of the votes.
On the positive side, this could incentivize reviewers to hurry up and publish their work faster. On the negative side, it could incentivize reviewers to hurry up and publish their work faster.
Why not both? Just like we have best puzzles and best individual puzzle, we could have both best reviewer of the year and best individual review of the year.
And having two categories it wouldn’t even mean more effort. Since we can safely assume that mathbrush would always win best reviewer, we could just award it preemptively and then we would only have to vote for best individual review.
Since we can safely assume that mathbrush would always win best reviewer, we could just award it preemptively and then we would only have to vote for best individual review.
You know, my reviews have always been about quantity over quality. I think that even with speed and coverage on my side, I would struggle against other reviewers in any competition that included an element of quality.
Overall, I’m not entirely pleased with the effects of the comp. I wonder if it it had a strong effect on people writing reviews. I know that some people were influenced and considered this comp while writing their reviews, but it may have just risen stress levels more than inspired. Especially since everyone writing any reviews was evaluated; there’s no ‘opt-in’.
Due to that, and the relatively small number of voting participants, I think I won’t run the competition again next year, but would be happy to pass the baton on to others.
I think you are too hard on yourself. The stress levels argument may be valid, but making the competition opt-in is an easy fix. But even if we assume the comp had no effect in promoting more reviews (which I wouldn’t), I think recognizing the work of reviewers is a worthy goal per se.
I think that’s been suggested before, but I remember Sam Kabo Ashwell, in his XYZZY-organising capacities, said he thought it was kind of too self-congratulatory. I think I agree with the spirit of his decision. I’ve just paraphrased him, though. I might have a look for that comment later (I expect it’s probably here on intfiction) if someone else doesn’t unearth it first.
