The 2011 XYZZYs are complete; congratulations to the winners, and many thanks to the presenters and voters who helped put this thing together.
Now we get to sort out the long-term issue: how do we run the XYZZYs in future?
Goals:
a) encourage higher rates of voting from IF-literate voters,
b) discourage tactical, crowdsourced or random voting,
c) maintain the values of the XYZZYs:
- as a reflection of informed critical opinion,
- as more than just a recap of Comp results. There are games that get mediocre results in ratings-based comps but do very well in the XYZZYs, and vice versa. I think this is awesome; it gives a more well-rounded picture of the State of IF. It’s also important that the XYZZYs are able (and likely) to reward games released out-of-comp.
- as a flexible, living institution that can evolve with the IF community.
d) keep the system straightforward to implement and easy to understand,
e) as far as possible, be welcoming to newcomers.
The following list is intended to cover every crazy and not-so-crazy idea that someone or other has suggested in my hearing in the past weeks; inclusion on this list doesn’t mean that I think the ideas are sound. Some are directly intended to deal with the outsider-voting problem; some aim more generally to boost in-community voting; and some are unrelated to that whole issue and are just intended to improve the awards. Your thoughts are welcomed.
1) Get rid of open voting and replace it with a restricted pool of judges.
a) A named panel of judges, rechosen each year. (By whom, and how?)
b) A narrow qualification. Might include past nominations or victories in the XYZZYs, placing in the top N of the IF Comp, etc.
c) A sponsorship system, with judges added to the pool by invitation from other judges (possibly, as with the Oscars, a XYZZY nomination gets you an automatic invitation.) Allows the judge pool to grow in an organic fashion, and to whatever extent the community feels makes sense. Might require a limit on individual invites (otherwise one person could flood the judge pool). The judge pool is inevitably christened the Ancient Qabalistic Order of Old Ones Solemnly Assembled for the Destruction of All that is Good and Pure in IF.
d) A broad qualification. Might include N posts on intfiction, N Frequent Fiction points on IFDB, an ifMUD account at least N years old, authorship of any game listed on IFDB, having a tester credit on any game listed on IFDB, or having been declared evil by Poster. This would be preferable to 1b), but much more of an administrative hassle.
Generally speaking, I dislike all the options here. When I first came to the IF world, the ability to vote in IF Comp and the XYZZYs was a really big part of feeling invested in the community. If we had to pick one, though, I think c) is probably the least troublesome.
2) Weight votes to benefit more desirable voters.
a) Ask voters which games they’ve played, and give more active game-players more weight in vote tallies.
Problems: creates disincentives for the honour system (or, if you prefer, ‘rewards liars’). Unclear (have you ‘played’ a game if you quit after three moves? If you just read a transcript?) Even if everybody’s equally honest, I think that someone who’s played a dozen speedIFs probably isn’t as qualified a judge as someone who’s played the three biggest games of the year.
b) Weight voters more highly when they vote for a more diverse set of games (but that would encourage random voting in categories you don’t care about, which we really want to avoid.)
c) Weight voters according to various community qualifications as in 1). (Most of the same problems, only more fiddly. And just because you have community cred doesn’t mean you’ve been paying attention this particular year.)
d) Weight voters according to their participation in the events of the previous year. This could be more easily automated, and would be more difficult to game. Still, I’m uneasy about this because a big part of the value of the XYZZYs is to be distinct from the comps; non-comp games are at enough of a disadvantage already.
e) As with d), but with points accumulating indefinitely. (If you voted in Comp '12 and Comp '13 and Comp '14, you get extra weighting for all of them.) Not loving this one.
f) Weight voters according to how much money they give us. (In the first year we collect $0.02 and a counterfeit zorkmid.)
In general, I dislike this kind of solution because whichever way you slice it, it boils down to someone superimposing an artificial hierarchy on the pleasantly organic field of community membership -- inevitably we'd screw some people and over-represent others. It'd suck if we did it openly (so that everyone knew what their weighting was) and it'd suck if we did it secretly.
3) Allow voters to leave comments with their votes; in effect, ‘prove that you’re human and actually like IF.’ (Comments could then be given out anonymously to authors.) This could help separate out stuffed ballots (or at least force ballot-stuffers to be more clever), while giving authors the sweet feedback they all lust after. (Possibly weight votes with comments more heavily? More voter burden.)
4) Do the nomination and final-voting rounds differently. (For instance, have the nomination round be restricted or weighted voting and the final round open-voting.)
This could allow one system to offset the excesses of the other, though it would be even more fiddly. Don’t put too much weight on this, though; if people can’t vote in both rounds they may not want to vote in either.
5) Make the expectations of XYZZY voting more explicit, put them somewhere more obvious, and set clearer standards for the circumstances under which votes will be discounted. We’re definitely doing this, in one form or another; it’s the exact manner of doing so that’s the issue.
a) Forbid authors from campaigning (and maybe testers and other interested parties, too.)
b) Forbid anyone from campaigning.
c) Allow discussion but not campaigning, so ‘Here are the games I think should win’ is uncool, but ‘Here’s some discussion of the Best Setting games and why you might be interested in them’ is okay. (But distinguishing between the two would be wildly impractical.)
d) Ban campaigning on non-IF-related social networks. Sensible, but difficult to distinguish: my Planet-IF presence is just a selection of stuff from my personal Livejournal. You might use your Twitter account mostly for talking about game design, but that doesn’t mean your non-IF-playing relatives don’t all follow it.
e) Allow people to self-police on campaigning/discussion, but set a clear standard for when the organisers can throw out votes. (Probably helpful only against naive or unintentional ballot-stuffing.)
f) As with e), but allow for the throwing-out of games, not just votes. (Probably unfair to authors, who may not be able to predict whether a casual mention will unleash the floodgates.)
g) Ask second-round voters not to vote in categories where they haven’t played N games, for some value of N that is probably 2.
6) Use some kind of preferential voting system for the second round, rather than first-past-the-post.
This would protect the Awards against relatively small-scale vote-flooding, such as happened this year. Problems: more complicated, and mathematics geeks will bemoan whichever system we choose (probably at length and with formulae). Also, this is likely to lead to even more skew towards heavily-played games, particularly comp games. The results would begin to look more like a ratings system.
7) Allow voters to nominate as many things as they want in the first round.
a) Go to three rounds: in the first, you can nominate as many games as you want, and 1 vote is the minimum requirement to reach the next round. This could whittle the list down to a somehat less scary size, and give voters more opportunity to play more of the nominated games. But it’d be extra-fiddly; do people really want to vote three times?
b) Replacing the first round, producing a nominee list of the same size (i.e., four or so). This becomes more like instant-runoff voting; it would strongly favour well-known, broadly acceptable games (i.e. Comp winners). This, I think, would do little to safeguard against crowdsourced votes, and I’m unsure about how much help it would be with voter apathy.
8) Get rid of the first-round list and make all votes text-entry.
Problem: would probably lead to more votes wasted on ineligible games, could lead to games being overlooked, etc.
9) Get rid of list compilation; to get on the list, authors have to propose their own games (and puzzles, NPCs, technological developments, whatever) for consideration. (They could do this throughout the year, so that if you wanted to you could release your game and submit it for XYZZY consideration at the same time.)
Benefits: makes for a rather shorter, less intimidating list. Deals with the ambiguities of the text-entry categories.
Problems: rewards authors who pay close attention to this kind of thing. Runs afoul of the Culture of Authorial Modesty. (SpeedIFs have won XYZZYs before, and that’s great; but it looks kind of arrogant to suggest that your speedIF might deserve a XYZZY.) If anything, it could make community-invasion tactics easier (for your consideration: Minecraft.) If everybody always submits every game they publish, we’ve gained nothing. If enough of the year’s significant authors (or tech developers, etc.) forget to submit their games, the XYZZYs aren’t really a reflection of the year’s best games any more. Finally, it forces authors to guess about what the best thing in their game is (since it doesn’t tactically make much sense to nominate yourself for three different NPCs or whatever).
10) Compile a first-round list for Technological Development and Supplemental Materials.
Problem: we don’t quite know how to do this. Just as it’s not clear what counts as a single, discrete puzzle, these are rather fuzzy categories. Does every bullet-point of a new I7 release make it onto the list? Every piece of cover art? At the moment we can leave this kind of question in the voter’s court, but that leads to relatively low voting in those categories; and some things may be overlooked because people don’t associate them with a particular year, etc.
11) Ban CYOA from consideration for most of the awards.
a) …and create a category for Best CYOA.
b) …and duplicate all the categories, e.g. as Best NPC in a CYOA, Best Puzzle in a Parser Game.
Both of these would mean that we’d have to compile an entirely separate CYOA list, which I don’t know would even be possible. We’d have to distinguish between IF and CYOA, between CYOA and not-CYOA-webgames (both of which are difficult now, and not getting any easier), and then we’d have to dredge up a year’s worth of CYOA from a thousand little corners of the Internet. Even if we could do this, we’d be taking ownership over CYOA to a degree that I’m not sure is merited.
c) …and just declare the XYZZYs parser-only.
The issue of whether CYOA can legitimately be judged on the same scale as IF is fairly contentious, but the average position seems to be that it's okay to do so, even if the standards are a bit different. Of course, this wouldn't address the risk of vote-flooding for parser games, which there's plenty of potential for.
12) Extend final-round voting, find other ways to motivate people to play more of the nominees.
The rationale here is that many potential voters, particularly the more conscientious people who we really want to vote, end up not voting because they haven’t played enough of the nominees.
The second round usually lasts about three weeks. As a thought-experiment, let's imagine that you want to play all the nominees, either to refresh your memory or because you haven't played them before. For the 2011 Awards, that would have meant playing twenty-two games, even if you count the four Hat Mystery games as a single unit. Compare this to the task of playing all the Comp games. Usually that's 30-something games. Comp voters have twice as long -- a month and a half -- to play them all. And for a number of obvious reasons, the average XYZZY nominee is going to take longer to play than the average Comp entry. Of course, XYZZY voters will usually have played a good number of the games already; but it's usually still a tall order to finish up the list.
a) [b]Email a reminder to voters a few days before voting deadlines.[/b]
13) Do something to stop motivating authors to tactically avoid voting.
Right now, authors are allowed to vote on any game but their own. This means that, if you think you have a decent chance of being nominated in a category, your best bet is to abstain from voting. This creates a perverse incentive: competent, active authors are precisely the kind of conscientious, informed voters we’d like to see voting, but we’re getting them to abstain.
a) Let authors vote for their own damn game. Every other democracy allows it.
b) Don’t let authors vote for their own game, but instate a mercy rule: in the event that your vote for another game would hurt your own game’s chances, that vote is void. (Fiddly to manage, but not impossible.)
14) Rename some of the awards to be more transparent.
a) NPC and PC are terms derived from role-playing games, are fairly common in discussion of computer games; but outside the Geek Arts they’re effectively unknown. Changing them to something less opaque might help the idea that people who don’t have that background are welcome too.
b) The awards don’t really account for games with multiple or highly-variable PCs. Slightly change the sense of ‘Best NPCs’ to ‘Best Characters’, to include any and all PCs.
c) Rename, restructure and define some or all of the categories to make them more IF-specific. (But getting that right would be pretty difficult.)
15) Change nothing. The kerfuffle this year was a one-off incident; such incidents should be dealt with on a case-by-case basis, and we shouldn’t fundamentally change the awards just because we’re nervous.
16) Split the awards into Judged and Open-Voting.
a) Duplicate all the awards. This would effectively create two awards that happened to run in parallel; it would be easier to split off one of these into a different set of awards (and let someone else run it). Also, there’s a risk of dilution: the more awards we hand out, the more they start to look like third-world dictator medals.
b) Have some awards judged and some open. Some awards, like Use of Innovation, Implementation and Technical Development are more obvious candidates for restricted judging, because they require more background knowledge. For most of them, though, the choice would feel pretty arbitrary.
17) Only include games that are listed on IFDB.
That way, if anybody feels that a game really should be included in the list – anybody to include the author – they can go and add it themselves with a few minutes of effort. This would make it easier for the XYZZYs to compile entries and to link to all the games.
18) Add categories for Best CYOA and Best Parser Game, leaving the other categories open to both. (Again, if we have Best CYOA we run into a representation problem; it’s silly to have a CYOA award just for the relatively small number of CYOA works released annually in the IF sphere, but it’d be very difficult to compile a full list of all the CYOA published anywhere in a year.)