Best practices for testers

The first piece of advice on the IF Comp guidance for authors page is to playtest your games. But with Comp season coming up – and the concomitant testing requests – I realized that though there’s copious advice for authors (not to mention prospective judges), there’s not really much out there to help folks be good testers. So I thought it might be helpful to open up a thread where folks can share how they approach testing, or for authors, things testers have done that have been especially useful (or things to avoid), and hopefully create a useful resource moving forward.

(My other motivation here is that I while I feel like I’ve got a good handle on how to be a good tester of parser games, I’m less confident about how to approach choice-based games. So I’m hoping to get some advice here, especially on stuff like learning how, or whether, folks communicate the particular path they take through a choice-based game in the absence of built-in transcripts)

To get things started, here are some things that have worked for me (at least I think they have; if folks whose games I’ve tested have feedback for me!):

  • Per the above, if a game has a transcript function, make sure to use it! This is a no-duh, of course, but I’ve personally messed this up in a few ways, usually when forgetting to re-enable transcript tracking after reloading or restarting the game (in Gargoyle at least, I think the transcript will continue after typing RESTART for a Gluxe game but not a z8 one…) So these days I try to open up the file after a couple moves at the beginning of each session to make sure everything’s working.

  • When I have time, I usually like to do an initial playthrough more or less straight – mostly approaching things as a regular player, albeit one who likes to examine everything mentioned in room descriptions and try out lots of synonyms for parser games – but then a follow-up one or two that’s more focused on trying to break stuff: taking everything even if it seems nailed-down, cramming stuff into containers, stacking things onto each other (I had a lot of fun with this testing The Impossible Bottle), taking wildly suboptimal choices or ignoring clear prompts to do X Y or Z thing, to try to stress-test things. The theory of separating these two is that providing a sense of pacing and how the game flows can get really lost in the latter case.

  • Speaking of that, I try to provide feedback on big-picture issues like pacing! Usually in addition to sharing a transcript or detailed list of feedback (for a choice game), I write 3-5 bullet points summarizing my overall take on the game, and flagging areas where I think the author could consider making more significant changes. Often this is where things like structure, voice, etc. come in since it’s hard to address those in a granular way.

  • And then the flip side of the previous one is that it is of course helpful to provide that super granular feedback – I like to drop lots of comments, earmarked by *'s, in parser games, to flag typos, missing scenery items, buggy responses, etc., but also positive feedback where a joke lands or a puzzle clicks so authors know not to mess with something that’s already working. I also like to provide a bit of running commentary, like what I understand my current goal to be or whether I’m frustrated by some busywork or finding it no big deal to work through, since sometimes it can be hard to assess that kind of stuff just from looking at the transcript. For choice games, as I mentioned, this is often harder – I try to keep a text file where I paste in whatever chunk of the game I’m on and add comments from there, but honestly that can feel clunky.

  • This is a suggestion for authors, but I think it’s helpful when they provide specific prompts for where they’re looking for feedback beyond just “look for bugs and typos”; it helps focus my attention as I’m playing. The flip side of that is that sometimes alerting a tester to an issue makes it harder for them to assess it the same way a player who comes to it “cold” would, of courses – but I think the tradeoff usually cuts in favor of asking for the feedback that will be useful.

  • For puzzle games I really try not to rely on hints to the greatest possible extent – this isn’t always possible, but I find a good middle ground when stuck is to send a status update to the author lightly fishing for clues (“I made it to the inner cloister but now can’t get past this one ornery monk – I think I need to get him to inadvertently violate his vow of silence so he’ll give up the habit and let me through”), since that lets the author step in if there’s a bug or I’m wildly off base, or let me trundle along on my merry way if things are basically fine.

  • Lastly, I think it’s important to communicate clearly with the author on your timeline. Sometimes testers can’t get to a game for a couple days, or even weeks, which is totally fine – we’re all doing this for free (or so I assume!) – but it can be rough on authors not to know when to expect feedback, or whether they should get the tester an updated version reflecting bug-fixing they’re doing in the meantime.

  • For parser games specifically: examine anything mentioned in a location description, including using any adjectives mentioned. Then examine anything mentioned in those descriptions, until you run out. LISTEN and SMELL whenever it seems even slightly interesting to do so. Try any custom verbs on any object you can, especially inappropriate ones (in Inform at least, it’s easy to write an action that applies to more kinds of things than you’ve written responsive logic for). TAKE ALL whenever you can. Always X ME. Try to drop plot-critical items and leave them behind in inaccessible places. Try all the different potential conversation verbs (TALK TO, ASK ABOUT, TELL ABOUT, SAY…)

I’m sure there are lots of others – and lots of things wrong with the stuff I’ve listed above – but I’ll stop there since I’m curious what works well for y’all!

18 Likes

Also: be honest. I appreciate people trying to be nice in their criticism, but it isn’t kind to let big bad game design flaws go unchallenged. It’s one thing if the game just isn’t your cup of tea, but it’s another thing entirely if the gameplay or the narrative has big problems.

I always appreciate my testers pushing me to be better and telling me if I’ve got issues. Some of them (definitely looking at you here, Mike) will even help you brainstorm a little to fix the problems. So I think it’s important to say what you like and what you don’t like, and to be kind but honest about it.

7 Likes

I think one thing you and Amanda both touched on is that you want to be positive without being deceitful and so I use a lot of stock phrases such as “this worked well, but this can be improved”. I keep remembering: I’m not here to pass judgement. That’s for the judges! I’m there to help give them something more likable to judge.

It sounds a bit unemotional, but on the other hand, emotion can get in the way of fixing bugs. I avoid overreaction. I know as an author it’s disappointing to let stuff I thought I’d fixed, or I know I should’ve checked, slip through. My task as a tester, as I see it, is to remember someone has shared something they suspect has faults, and then help improve that work as efficiently as possible, because testing is about how much the author wants to improve, not how clever I am noting what is wrong, or how persuasively I can sigh “it’s been done.” Also I note if someone makes the same sort of mistakes I might have made, and I say so, and I express confidence the author will straighten it out. This feels like a Golden Rule.

One other thing I like to do is have a game surprise me. I’ve made a critical comment, and suddenly I realize I overlooked something! I realize I may not have been careful looking at certain things. Or I get through part of the game and I realize I wish I had spent more time in another area, and the author should know that. To this I’d add–every time you make a criticism, note that it might only be worth changing if someone else finds something wrong. Note if it’s a pet peeve and also note that if you’re the only one making the criticism, it’s bad (Note: for obvious technical errors, there are no two ways. Report it and move on.) When a tester admits they were wrong, it flips the script and can make the author feel smart.

I’ve also come to grips with this: I can’t have a universal style. I try to be thorough, and I want to do more than just plow through, but there are things I am just not going to cover, whether it’s due to lack of interest in the subject matter or whatever. So I try to leave notes to the author to say “Hey, maybe a different style tester would enjoy checking this” or “this might be good to engineering-test.” In this vein I’ve also found open-ended questions to be useful. Maybe only 2 of 10 hit. But the ones that do are often useful. And I know as a programmer, I need to be asked some open ended questions more than once before I get a good answer. Sometimes questions testers ask are things I wondered if I should bother with, and then I say aha, yes, it was.

And a lot of times for technical stuff I’ll provide 2 suggestions: one, a quick and dirty solution, and two, something more detailed if the author has more time.

I know it stinks to forget to make a transcript, but here’s where I’d suggest a trick that works for inform, for the authors.

After reading a command (this is the ignore beta-comments rule):
    if the player's command matches the regular expression "^\p":
        say "(Noted.)";
        reject the player's command.

I suspect other programming languages have other catchalls. In the case of inform, you can also easily put this (and other code maybe describing what you want from beta testers) in an extension, and you then disable including that extension one week (or whenever) before release.

One of the big things I do is to have an email ready at the start of testing. Every time I make a comment I want the author to see, it goes into my email. I roughly try to sort things by expected boost to game for the time taken. As an author it feels good to have a flying start by fixing a big bug quickly. Ideally, of course, there are none, but fixing them always feels like a win, and I want to give an author as many quick wins as possible so they can get on a roll. I also try to provide a roadmap of what can wait for a post-comp release, but if the author doesn’t get to it, no problem. If they get ahead of schedule and slip it in, it feels good. But I want that sort of thing to be something you add so a post-comp release feels substantial.

So typos don’t get a mention (though I may say “search the transcript for the word TYPO,”) but a runtime error would be at the top of the list, especially if I know it’s the sort I’ve seen in my own works due to one careless line of code. Or being locked out of a room, or feeling locked out, or noticing intended hinting pushes the player away.

Logic/continuity errors where I try specific things, or internal contradictions in the story, are lower. I also try to look for big-picture stuff where the author probably needs to reorganize huge chunks of code and let them know. Maybe it’s something they’re aware of, but I also want to have it on record how complex I think it will be and how critical it is to me. I try not to mention typos in my email, because I’m not morally attached to them, but I do tend to find them.

Yes…speaking of big-picture, this sort of stuff is important for me to sit and think out. I’m good at finding out the technical side quickly, but a lot of times I just have a sense something could be better, and I don’t know how or why, and it’s hard to express, and I have an idea 2 days later. So I let authors know I may have technical stuff immediately and then subjective stuff later. Also, as you alluded to, it’s important to communicate with the authors how you work. I know if I’m going to get busy, I like to give them immediate feedback for stuff that might have low risk high reward, not to say “HA I FOUND A BUG RIGHT AWAY,” but so they have something to do and to show I’m thinking of stuff in good faith. I often try to add something that worked for me. Then I follow up with more detailed stuff later.

I also try and look for pieces with moving parts that might conflict and try to test them. Certainly I do what I can to get the game in an unwinnable state, and if a game survives my stress testing, I let the author know.

As an author I find getting too large a transcript at once has a risk of making me wait until I have a huge time chunk, and I actually appreciate them being broken into small chunks, even/especially for a long game.

On that note, if you feel you’re just plowing through, also maybe take a break and come back later. I get cranky and start complaining to myself about irrelevant stuff when I’m tired. Don’t feel you have to tackle the game at once. The author should have given enough time in advance.

I agree here. I try to let my thoughts leak out, and often I’ll find me correcting myself later. I sort of mentioned above it’s cool when the penny drops. That does feel clunky for choice games but it’s not too bad to hit ctrl-a and paste in a commentary.

I agree on this (more info is generally good) and generally try to balance one author suggestion with one thing I think would be tricky for me to code, so there’d be a lot of things to work on there. My wheelhouse is 1) change “You can’t go that way” to something that provides atmosphere and 2) be more helpful with parser errors or standard Inform rejects (e.g. YES and NO feel a bit snarky.)

As someone who knows inform pretty well I also think I have a good handle on when an author just doesn’t know certain code would be easier than they think, so I look out for that. Stuff like having a “not for release” section that does

every turn when hint-flag is on:
    try hinting;

So they can test the hinting code, if it seems to misfire more than once.

Also sometimes the author’s suggestions what to test provide clues as to a blind spot. If they say “Well I had someone test this thoroughly so don’t waste your time there,” I’ll listen. If I see something odd in an area they didn’t mention, I look at it more.

One other thing I’d add–it’s a nice touch to check what time of day the author can work at what they have. I’ve had a lot of fun waking up to an email saying “New build!” from someone in Australia and knowing I can wait a few hours and still give what seems like immediate turnaround–but of course they will wake up to a hopefully helpful transcript. Sometimes when I get a game I think “when should I test it” and actually narrowing it down to just before when the author could actually use it is a big help.

As for the bad side? I remember one tester who told me “This game suffers from what I call AGI-itis” and the thing was, I knew it was a dry goods game, and I was willing to accept that as a weakness, but that wasn’t the focus. This wasn’t a huge slap in the face, but it felt unnecessary and obtuse. A vernacular way of putting it is, stick to the facts when you can, but don’t be all Mr. (or Ms.) Actual-Factual.

4 Likes

I’ve been unable to figure out what “AGI-itis” and “a dry goods game” might mean. It’s clear that you found the tester’s comment unnecessarily insulting, but I don’t get it.
I don’t feel I’ve ever been insulting as a tester, but my advice is to err on the side of honesty. What an author really needs is information. If something doesn’t work for you, it may not work for others. The author wants to know that before release.

6 Likes

Well, it was part of a much longer screed, and words like “Your game suffers” is sort of a red flag. More simply, stuff like “I cannot possibly see what possessed you to write this” could, well, be condensed. Let’s not be that guy!

But the evaluation sort of meant the game was something you’d see in the early Sierra days, where you traded items with NPCs until you got what you wanted.

“Dry goods game” basically means find item A, give item A to person B, get item C, give C to person D, and so forth. Which has its limitations, and I was willing to accept that. But I wasn’t so willing to accept hearing that I apparently had no clue of just how limited my game was!

I agree that honesty is a good thing. If I think a fix would be too risky to implement before release (whether due to the author’s technical skill or the complexity of the fix) or the author would turn off some people with certain content, I let them know. I always ask myself if I am being honest or being blunt. But I think it’s useful to use stock phrases like “this seems relatively weak” or “this seems relatively strong” or “this seems worth a roll of the dice to try and get an a-ha moment to try and fix.” I try to avoid full-on punditry mode when giving advice, because 1) general principles and 2) I wasn’t asked to be a pundit, and the author probably doesn’t want that sort of drama.

4 Likes

I started beta testing after a lot of experience finding bugs in Infocom games. I’ve also found spectacular and interesting bugs in The Pawn, Eric the Unready, Not Just an Ordinary Ballerina, and other games no one can ever fix. So my focus has always been technical–I’m looking for the bugs. That’s a good kind of tester to have, as long as you have other styles in the mix as well. And here are a few of the things I do, although @DeusIrae already covered a lot of it.

  • Learn the “right” way to play the game. Going through and trying to figure it out like a “regular” player should generate a lot of helpful feedback.

  • When you’re comfortable playing through the game the way it’s intended to be, stop doing that! Try to break everything. Use objects the wrong way, Use special-purpose verbs on the wrong objects–I’ve had great success with verbs like EMPTY and THROW doing things they shouldn’t, when they’re not even important to the game. Leave things where they shouldn’t be. Go places you shouldn’t be. Try things that shouldn’t work. Act on knowledge the PC shouldn’t have yet. Try to violate every assumption the game might be making.

  • When you find a vulnerability, try to exploit it, to leverage it like some hacker trying to take over a system. If a bug is letting you get away with something you shouldn’t be able to do, use it to make the biggest cheat you can. Comments are good, but it can be more effective when your transcript shows the author how you hacked the game.

8 Likes