With IFComp coming up, I thought I’d bump this to add some general thoughts, which might not be concrete stuff to look for, but how to look for it. There also might be “what I don’t look for.” Fellow testers, steal/borrow/ignore as you please.
I realize I have one rule which is hard to quantify, but I try to keep it in mind: what will give the author the most benefit for the least effort on their part? This isn’t “help the lazy author out” but rather it’s an interesting challenge to focus on what I feel is the most important thing and potentially help the author not be blindsided with an “oh, if only I’d known.” I remember seeing this a lot in the IFComp author forums, where I show a line or two of code and the author realizes, oh, I can do a lot with that, I’ll want to for a post-comp release.
Of course you want to go through and mark off bugs you see, but you want to do and be more than just a pedantic clerk. (Some days if I don’t have a lot of energy or creativity this is a valuable mode to be in. But I want to take at least one shot doing more than that.)
I generally try to balance stuff I know (CREDITS, ABOUT, parser errors, can’t go that way errors, listing exits) and even contrarian-testing/trying to break stuff with exploration-testing. CREDITS/ABOUT is a great way to get warmed up–I suspect others have different ways to mentally warm up, and they might not seem like they work well, but they do!
I really do enjoy text extraction to see the insides of a story, or to read a chunk of text, or even to try to figure how to hit it, or even use them to see how to solve a puzzle and then evaluate whether I or the author missed something there.
I also sometimes change up what I need in the middle of a game. If something isn’t implemented, or there’s a nonfatal bug going through it, obviously I report that, but I try not to report bugs in the same category that seem to go through the same code path. I’m generally okay with one or two big things missing, with the faith that the author will fix them before release.
Obviously the closer something is to a deadline, the fewer detailed bugs I look for.
If there’s a week or so left, I look for a game-breaking bug, or if there is none, I look for something that could be fixed with a quick feature. Maybe disambiguation or better error messages with a parser. But I don’t throw too much at the author. There’ve been a few times where I say “Okay, I’d fix these two things, but I’d focus on number one for IFComp, because you want to test it and make sure it doesn’t break code elsewhere.”
I like to try to test something other testers haven’t looked at (so authors! Let testers know!) or something that is hard for a programmer to shift gears to test. From my own experience, it’s relatively easy to test the main line of my game. But I have trouble with alternate paths, paths that make sense but I can’t focus on.
So the #1 thing is, I generally look for stuff I’d bungle or avoid and then I test that pretty hard. In that vein I also ask, what part was tricky for you to program?
Again this isn’t covering for the author’s laziness–it’s just I recognize testing certain minute details would get in the way, and the author doesn’t have infinite time. It’s more useful and enlightening for someone who hasn’t seen the author’s world to poke at it.
I also like to ask the author several questions about what they feel confident works and what doesn’t. Sometimes knowing what is low value to test works well e.g. if someone has vetted the game text from the source code, I’ll worry more about how the narrative flows. If someone says someone else has tested hints, I’ll look for alternate ways through. If someone has hit the beginning or the end hard, then I’ll look for the middlegame.
(Side note: in one WIP this year in fact we had someone who extracted game text into four files: NPC dialogue, descriptions, etc. So one person could focus a bit extra on each. That worked really well.)
Sometimes I’ll just concentrate on a room that sounds cool. Or an incident I can relate to. For instance, if there is a sports or chess game as part of the narrative, I might see how authentic that feels.
But I think the most valuable things I enjoy looking for is stuff that takes time to sort out, maybe even replay.
When something doesn’t feel right, but it should. Perhaps there’s an inconsistency that I can’t explain. Or a puzzle is too easy or too hard. Or it seems like the game’s being fair to the player or giving enough hints, but I can’t put my finger on it. These are the most rewarding things for me to find and figure, and sometimes I need to play through twice. They also help me with my own game and puzzle design. I like going from “this feels like a weak spot but I don’t know why” to having something concrete to say.
Or if I’m given a walkthrough, I try a new path through the second time. It’s a way of showing myself I understand the world. There’s a balance here–spoiling everything, well, spoils things, but fighting without a walkthrough uses valuable time.
I think I understand things best when I read a walkthrough and play through again a few days later. I find it easier to evaluate 1) how well did the author clue this and 2) how rewarding is it to figure things out and 3) what tweaks could be made so that the puzzles work. And I enjoy those moments where I figure “Oh! That’s what the author means to do!” whether or not the disconnect was my fault or theirs. Obviously we’d rather have them before release than after release.
On the slightly greedy end of the spectrum, I look for two things.
- there have been rare times when a reviewer says “I really liked (neat feature X)!” when I, in fact, suggested X. Bragging publicly about this is wrong. And no, it hasn’t happened THAT often. And fixing a feature has nothing on creating the whole big story. But it feels good. And it gives me confidence that my testers will point out features worth implementing or tweaking. Sometimes I get a sense that the author has a good command of what they’re doing, and if they only knew this trick or method I bet they could do other things too … and I hit the mark enough that it’s worth sending them weird I6 code or whatever.
- I like really looking into a game to give myself ideas for my own. Obviously plagiarism is all kinds of bad, but a half-created game inspires me as full games can’t quite. I say, instead of “wow! How did they do that?” more “Wow, look what they’ve built and are building. Maybe I could build something similar, but make it my own. I’d have to change X and Y and Z, of course.” So I look for those moments, and they inspire me to look some more.
There’s a lot of golden rule/platinum rule stuff here. But I don’t think it’s too kum-ba-yah.
I think one thing to remember is that we’re not trying to be the best tester for the game, and there’s no reward for finding the most bugs, or the biggest bugs. I suppose this gets more into the general mindset for testing than what I am specifically looking for, but I like to say “Okay, what can we find today?” without being too free-form about it.
And another thing is, there have been entries where I’ve seen stuff that I know is tough to fix, and I genuinely enjoyed them despite that tweak not being in place. It is a relief to me–that I don’t have to do things perfectly to have something people will like, and they’ll accept if I miss something here or there. That’s not an excuse to deliberately miss stuff.
But it’s reduced my fear of making mistakes, whether in writing or playing games, or in real life. I can say “I can punt on X” or maybe even “I’ve been wasting time. It’s worth using my tieme better to focus on X.”