Ghosterington Night Test Plan & general test plan questions

So people here probably remember EctoComp. I tested Wade Clarke’s (severedhand here) Ghosterington Night and added some advice. He added some features, and I decided to see how long it would take to write a test plan to test them.

The answer was, not long with Google Spreadsheets. And it definitely saved me more time in verifying his intended changes than it cost to write it up. Perhaps it saved him time having all the stuff he needed fixed organized.

Though I’m not sure how ultimately useful test documents are. In Inform, it seems the simple stuff can be (somewhat) trumped by the “test x with ‘s/e/take all/open door’” command. And the smaller stuff may be too picky. But it seems there’s a lot of room for addressing stuff a test-script can’t easily pick off, or for checking on stuff that is random or semi-random.

The link is here, and naturally there are some spoilers. But I wanted to give it as an idea for people who wanted to write a game or see about a test plan. And someone could look at the original version of Ghosterington, see what could be implemented, and try the 2nd version to see how testing would work.

I’d also think that instead of x’s we could have the date or version of the last build where a case was tested. And of course, test cases could be sorted by urgency e.g. stuff we need to test all the time (getting through the game) would be on sheet 1 and minor stuff would be on subsequent sheets.

And I think having a googledoc for test cases would help a lot just to remind the programmer of what broke or to remind the tester of what the programmer worries will break.

Btw, terminology: conditional pass = “it mostly seems to work, but I’m not sure if it works as intended” or “it works except for an annoying special case that might be too tough to fix.” And “fail” is not an F-grade in the traditional sense–it’s more, yes, this still needs to be fixed, even if it looks better than before. “Pass” is actually a substantial achievement, especially with complicated features.

This may be too formal for some people, and I suspect I have been too technical in some places and not technical enough in others.

So I will ask–what do other people have? What have they tested? Are there any other test documents or guides or game-specific examples? Maybe we could create basic test plans for various languages e.g. X ME not being generic, SCORE being implemented correctly, and ABOUT/CREDITS/HINT as well?

I’d also be curious about format improvement suggestions anyone has for the test plan. I’ve only scratched the surface of spreadsheet functions with =counta(B3:B43) etc. and suspect there are easy ones that can do more.

I built an IF-test framework this year (for any system with a “dumb” stdin/stdout interpreter). It’s low-level, but somewhat more capable than either I7’s “test” command and I7’s skein: it can check responses, and also cope with a certain amount of randomness.

I wrote this for the giant game I’m working on, but it’s turned out to be useful even for tiny games. Just having a thing I can run through to verify that the walkthrough works is a relief. (That is something the skein can do, but I have terrible luck with Mac skein bugs so I’ve never tried to rely on it.)

On the other hand, it’s not a test plan that I write in advance. Since it’s checking specific output strings, I have to implement a situation or event in the game, run through it by hand, and then turn it into a test case. So it doesn’t address that part of what you’re talking about. (It’s great for the “test every time” part.)

(Also, of course, it’s strictly pass/fail, because it’s automated. If I was worried about an annoying special case, I’d break that out into a separate test. HL is up to 309 test cases now…)

Inform 7 has several test extensions, including my own Simple Unit Tests. I’m not sure exactly what your tests involved, but it should be possible to do it all within your code, if you want to do that.