Public alpha & beta testing vs. spoilers

Authoring General Design Discussions
1

after the major criticism on written english and proofreading along three comps, I’m seriously assessing if is best or not changing the testing programme for Isekai.

That is, instead of a final public testing of the release candidate, I’m assessing releasing first a final public alpha test release in the last two month of this year (2024), followed by a final beta test release after the next IFComp (2025), then the already-planned final RC test after the IFComp 2026.

What held me to switch into this programme is that, albeit largely incomplete, there will be spoilers, and I dislike the taste of “installments” this procedure carries.

But the advantage of the “might army of proofreaders” noted in the reviews is not only considerable, but also necessary, as the twin fiasco during the last pair of the IFComp shows painfully.

So, there’s two conflicting instances; on one side delivering a finished, enjoyable story, on the other story, parcelising that story in multiple partial installments can easily ruin the final, complete story

OTOH the, let’s call it uncommon, nature of the story, narration & setting creates a bias in the interest of prospective close ßtesters, attracting testers very inclined to being favorable and/or appreciating said uncommon nature (that is, not only the “scientific research” :wink: ) of the work. And the constructive criticism in both IFComps (and the backgarden) shows that, albeit without compromising my narrative, something should be done ecumenically toward different tastes. and this requirement needs an open & public testing.

So many pros and cons, so I ask request for comments on this issue, whose I think is of interest for other authors, even those of mothertongue english.

Best regards from Italy,
dott. Piergiorgio.

2 Likes
2

I have experience of this kind of dilemma as my WIP is very long, with ongoing continuity, and is in chapters (which amount to your instalments). It needs lots of testing overall. But I really want to preserve the overall surprise of the final work.

For testers I admit to the process, there is the problem that it is months between each chapter arriving. They can forget what’s happened in the game, or become overused to some bits they’ve played repeatedly. And if they only experience it in pieces, then in a way, no-one is actually testing the effect of the whole thing.

There are no easy solutions to these issues, but my main strategy has been to identify and reserve testers for different tasks and time periods. I think it’s always wise to do this with testers in general (unless you go with an open beta, in which case you get a wide range of testers but no control or planning, and probably some will drop out) but I take a more targeted approach to it than usual with my game due to the length.

For instance, I have done things like keep a particular tester out of the process until many chapters were done. In the meantime, my core tester was testing episodically, resulting in lots of ongoing polish, but without them being able to follow the story closely. And I’d have other people who would just test one chapter in isolation here and there, or they’d test one then leave. Eventually, I bring in the reserved tester, who can now try many chapters at once, getting their combined effect, and in a state that’s already been massively improved by others.

So overall, for a long or multi-part project: Having identified testers, their abilities, availability and reliability (and in some cases, I know these things very well already) I deliberately bring them in and out at different times, to combine all of short, medium and long-term types of testing.

Any one of them only gets their first impressions of any part once. This is why I try not to bring in everyone at the same time, at the start. I want to keep different people up my sleeve for different stages of development and perspectives.

I don’t know about catering to different tastes. I think aim for your ideal reader or readership. The thing is, there are so many kinds of fixes in IF games that improve the experience for anyone, whether they’re your ideal reader or not (but including the ideal reader) that that’s why having lots of testers works. And I guess that’s why you said ecumenically :slight_smile: Some things, it doesn’t matter who says them or how – they’ll be right. The rest of the time, you will wield your wisdom to decide whether or how to act.

EDIT PS - But yes, as you can see, I have no open betas of the whole game. That would ruin surprise for me way too much.

-Wade

7 Likes
3

I believe that beta testing is, per its intrinsic nature, spoilery. You will eventually lose a “new reader” when that person is a tester (or proof-reader), but the improvements that will impact your work are priceless and are certainly worth a couple (or three!) fresh mouths.

6 Likes
4

As Marco said, during testing spoiler consideration is basically out the window. Invariably at some point you need to explain your intentions to the testers to help them: “The point here is to collect and assemble the bomb parts because you’re ultimately going to blast a hole in the prison wall.”

In the best of all possible worlds, you’d want one or two ongoing “buddy testers” - even better if they are familiar with the system you ware working with so they can make actual coding suggestions - whom you can throw little bits of the game at as you’re writing. In this stage you’re not thinking about story flow, you’re just making sure parts of it work correctly. This is basically alpha testing where the game is not in a shippable state.

Once you get the game all hooked together and working completely, that’s when you beta test where the game is complete but you’re quashing bugs that crop up with the entire story working as a whole and polishing the prose and the pacing. This is where you are looking for bugs, but also grammar/spelling corrections, and if writing in you non-native language, suggestions about phrasing in the prose language.

Ideally you’ve had your buddy testers make suggestions during alpha, and then you want to hopefully get a couple new testers who can play the game as players initially without spoilers and that’s where you get feedback about “Does the story work?” “Do you like it?” “Are there parts that are slow or should be expanded?” - along with seeing ways that unfamiliar players try to interact with your game that you might not of thought of or solutions that are valid you did not implement. Occasionally you will have to spoil your testers if they are lost and tell them what to do and what’s supposed to be happening as part of the testing process.

I’ve had testers who have balked at answering the question “What kind of feedback do you need?” which makes it difficult to know what to focus on.

  • Do you want me to play the game as a player and just transcript so you can see how someone will experience it?
  • Do you want me to comment on typos or grammar or language? (Often you might want to not concentrate on the prose initially and want to focus on getting everything in the game to work structurally before you polish.)
  • Do you want me to “troll-test” and try unexpected things and attempt to break the game?

All these are valid separate tasks you want for testing, and sometimes testers will find it overwhelming if they’re un-directed and it’s expected for them to do all three at the same time. I appreciate when an author breaks it up - instead of wanting me to test the whole game, they say “Just see if you can get out of the first room.” That will go a lot faster and you’ll get focused notes since you’ve told them what you need.

TL;DR: Testers are not players and it isn’t helpful if you are reluctant reveal plot spoilers or puzzle mechanics to them. During the testing process you often will have to spoil parts of your story to give the testers knowledge of what feedback you need and how to help you.

5 Likes
5

Thanks, Marco and Hanon, for the insight !

The issue is finding some ßtesters which can be capable of the first and second point, and optionally the third: some of the alpha testers have opted out for various reasons, but I suspect that they actually want to enjoy the full, complete experience.

on the second point, the most critical, TADS3’s text output (Os) is horribly broken, outputting many lines of garbage for every actual text; If worked, with a simple randomiser script (text are outputted in the order are found in the source, and source is by (theoretical…) definition well-organised) I can generate a “limited-spoiler textdump” good for the language issue. (oh, trying to figuring how a story goes from randomised-ordered strings can be a fun puzzle, even worth the resulting spoiling :smiley: )

but anyway, when I consider stable enough, I’ll give a call for the second, hopefully final, alpha test… Thanks to everyone for the insight !

Best regards from Italy,
dott. Piergiorgio.