Transcripts collection proposal

Technical Development Interpreters
1

There have been quite a few discussions this year about automatically collecting transcripts from testers/players, as some of the following links show. I’d like to make a more formal proposal now of a set of technologies which hopefully will meet everyone’s desires. I will make three largely independant proposals here, the protocol, a metadata extension, and a Glulx/Glk extension for tagged output.

[size=150]The transcript protocol[/size]

I’ve wanted the protocol to be simple and extensible. It should be easily implemented in all languages, and useful for all IF systems. I think JSON sent over HTTP POST requests will fulfil these needs. In the following descriptions, almost everything is optional, though the more that is sent the more useful a transcript will be, of course. Remember that within an object {} the order of elements does not matter, and that each key can be used only once.

The handshake

Firstly, the interpreter sends a message to the server giving the details of the story file. Either an IFID or a URL must be sent, with an IFID being strongly preferred. The other items are all optional, though as much information should be given as is possible. The interpreter must have the user’s permission to send contact details.

{ "ifid": "ZCODE-2-080406-A377", "url": "http://mirror.ifarchive.org/if-archive/games/zcode/LostPig.z8", "release": 2, "serial": "080406", "contact": "Dannii <curiousdannii@gmail.com>", "interpreter": "Gargoyle 2009-08-25 (Git 1.2.6)" }

The server then replies with a session ID and a list of supported transcript types. If the server sends a HTTP code other than 200 or a non-JSON response, or it does not reply with both a session ID and at least the “input” type, then the client must not attempt to send any transcripts. Due to the protocol’s extensible nature the client can send more data than the server supports, but that just wastes bandwidth. An example response is as follows:

{ "session": 7253, "input": 1, "response": 1, "tagged": 0, "time": 1 }

If the client returns both a session ID and at least the “input” transcript type, then the client may proceed to send transcripts. Note that the “tagged” item could have been left out, with no difference to client behaviour.

Sending transcripts

Again, the client sends transcripts over HTTP POST requests, but after the handshake, it does not need to listen to the response. A fairly full example is as follows, with an explanation of the elements after that:

{ "session": 7253, "log": [ { "time": "2010-07-18T03:04:12", "input": "x fsih", "response": "You can't see any such thing.", "response-type": "libmsg", "response-code": 30 }, { "time": "2010-07-18T03:04:48", "input": "x fish", "response": "Looks tasty." } ] }

A client first sends its session ID, followed by the logged commands. A client may send one command, or 100, and the server must be able to accept as many as it sends.

If the server indiciates that it supports time stamps, please send them with the following ISO 8601 format: YYYY-MM-DDTHH-MM-SS. If timestamps are not sent then the server should do its best to keep commands in order, and may use the time of the HTTP request instead. As JSON arrays are ordered structures, these timestamps must be strictly increasing.
The input command will always be sent, and if the server indicates that it supports response, the corresponding response may be sent too.
If the client has a way of detecting response types and codes it may send those too. Suggested types are “libmsg” for library messages, “parser” for parser errors and “runtime” for runtime errors. Again, check the server supports “tagged” logs first.

Here is an example of a minimal log, which may always be sent, even if the server supports more detailed logs:

{ "session": 7253, "log": [ {"input": "x fsih"}, {"input": "x fish"} ] }

Hanging questions

  • What to do with key/mouse/hyperlink etc input?
  • Have an explicit exit code?

[size=150]Metadata extension[/size]

An custom interpreter may be set up for sending transcripts, or the transcript server may be specified with a querystring parameter (for web interpreters). But in most cases it will be useful to keep the server info with the story file itself, so that it may be used with any terp which supports transcripts. I propose a simple extension to the iFiction XML record, which in most cases will be included in a blorbed story file. Both and are required. must use the HTTP protocol, and must be in the format of “3-letter-month-name day, four-digit-year” eg, “Aug 9, 2010”. The is required so that terps won’t attempt to send transcripts long after the author has stopped wanting them (or after the web server has been taken offline), it may be set to many years or even centuries into the future (but don’t do that unless you can guarantee your server’s uptime!)

<transcripts> <url>URL</url> <lastdate>DATE</lastdate> </transcripts>

[size=150]Glulx/Glk extensions[/size]

This part of the proposal is actually where I really need help. Basically it would be good to be able to compile Glulx games which output tagged responses (with a type and code) so that the terps may then easily capture that information for the logs. Juhana’s proposal emitted stuff like “#ERROR:[parser error code]#” to the normal output streams, which is a simple option we could add, with a gestalt check so that other terps don’t see this nasty message. One advantage of this is that it could also be used with other VMs (in the Z-Machine for example, by first checking for a new previously unused interpreter number.) But it might be better to use new Glk streams. I don’t really know.

Suggestions?

2

Yes, this looks very good!

Have you thought of what the session ID should be? I realize this is just an example but 4 digits are not enough to prevent collisions. Perhaps it would be good to give some recommendations on what kind of ID the interpreter should generate.

I would also like to see an option to add the player’s name and contact information to the handshake. This could be used for beta-testing games online and the player’s information would be for the author to credit them and contact them for additional feedback.

3

Also, I don’t particularly like the “#ERROR:[error code]#” way of doing it. The problem with that is that you have to make two versions of the game, one for online play and one for off-line interpreters. If there’s a way to hide the codes from interpreters that don’t support this feature it would of course be the best way.

4

The server generates the session ID, not the client, and can generate anything it likes: increasing integers, random numbers, IP addresses, timestamps, names of fruit. Anything can be used, and the client will just send it back again.

I like your idea of a contact field. How about “contact”: "Name "? Leave it up to the terp to collect the name before the handshake.

We do of course need a way for compatible terps to be detected. With Glulx this is very easy, we can just have a new gestalt value. If we also want Z-Machine support then it would be possible to do by checking the interpreter number… which is not nearly as good a solution as doing so would preclude other extensions. Perhaps we could have a new interpreter number and a new custom gestalt opcode? Then if we ever wanted to extend the Z-Machine again we could use the gestalt opcode.

However even once we’ve detected that we’re using a compatible terp, we still need to send the data somehow. Just printing out #ERROR etc is probably the simplest, but it’s definitely not a clean solution… that’s why I’d like feedback about Glk streams.

5

In order to allow more detailed review of transcript data, I think the “location” should also be added to the basic response call. This allows someone to write all of the responses to a database and then query based on location.

David C.

6

I’d also suggest a way to capture the version of the interpreter / library running the story file.

Certain bugs can cause otherwise valid player input to be flagged as errors. Glk libraries that lack Unicode support, for instance, will often display such parser errors. This could be frustrating for authors to troubleshoot without some way to group transcripts by interpreter.

On a related note, an exit code field could allow authors to distinguish sessions that ended normally from those that ended in a crash.

I’d encourage you to specify a more rigorous session ID mechanism. I often play games over the course of days or weeks. Any transcript session ID should be valid for an arbitrary duration, and not result in different transcripts being interleaved.

Unless you add a TTL field for the session ID, which may be useful for its own reasons: it provides a way to mark otherwise active transcripts as complete.

7

DavidC, what kind of location do you mean?

In any case, you could always add it to your own terps/servers. If other people think it would be useful to them too, then I’ll consider adding it to the recommended handshake.

Ben, sending the terp and terp version is a great idea! Should it be a bunch of items, or more like “Gargoyle 2009-08-25 (Git 1.2.6)”

I also like the idea of an exit code. But, what about if you want to pick it up later from a savefile?

I still don’t think that we need to specify how to generate session IDs. Wouldn’t it solve the situation you describe if the session ID gets stored in the savefile? However that makes me think of another issue… what if the server has already cleared the original log? If you have a session ID you should send it with the handshake, but then use whichever ID the server returns.

8

One possibility would be to extend the initial handshake to include not only story file information but also previous session ID, if available. That would allow interpreters to provide that information to the server if it’s stored in a persistent cookie (for web interpreters) or in save game information. The server, of course, is welcome to ignore the previous session information, and at the very least would need to make sure the story file information matches that of the actual previous session.

9

The latter, I think. I have something like the browser UA string in mind.

Well, you could probably just ignore the save / restore question, since it should be mostly clear from the transcript where the player is in the game world. Restoring from a save within a single game would append to the current transcript. Restoring it at the start of a new game would print the introductory text through the first prompt, but there would be an early save command to signal the transition.

I’m not sure how I feel about storing the session ID in the save file, if it’s meant to be a temporary variable tied to the individual player. In theory save files could be shared by any number of people, though in practice I doubt this happens often.

It might make more sense to task the interpreter with generating a UUID for each transcript session and ask the server to validate that identifier. It could either reject (non 200 code) or reply with the session ID. Then the interpreter can just write out _.txt somewhere and submit at a convenient time.

The session ID would function as a shared secret; the server will only accept updates for that UUID if the client also sends the correct session ID. In the event we wind up with a single community server hosting these transcripts, we want to avoid the situation where a malicious user or spammer could corrupt valid submissions by sending text fragments to random UUIDs. This is especially important if the transcripts would be available for search engines to index.

10

You make some sense Ben. Although I would hope no one would feel the need to be malicious, we should be thinking of it anyway. Can you outline in full how you suggest sessions be managed?

11

Tweaking the session handshake should be fine.

From the interpreter, UUID generated per RFC 4122:

{
   "ifid": "ZCODE-2-080406-A377",
   "url": "http://mirror.ifarchive.org/if-archive/games/zcode/LostPig.z8",
   "release": 2,
   "serial": "080406",
   "contact": "Dannii <curiousdannii@gmail.com>",
   "interpreter": "Gargoyle 2009-08-25 (Git 1.2.6)",
   "uuid": "f81d4fae-7dec-11d0-a765-00a0c91e6bf6"
}

The server should perform minimal validation on the UUID, to ensure that a “future” UUID has not been requested. This prevents the (improbable) case where an uncommonly dedicated attacker decides to bombard the server and claim all possible UUIDs, which could otherwise produce a denial-of-service scenario.

Assuming a database-backed transcript store, a further sanity check would query the DB to make sure that the UUID was not already in use. Failing either check would result in a non-200 HTTP result. A UUID that passed these checks would be added to the list of claimed UUIDs, along with the randomly generated session key.

Then from the server:

{
   "session": 7253,
   "input": 1,
   "response": 1,
   "tagged": 0,
   "time": 1
}

Where session represents the “secret” key that any subsequent transcript submissions must include alongside that UUID. Per your original proposal, this could be virtually anything with at least a few bits of entropy.

Subsequent client submissions then look like this:

{
   "uuid": "f81d4fae-7dec-11d0-a765-00a0c91e6bf6",
   "session": 7253,
   "log": [
      {
         "time": "2010-07-18T03:04:12",
         "input": "x fsih",
         "response": "You can't see any such thing.",
         "response-type": "libmsg",
         "response-code": 30
      },
      {
         "time": "2010-07-18T03:04:48",
         "input": "x fish",
         "response": "Looks tasty."
      }
   ]
}

Upon receipt, the server would immediately check the UUID and session keys. Invalid ones would cause that submission to be discarded, ideally without any further processing since the remaining content should be considered malicious.

Certainly this is overkill, given the overwhelmingly polite behavior of the IF community, but remote transcript logging presents a somewhat unique case of two-way trust. Both the client and the server deserve a way to determine that the other side plays by the rules. Otherwise the value of the data being exchanged is sufficiently low that there’s no reason for either party to accept the risk.

12

Having both a UUID and a session ID mean the session ID could be shown on the web (in URLs etc)? I guess that’s an advantage.

What would get associated with a savefile? And do you think it should be stored externally to the savefile?

You suggest using the old type of UUIDs which were based on time and MAC address rather than version four which is almost entirely purely random? Why?

13

The room location in the game. I’m not sure what part of your proposal is “required” and what parts are fluid, but I assume the actual JSON object that’s logged by the client can any number of elements, as long as the “well known” elements are assigned to the proper names.

So if I wanted to have a client report:

{
“uuid”: “f81d4fae-7dec-11d0-a765-00a0c91e6bf6”,
“session”: 7253,
“log”: [
{
“time”: 1
“input”: “x fish”,
“response”: “You can’t see any such thing.”,
“location”: “The Kitchen”,
“inventory”: “You are carrying a knife, a wooden spoon, and a small crystal.”,
“score”: 45,
“max-score”: 100,
“score-ranking”: “amatuer”
}
}

I would think “input” and “response” are “required” and everything else is optional.

I would say that I would prefer “command” or “user-command” over “input”, but I’m just being picky.

I’m sure it’s also obvious that using this as a transcript logging feature is only one use. This methodology could also be used to implement Channel IO.

David C.

14

Room name would be very useful. That way if you see an error message you can pinpoint the problem to where in the game that happens.

Maybe it would be wise to leave that part of the specification open, so that you could add whatever info you want to the client’s response? Room name, current scene, inventory items, debugging data…

15

Ahh room location! I thought you meant some real world location.

Well the minimal example I gave had only the session info and the bare list of commands entered, the “input” item. But you could definitely include everything sent over all channels. I’d encourage you to implement a compatible log protocol in the FyreVM, though I’m sure you’ll understand that the particulars of the channel IO wouldn’t be part of the recommended standard (just because no other system uses the data in the same way).

While you could use this as some sort of IO transport layer (and I would plan to use something similar for my web IO system if I ever get around to making it) I’d hesitate to make it too complicated. I don’t think it would be appropriate for it have rich media (ie, formatted text, windows, images etc.), but instead represents the plain text transcripts that most terps can produce.

You’ve both raised something though which I think could be worth putting in the standard: how about submitting the status line when errors are logged (leave it out for the rest.) Now the question is, log the status line as it was before the command was entered, or after (as sent with the response)? Are they likely to be different?

16

I wasn’t suggesting your proposal turn into Channel IO…I was just pointing out the similarities and secondary usage possibilities. I like JSON and RESTful stuff. I’ve been playing with Comet lately and am trying to figure out if there’s an IF usage with it.

David C.

17

I wouldn’t think there’d be too much use for Comet (server push) technologies… unless you’re doing a lot with timed events etc. Standard HTTP would work fine for non-timed stuff. Though perhaps there are uses surrounding IF but not in IF itself… you could have live chat for others playing a story for example.

18

I would probably use the UUID as the public field, since that is the one that should be reasonably unique. Guaranteed unique, if the server disallows reuse of the same UUID, which I think it should.

I would be inclined to not store anything with the save file, or at least make that optional and up to the library author. Clever libraries could track UUID / session ID across different game launches, appending transcripts from a restored game to the previous session. Regular libraries could just generate a new UUID / session ID every time they launched the story file, even if the player subsequently restored from a save that had a transcript underway.

I’m not sure there’s a lot of value in one monolithic transcript versus several partial ones. Authors that have been through the testing process might disagree. One thing I would like to see on the server is some sort of automated transcript processing that builds on zarf’s demo of interactive transcripts, posted to raif a while back. For that to work you’d have to match and interleave heterogenous script fragments, so it should be OK for clients not to guarantee one transcript per game played.

Another reason to favor multiple transcripts is that if you have exit code tracking, plus some way to get at the location per David’s suggestion, you can see if there’s a high rate of abandonment at a certain part of the game. That could point to content problems with that area.

Well, time-based UUIDs could be validated based on the encoded time, where random ones cannot be. But that validation isn’t that useful, and I guess it must be easier to use the random UUIDs from the JS interpreters. So that’s probably a better direction.

An additional security feature would be to have the initial client request send a checksum of the story file alongside the story UUID. The server would have to store a list of valid checksums for a given story file, and compare that against the client submission. Invalid checksums could be blocked outright or cause the transcript to be graylisted, pending moderator review.

This would prevent the case where a malicious author publishes a new IF game claiming the same UUID as a prior IF work, causing players to unwittingly corrupt the transcript pool for the earlier game. It doesn’t stop an individual from spoofing both the checksum and the UUID on his own system, but it does more or less prevent him from getting other players to participate.

19

I’m not sure if it’s a good idea to send the status line as it is. Inform default is room name on the left, score and turn count on the right, but the author can change this to whatever else. Perhaps the client could send the status line as a backup if the game doesn’t specify the current room’s name, but it shouldn’t be trusted to contain any useful information.

20

From the point of view of an author who’s keen on this, the most useful things to receive would be a printout/transcript, which you could read through looking for errors/improvements which don’t necessarily look like bugs; and some kind of “replayable input transcript” - just a list with line breaks of everything the player inputted, so you could replay to the error point yourself.