Counterfeit Monkey: Restoring fails in Parchment?

There is a bug report at Github here. I can’t reproduce it and I know little about Parchment.

It sounds a little similar to my old problem, running out of disk space quota when running it in Safari on an Iphone:
https://intfiction.org/t/testing-counterfeit-monkey-on-iphone/10941/7

Any help would be appreciated!

1 Like

I can reproduce it. The issue is that the localStorage is getting too full to store the save file, which is about 150KB, out of the total 5MB that the browser can store. If you’ve been playing lots of things in Parchment then even if you haven’t been making lots of save files there could be lots of autosaves taking up space. Then when you try to restore, it fails because the save file is empty.

Parchment/Dialog could be storing data in localStorage much more efficiently, and it could be using IndexedDB, which has much bigger limits (over a GB I think), and it could also be evicting autosaves to make space. I have plans to make those sorts of changes, but they’re not short term plans, it will take a lot of work to get everything ready.

What can we do about this?

  1. We could make it remove autosaves to free up space. It currently doesn’t store dates for when autosaves were created/used, so we’d probably just have to delete all of them.
  2. Without changing the whole Dialog system, we could make it store files more densely. Each byte currently takes up about ~3.5 characters, but it could be compressed using base32768 into ~0.5 characters/byte. (I was researching this literally 2 days ago.) That’s an improvement of 7 times!
  3. Inform doesn’t check the results of glk_stream_close to see if bytes were actually written. I’ve reported this as a bug to the I7 tracker. But we should also check that GlkApi is returning 0 when Dialog can’t write to localStorage. If you wanted to change Counterfeit Monkey’s save code you could patch that part of the template layer. Actually if anyone did want to work on that, it could be made part of the 6M62 bug fixes extension.

Although I do want to overhaul how the Dialog system works in Parchment, I think it would be worthwhile looking at doing 1 and 2 before then. They would make a big difference to how much data can be stored even within the limits of localStorage’s 5MB.

Any thoughts @zarf on this?

1 Like

base32768 looks nice.

Using IndexedDB should be reasonable.

Evicting autosaves… there’s one autosave per game, so this would be silently trashing games you’ve played earlier in order to support the one you’re playing now. This is going to be reported as a bug either way.

1 Like

IndexedDB is an async API, so it will need the whole JS interpreter ecosystem to be rewritten to be async. Which is my long term goal. I’ve identified 7 GlkApi functions that will need to be made async, but now I’m wondering if glk_stream_close should also be made async, or else how will failures be returned to the VM?

Autosaves in my mind are an optional improvement, so losing them isn’t a bug. Some VMs don’t even have them after all. But when the Glk API is made async then it would be possible to prompt the user and check if they’re okay with sacrificing autosaves to make space for save files… or if we record dates for them then we could make it just delete autosaves that haven’t been accessed for 6 months or something.

Ah. I didn’t realize that IndexedDB turns it into A Project. :)

Autosaves are optional from the 1980s point of view, but pretty necessary in the context of browser-playable games. But then if space is limited, you’ll run out eventually. But then it might not be the right answer for a very large game to swamp the needs of every other game.

@Zarf Just wanted to check if my thinking in regards to glk_stream_close is reasonable. I haven’t checked what Quixe does, but Emglken buffers files itself and only writes when the stream is closed. (It would probably be good to periodically write open files too, right now will be written if you close a page while the stream is open…) This means that glk_stream_get_position will return meaningful values even if the file stream won’t fit when it is eventually closed. Is it appropriate to have those other stream functions work and return stream positions based on how much has been written, and then have glk_stream_close return a result saying that 0 bytes were in fact actually written?

I see that RemGlk writes the glk_stream_close results before actually attempting to close the file stream, so this would require shifting some code around.

Ugh. Now I’m imagining the situation in which the periodic writing does work, and so the file isn’t empty, but the final closing fails because it just filled up the storage… The typical situation will be writing a save file which all happen in one go at least, but this could happen for a transcript. Maybe the thing to do is if fclose errors then actually open the file and check how much was written to it.

Or maybe glk_stream_close shouldn’t return 0, but Inform should try reading the file to check that it’s a reasonable length for a savefile. If you keep calling glk_stream_set_position(str, 0) and overwrite the first byte 100 times, should glk_stream_close return 1 or 100? If the latter then I guess there really is no way within the Glk API to be told that a file write failed, and instead Inform should just check itself.

So no, my thinking on glk_stream_close was not correct, the function should return the number of characters that were attempted to be written, as section 5 says:

Each stream remembers two character counts, the number of characters printed to and read from that stream. The write-count is exactly one per glk_put_char() call; it is figured before any platform-dependent character cookery. [For example, if a newline character is converted to linefeed-plus-carriage-return, the stream’s count still only goes up by one; similarly if an accented character is displayed as two characters.] The read-count is exactly one per glk_get_char_stream() call, as long as the call returns an actual character (as opposed to an end-of-file token.)

So this means that there is no way within the Glk API to be told that a file write failed, and instead the users (interpreters or Glulx storyfiles) should check themselves whether the file was written correctly. I will edit the I7 bug I wrote.

I’ve also made it so that Parchment now encodes files in base32768 for localStorage, so you should be able to store about 7 times more data there now. This is live on iplayif.com and from what I can see everything is working properly. Let me know if you see any bugs.

I haven’t made it delete old autosaves, but I intend to look at that soon.

Here’s a pretty base32768 encoded savefile from Worlds Apart :wink:

傀睴貤⏖⨳㫽瑴ⳚҤ晼铭鶇熓䯱焈枤俹▼銎貰盈蹡鋄咒㵘晵錌桤程廠曂䙰㿝Ⲍ润䥂ᆑ木詴Ҡ⪠ᒀᘀ⭐㙈㙀ҠҠڥݠ䛀Ҡ晠Ҡ硦ᛀҠ䤀Ԁ晠ҨҠҠ㙁几ᴀᇠԀ㙀ႤጼብҠҤ䙀隰Ҡ㩀ᡆႠ䛀欠ᑠႰԘ剠ҶҢ⋁ރݠ劀䡐ᖠҠҠ嫠欠ႠႰԘ■ҤҠ翠湠ሀ▰ԈҠҺҢႠҤԀ⻀♘ҸҺᒅҠҠ䏀㡐ڠڌᯊҠҪ߁䙀Ҡڨ◜Ұұң曠Ҡ䙀ᗠ㙘Ұұң曠Ҡ䙀ᯐ㙜Ұұң曠Ҡ䙀▟ꊠҠҠҥ晡䙍浖鹠ҠҢဎҢ晻㐭陠ҠҨረҥԖ䈛虠Ҡ▟ꞐҪڬ帗晠Ҡ㘿ꝀҴߘ闏ҠҠ䘿ꛀԈᇱ蔾ҠҠԟꡀҠ▟ꋠƃ

2 Likes

I’ve submitted a pull request to check for an empty savefile in 10.1. The same code should work in 6M62 though.

1 Like