Q: Serializing/Deserializing for nested class objects

Twine Version: 2 / twee version: 3
SugarCube Version: 2.37.3

Hi everybody,

after much searching and tinkering I finally managed to use my own JavaScript classes to properly serialize/deserialize between passages etc. by using the pattern described here: Non generic object types. I used the second approach (Discrete parameters constructor), so a “simple” class (meaning without containing other classes) looks for example like this:

class Monster {
constructor(isBoss) {
this.bossFlag = isBoss;
}
isBoss() {
return this.bossFlag;
}
clone() {
return new this.constructor(this.bossFlag);
}
toJSON() {
return Serial.createReviver(
String.format(‘new {0}({1})’,
this.constructor.name,
JSON.stringify(this.bossFlag)));
}
}
window.Monster = Monster;

Now initializing objects with new Monster(false) and saving/loading as well as using the ‘reload page’ browser button too.

BUT!

This only works for “simple” classes with no nesting, like the one above. But I also want to use “complex” classes with a lot of containment and here the deserialization breaks. I get the “patched is undefined” error, like this:

Error: can’t access property “junctionEast”, patched is undefined

By “complex” I mean e.g. my dungeon contains several floors, each one with several rooms, each one possibly containing monsters, loot, etc.

I have written clone() and toJSON() methods for all my classes like in the example above. I have hoped that the serialization/deserialization engine would automatically recognize nested objects and handle them recursively correctly. But apparently this isn’t enough.

I have been debugging the code now for two days and can’t quite figure out how to proceed.

Has anyone of you encountered a similar problem and found a solution?

Cheers,

Pesha.

I suppose the first question is, why do you need to store these room/map descriptions in stateful objects? They sound like perfect candidates for holding the structure as data on setup instead.

The second question is, can you provide an example of the sort of thing that doesn’t deserailize?

Hi David,

the dungeon is not a static thing. It is generated randomly each time the player enters one. As I understand, the objects in the setup are only for immutable things?

As for an example. You know what? As I was making a simpler example (PlayerCharacter has an Inventory, this in turn has (very simple) Potion and Scroll objects.) I ran into something strange.

When doing this:

  1. create Inventory
  2. create Player
  3. add inventory to Player
  4. create Potion/Scroll and add to Inventory

This seems to work, despite being three nesting levels.

When doing this:

  1. create Chest
  2. create Potion/Scroll and add to Chest
  3. create Inventory
  4. create Player
  5. add inventory to Player
  6. move Potion/Scroll from Chest to Inventory

This breaks. I can see this in the JavaScript console:

inventory: Object { gold: 0, potionVolume: 1, scrollVolume: 5, … }
​gold: 0
​potionVolume: 1
​potions: Array [ (2) […] ]
​0: Array [ "(revive:)", (2) […] ]
​0: "(revive:)"
​1: Array [ 'new Potion("HEALING")', null ]
​0: 'new Potion("HEALING")'
​1: null
​length: 2
​<prototype>: Array []
​length: 2
​<prototype>: Array []
​length: 1
​<prototype>: Array []
​scrollVolume: 5
​scrolls: Array [ (2) […] ]
​0: Array [ "(revive:)", (2) […] ]
​0: "(revive:)"
​1: Array [ 'new Scroll("ESCAPE")', null ]
​length: 2
​<prototype>: Array []
​length: 1
​<prototype>: Array []
​volume: 6

Testing the Chest on its own doesn’t show errors. Strange.

I have to look into this a bit deeper. I have to make sure I can provide a simple example that shows the error. At the moment I guess there is something wrong with my code somewhere. I will try to reproduce the exact circumstances. At the moment the code I use is just too complex for posting. I will try to streamline it properly.

Thanks for your help,

Pesha

All right. I managed to write a simple example.

The code is here: SugarCube Test

You can rebuild it with tweego or just run the Test.html. It’s just what I have built from the sources in the src directory.

To reproduce the error just start the file, then click at the links in the main window until

<< — T H E    E N D — >>

appears. Then click at “Show Inventory” and then the reload button of the browser.

The error shows that the potion array in inventory wasn’t properly deserialized.

You can also look it up in the JavaScript console.

Both SugarCube.State.variables.pc and SugarCube.State.variables.pc.inventory are correct but the potion array is wrong.

Any idea why?

Thanks,
Pesha.

The following paragraph is for clarity, in case you or any future reader isn’t aware…

Each time a Passage Transition occurs the current value assigned to all active Story variables are cloned. The original copy of those values are added to the Progress History, and the clone copy is made available to the Passage being visited. This cloning process breaks Object Referential Integrity, which is a fancy way of saying that if two (or more) variables / array-elements / object-properties referenced the same object instance before the passage transitioning then those variables / array-elements / object-properties will each reference their own new unique instance of that Object after that transitioning. And there is no means to “turn off” the cloning that occurs during a Passage Transtion.

note: The following is a Twee Notation based.

eg. In the Define the Object Passage all three variables are referencing the same Object instance, and in the Object was cloned Passage each of the three variables are referencing a different Object that has the same “value”.

:: Define the Object
\<<set $first to {name: "Mary", room: "Library"}>>
\<<set $second to [$first]>>
\<<set $third to {md: $first}>>

''Before Property Change''
First: <<= JSON.stringify($first)>>
Second: <<= JSON.stringify($second)>>
Third: <<= JSON.stringify($third)>>

\<<set $first.room to 'Hallway'>>

''After Property Change'', the room changed for all three references
First: <<= JSON.stringify($first)>>
Second: <<= JSON.stringify($second)>>
Third: <<= JSON.stringify($third)>>

[[Next|Object was cloned]]


:: Object was cloned
''After Transition'': all three seem to be the same Object instance
First: <<= JSON.stringify($first)>>
Second: <<= JSON.stringify($second)>>
Third: <<= JSON.stringify($third)>>

\<<set $first.room to "Study">>

''After Property Change'': but they aren't
First: <<= JSON.stringify($first)>>
Second: <<= JSON.stringify($second)>>
Third: <<= JSON.stringify($third)>>

An example of this same behavior in your own code is:

1: An instance on a Inventory object is assigned to the $inventory variable in the Debug Inventory Passage.

2: After transitioning Debug Character 02 Passage, the $inventory variable now references a clone of the original Inventory object. A reference to that Inventory object clone is assigned to the is assigned to the inventory property of the PlayerCharacter object being stored in the $pc variable. At this point both the $inventory variable and the inventory property are referencing the same Inventory object instance.

3: After transitioning Inventory Description Passage the $inventory variable and the inventory property are no longer referencing the same Inventory object instance, which means any changes to either of the Inventory object instances has no effect on the other instance.

warning: There is a potential issue with the code the Inventory Description Passage.

The two widgets being called are using the inventory property of PlayerCharacter object stored in the $pc variable as their source, where as the amount of Gold is being sourced from the Inventory object being stored in the $inventory variable. Because they aren’t referencing the same Inventory object it is possible for there to be an inconsistency in the gold value being shown, although not in this specific test case.

The fact you keep assigning a reference to the Inventory object being stored in the inventory property of $pc to $inventory shows you’ve become aware of the Object Referential Integrity breaking behavior. But that re-assignment breaks as soon as the next Passage Transition occurs.

This behavior is why it makes little sense in having multiple variables / array-elements / object-properties reference the same Object instance, unless those other references are temporary in nature. Like using temporary variables inside a Passage to store the 2nd reference, or the temporary arguments of a function / widget / macro definition.

Hi Greyelf,

as you have guessed correctly, I am already aware of the mechanism of passage transitions dealing with cloning of objects. It’s true, that’s exactly the reason I dereferenced the $inventory variable from $pc and reinserted the changed values into $pc again.

But, I’m still at the beginning of the process of understanding how the SugarCube engine handles the serializing/deserializing of variables. Am I understanding it correctly that only the SugarCube.State.variables objects are handled this way and the SugarCube.State.temporary and SugarCube.setup are not?

This code is not the real code of my story/game but just a PoC. It has parts of my original code, but it was slimmed down in a very short time just to present the problem with the serializing/deserializing of them. So while I’m not 100% sure how I will address this SugarCube variable dereferencing issue later, I’m aware of it and will address it for sure.

I’m somehow experienced in OOP, so after making simple stuff with SugarCube I was quite unhappy with the limitations of the simple variable model and wanted to switch to a more convenient method for me. And since the whole SugarCube engine is written in JavaScript, why not use the full JavaScript repertoire, right? Wrong! :wink:

I have become aware of the dereferencing issues you describe and found some information in the reference manual and online, so I have played around with the clone() and toJSON() methods to find the solution. Also with different ways with inheritance and containment. But apparently I’m still missing something.

What really stuns me is the fact that at the time when the error occurs, both the $pc and $inventory are correct. Apparently both of them are deserialized correctly. Also both the referenced and dereferenced objects of Inventory are deserialized properly. So why isn’t the potions and scrolls arrays? I would like to really understand the reason so I could avoid the pitfalls. It’s apparently not as I thought originally that it’s an issue with the depth of object nesting. But since I mixed these two issues (dereferencing and serializing) into one piece of code I’m not 100% sure.

In the next iteration I will refactor the Potion and Scroll classes to be in a proper class hierarchy and try again. How they are designed in the moment is ugly and comes from me trying to use the SugarCube.setup hierarchy. I think I will now try to refactor one class after another and try to find where I went wrong. It seems that the way to go as with the PlayerCharacter and Inventory classes is not bad. They both seem to work at least.

Thanks for your explanations and your general input. It really helps to talk to more people about this stuff.

Cheers
Pesha.

That’s right. Temporary variables are never saved to anything other than the current “in play” state, they aren’t ever cloned or stored, and they don’t make it into save files.

setup is just a convenient global object for storing invarient (or regeneratable) data on. It is rebuilt every time the page refreshes or the game loads, and otherwise is just a normal JS variable that you can do what you like with.

I’d caution that JS isn’t really an OO langauge. It doesn’t really have classes and hierarchies, classes are pretty much just functions that spit out generic objects with a prototype attached, and inheritance is … well similarly not quite real. That’s why it’s somewhat more common for JS authors to separate the data from the functions that act on it, rather than encapsulating both in the same object. SugarCube adds an extra layer of complexity that makes this even more preferred, because (as you’ve seen) every object gets both cloned, and also rendered to JSON and back (using eval) quite often.

Hi all,

I have refactored the Potion and Scroll classes like this:

class Treasure
  class Potion extends Treasure
    class HealingPotion extends Potion
    class ManaPotion extends Potion
    class StrengthPotion extends Potion
    class DexterityPotion extends Potion
  class Scroll extends Treasure
    class TeleportScroll extends Scroll
    class DispelScroll extends Scroll
    class EscapeScroll extends Scroll

instead of the last ugly combination of classes and setup constants.
(I could simply define all potions and scrolls as setup constants, but that’s beside the point here. This wouldn’t test the serialization/deserialization.)

Although this approach circumvents the “ugly” error this is simply because the classes have no methods to call and checking for type of object and similar operations on them isn’t handled by their methods but by directly checking their types. So it’s not that the problem doesn’t exist but it doesn’t show up as a direct error. The data is still no more usable.

I have debugged this a bit more and what follows is the JavaScript console log (beautified by me for better readability).

This is what inventory looks like after creating:

SugarCube.State.variables.inventory
  Object { gold: 0, potions: [], scrolls: [] }
    gold: 0
    potions: Array []
      length: 0
      <prototype>: Array []
    scrolls: Array []
      length: 0
      <prototype>: Array []
    <prototype>: Object { … }

….and after putting a HealingPotion and an EscapeScroll inside (and 5 gold pieces tbh.):

SugarCube.State.variables.inventory
  Object { gold: 5, potions: (1) […], scrolls: (1) […] }
    gold: 5
    potions: Array [ {} ]
      0: Object {  }
        <prototype>: Object { … }
​          clone: function clone()
          constructor: class HealingPotion {}
          toJSON: function toJSON()
          <prototype>: Object { … }
      length: 1
      <prototype>: Array []
    scrolls: Array [ {} ]
      0: Object {  }
        <prototype>: Object { … }
          clone: function clone()
          constructor: class EscapeScroll {}
          toJSON: function toJSON()
      length: 1
      <prototype>: Array []
    <prototype>: Object { … }

This stays then consistent through passages (!), but using the reload button of the browser changes the contents to this:

  Object { gold: 4, potions: (2) […], scrolls: (2) […] }
    ​gold: 4
    potions: Array [ (2) […], {} ]
      0: Array [ "(revive:)", (2) […] ]
        0: "(revive:)"
        1: Array [ "new HealingPotion()", null ]
          0: "new HealingPotion()"
          1: null
          length: 2
          <prototype>: Array []
        length: 2
        <prototype>: Array []
      1: Object {  }
        <prototype>: Object { … }
          clone: function clone()
          constructor: class HealingPotion {}
          toJSON: function toJSON()
          <prototype>: Object { … }
      length: 2
      <prototype>: Array []
    scrolls: Array [ (2) […], {} ]
      0: Array [ "(revive:)", (2) […] ]
      1: Object {  }
      length: 2
      <prototype>: Array []
    <prototype>: Object { … } (That's the prototype of class inventory)

(Here, I didn’t expand the scrolls array, but you get the idea.)

So it seems that it’s not a principal problem with serialization/deserialization as the data stays consistent over the passage transitions. It’s only after reloading the page in the browser.
I mean, the data is somehow still there, just in a strange representation.
What the heck is going on?

Perhaps someone can shed a little light on this?

Thanks
Pesha.

Thank you David for clearing this up.

I begin to consider switching my approach completely and abandoning the whole containment idea. This will make the code much uglier and very much less readable (at least from my point of view) but if this is what it takes to make it run smoothly? Well, who am I to complain?

I still wonder. What is the difference between changing the passage where it still works more or less and the reloading of the page where it doesn’t properly.

Thank you all anyway
Pesha.

Yay! I fond the edit button for the posts! \o/
I have originally messed up the console snippets and wanted to post the correct one. But then I found out how to edit my posts, so I deleted the unnecessary post with the correction.
Are you still with me? :grinning_face_with_smiling_eyes:

So, the fact that your classes stay working through passage transition means their clone methods are working, but the fact that they fail on a save/load means that their toJSON methods are not working.

However, I’ve seen the revival code in object listings before, even when the objects were actually revived just find, due to the console’s lazy representation of objects. Have you checked their values directly in game (i.e. not through the console, and not through the SugarCube debug object)?

1 Like

note: some of the following is not 100% technically correct, as it has been simplified for convenance sake.

You seem to be including the (deliberately) undocumented SugarCube debugging interface in this question.

If this is true, then it should be noted that SugarCube’s developer has repeatably warned about using that interface for anything other than debugging a (SugarCube based) Story HTML file via a web-browser’s Console.

Hituro has explained that Temporary variables and the contents of the special setup object / namespace are not included in Progress History or Saves.

re: Progress History & Saves

It should be noted that Story variable changes made during the processing of the current visited Passage are not stored in Progress History, they are instead stored in a special Moment (in time) that represents the current Passage visit. And because a Save is at its core a snapshot of Progress History, those same Story variable changes are not stored in it either.

This means that if you create a Save while visiting one of the passages where you run re-aligning code like <<set $inventory to $pc.getInventory()>>, that re-aligning wouldn’t make it into that Save, because that Story variable change won’t become part of Progress History until the next passage transition occurs.

And as the State.variables property basically references that special Moment (in time), the undocumented SugarCube debugging interface will show that $inventory and $pc.getInventory() are referencing the same object (after re-aligning) even though they aren’t in Progress History or in the Save.

@Greyelf:
I don’t know anything about the SugarCube debugging interface, so I don’t know if I’m using it or not. I guess, I’m not? The object snippets in my post are directly from the JavaScript console from the “Web Developer Tools” of the Firefox browser where I test my code. In my understanding they present the momentary view of the de facto present data. But I might be very wrong here. It’s just my educated guess. :grinning_face:

@David:
In the last example the objects’ states are from my last code where I removed all methods from the classes in question (Potion and Scroll). So I couldn’t really test the full integrity of the data. In my previous tests, where they still had methods, calls to those methods were producing a clear error, because the objects weren’t recognized as their expected types. In this example the data integrity was probably wrong es well, because code like potion.constructor.name === potionType.constructor.name returned true before the reloading and false afterwards. Where ‘potion’ was from an iteration over the array in inventory while ‘potionType’ was e.g. from new HealingPotion()

So as a summary:
Traversing passages seems to work fine in principle. Only using the reload button and saving-loading cause the error. So this directly poses two questions for me:

  1. Is the reloading of page similar to the loading of a state from disk?
  2. I believe to have seen somewhere that one has to take extra care during saving/loading of custom classes. Perhaps I am missing some vital details here?

Thank you guys for the wealth of information you provided. These are very welcome details about the engine I never found anywhere. Or perhaps simply missed in the overwhelming amount of documentation :sweat_smile:

Cheers
Pesha.

Hi everybody.
I made a much simpler test which pretty much shows exactly where the code breaks. Here is the full source code (with the compiled Test.html file included):
SugerCube Test2

Just start the html file and click a few times on the links.

You will see that the normal operation works fine.

After looping for a few times - to show that it works in principle - once you arrive at the passage where the John Smith player data is shown, click on the reload button of the browser. You can see directly the wrong deserialization.

This example is much better because it needs only one class and basically two passages to show the problem. The additional Potion and Scroll classes are just there to showcase what goes wrong.

I don’t have a clue what goes wrong here, but I have a hunch that the problem is the Array class containing custom classes. Strings in the Array work apparently fine. In the previous example I had a custom class (Inventory) directly contained in PlayerCharacter and it worked as well. So now I think that neither the nesting depth nor the principal usage of custom classes is the problem. It really seems to have something to do with Arrays.

Do I have to do something special in the toJSON() method for Arrays? Please note that I simply use JSON.stringify(this.testData) for the serialization of the Array.

Cheers
Pesha.

I think, I found it.
Instead of

  clone() {
    return new this.constructor(
      this.firstName,
      this.secondName,
      this.inventory,
      this.strength,
      this.dexterity,
      this.life,
      this.mana,
      this.testData
    );
  }

I switched to

  clone() {
    return new this.constructor(
      this.firstName,
      this.secondName,
      this.inventory,
      this.strength,
      this.dexterity,
      this.life,
      this.mana,
      structuredClone(this.testData)
    );
  }

so basically, while cloning an Array I use structuredClone(this.array) instead of a simple this.array. The same test I posted above (SugarCube Test2) works as far as I can see as soon as I do this change to the PlayerCharacter class.

@Greyelf: I found this solution while reading your communication with @BarryGlass (Class’s Object data member not playing nicely with history)

Does this make sense to you all?

Thanks and cheers,
Pesha.

It sort of makes sense, though I wouldn’t have expected the issue to be in the clone() method.

But if this.testData is an array, then you would be passing an object reference around in your first clone version, which means your clones will break the referential integrity for each moment. Maybe the result of that is when the moments are then saved, the original array is lost, so your serialised objects are themselves broken.