JSON extension

Technical Development Inform Extensions
1

I’ve written an extension to add JSON support to Inform 7. It could be used as an alternative to tables for persistent data, as well as potentially a way of transferring save games between different story files (if you’re willing to manually serialise all the things that can change in the world.)

Get it here: https://github.com/i7/extensions/blob/master/Dannii%20Willis/JSON.i7x

Also an extension for flexible arrays and maps: https://github.com/i7/extensions/blob/master/Dannii%20Willis/Collections.i7x

8 Likes
2

Personally, I think you kinda buried the lede here, Dannii. So far as I’m concerned, this is a game-changer, an I7 holy grail: a dynamically created arbitrarily complex data structure so that suddenly there’s a decent general solution for passing multiple parameters to a rule, or to receive multiple results from rules or phrases.

A whole bunch of code I’ve written to evade those limitations in much worse ways just became irrelevant and I’m grateful for it!

2 Likes
3

Haha, I guess you’re right. I hadn’t thought of using the structures in that way, but they can.

Don’t make circular structures though, there’s no safety checks, and it could send the program into an infinite loop.

This extension doesn’t let you include things, rooms, rules, phrases or many other Inform kinds within the structures. It would probably be possible to extend or modify the system to allow for all those kinds too, although they couldn’t be output as JSON. It might be better to do that as a new extension though?

3 Likes
4

That’s really great!

I’m working on a JavaScript evaluation extension (think Vorple but only the evaluation part) so I’m going to use your extension immediately.

I’m already playing with a shortcut syntax:

To decide which JSON reference is (obj - JSON reference) -> (k - text):
	decide on get key k of obj;

To (obj - JSON reference) -> (k - text) = (value - JSON reference):
	set key k of obj to value;

It allows:

json -> "foo"
(json -> "foo") -> "bar"


json -> "foo" = JSON number 1
(json -> "foo") -> "bar" = JSON text from "baz"

I’d love to remove the parentheses but Inform gets confused if I write json -> "foo" -> "bar" :frowning:

1 Like
5

Yeah a whole lot of short cuts could be made, but I’m not sure if they should be part of the extension proper. You could also do these:

To (obj - JSON reference) -> (k - text) = (n - number):
	set key k of obj to JSON number n;

To (obj - JSON reference) -> (k - text) = (t - text):
	set key k of obj to JSON text from t;
1 Like
6

I’d use => to avoid any potential conflict with the built-in ->. And it’s easy to fix the associativity issue with dereferencing, if crude… go, say, 5 deep and one wouldn’t be likely to need more:

To decide which JSON reference is (obj - JSON reference) => (k - text):
  decide on get key k of obj;

To decide which JSON reference is (obj - JSON reference) => (k - text) => (m - text):
  decide on get key m of (get key k of obj);
1 Like
7

I was hesitating with this solution and this more complex one where I embrace the right associativity and define phrases to collect the keys in a list.

To decide which JSON reference is (obj - JSON reference) -> (keys - list of text):
	repeat with key running through keys:
		now obj is obj -> key;
	decide on obj;

[ Group the two rightmost keys. ]
To decide which list of text is (t1 - text) -> (t2 - text):
	let l be a list of text;
	add t1 to l;
	add t2 to l;
	decide on l;
	
[ Prepend the other keys to the list. ]
To decide which list of text is (t - text) -> (l - list of text):
	add t at entry 1 in l;
	decide on l;
8

@Dannii I believe I found a bug.
Context: I’m JSON stringifying a text that I write into a file which is read by JavaScript.
When encoding a text with characters outside the ASCII range, the resulting string is not valid in JS.
Such characters should be escaped as \uXXXX.
I’ve replaced the default case in JSON_Stringify_String with:

			default:
				if (char > 127) {
					print "@@92u", (Hexa4) char;
				} else {
					print (char) char;
				}

where Hexa4 prints the 4 digits hexadecimal represention of the character.

Here’s my naive implementation (I don’t know I6 that much and haven’t considered any edge cases):

[ Hexa4 n;
	print (Hexa1) n / $1000;
	n = n % $1000;
	print (Hexa1) n / $100;
	n = n % $100;
	print (Hexa1) n / $10;
	print (Hexa1) n % $10;
];


[ Hexa1 n;
	if (n < 10) print n;
	else print (char) n + 'a' - 10;
];
9

Unicode is fine in JSON, the issue is that I7 only knows how to read and write ASCII. If you need higher codepoints then you’ll need some custom file functions. (Maybe there is already an extension for that?) So I don’t think I’ll add escaping for higher codepoints as it would just make strings so much longer. Except maybe for surrogate pairs, as they need to be escaped? I guess they’re easy enough to escape.

However the parser definitely does need to support Unicode escapes.

I thought about adding a traversal function, where you pass in a query string, kind of like the command jq, something like "key1 key2 [3] key4". I wonder if these extra utility phrases should be part of the main extension or in a second extension?

So it’s actually quite easy to add other types into the system:

JSON room type is a JSON type.

To decide which JSON reference is a/-- JSON room (R - room):
	(- JSON_Create((+ JSON room type +), {R}) -).

To decide what room is (R - JSON reference) as a room:
	(- JSON_Read((+ JSON room type +), {R}) -).

But I’m considering splitting off the object model into a new extension, perhaps called Collections.

It would be possible then to make the object/map type support non-string keys - any non-reference (array and object) keys would be easy to support. Would that be useful?

10

I posted an update to support parsing and stringifying non-ASCII. The default stringify phrase will only escape surrogate pairs, and there’s a new “escaping non-ASCII” option if you need that.

1 Like
11

These phrases let you store any kind of value in a JSON reference, but you have to manually track what kind it is:

JSON generic value type is a JSON type.

To decide which JSON reference is a/-- JSON value (V - value):
	(- JSON_Create((+ JSON generic value type +), {V}) -).

To decide what K is (R - JSON reference) as a (name of kind of value K):
	(- JSON_Read((+ JSON generic value type +), {R}) -).

To be used like this:

let D be JSON value 3 hours 15 minutes;
say "[D as a time][line break]";

I wonder if there’s a way to reliably turn every kind of value into a unique number… is there any way to get the name of a kind of value as a text? Or return the I6 kind number?

Well this works for any sayable kind, including objects, arithmetic values, scenes, external files, use options, etc:

To decide which JSON reference is a/-- JSON value (V - sayable value of kind K):
	(- JSON_Create({-printing-routine:K}, {V}) -).

To decide what K is (R - JSON reference) as a (name of kind of sayable value K):
	(- JSON_Read({-printing-routine:K}, {R}) -).

But the error message if you try to dereference it with the wrong kind doesn’t work, it says JSON type mismatch: expected <illegal json type>, got <illegal json type>. We can’t get the name of every kind, but I could make it say JSON type mismatch: expected <illegal external file>, got <illegal scene>. Better than nothing! Maybe with a little text editing we could remove the <illegal part too. But for kinds without illegal values (like numbers), it would just print the number…

I feel like this is very promising… not needing to make phrases for all the different kinds makes a generic Collections extension much simpler to write and use.

type of (JSON reference) won’t work with any of these, but we could have a phrase if type of (JSON reference) is (name of kind of sayable value K) which probably covers most uses of it.

12

9 posts were split to a new topic: Unicode File IO extension

13

5 posts were split to a new topic: Collections extension

23

I’m having trouble with double iteration:

	let json be parse "{'foo':1,'bar':2}";
	repeat with key in json:
		say "key: [key].";
	repeat with key in json:
		say "key: [key].";

Result:

key: foo.
key: bar.
key: 
Glulxe fatal error: Memory access out of range (C1E4082)
24

That was a silly bug. I’ve pushed an update that fixes it to Github. :slight_smile:

2 Likes
25

Thank you for this quick update!
I learned I could define my own looping phrases, it’s nice. And that you can have multiple loop variables.

I’m playing with:

To repeat with key (key - nonexisting text variable) and value (value - nonexisting JSON reference variable) in (R - JSON reference) begin -- end loop:
	(-
		{-my:2} = 0;
		if (JSON_Get_Type({R}) == (+ JSON object type +)) {
			{-my:2} = BlkValueRead({R}-->1, LIST_LENGTH_F);
			{-lvalue-by-reference:key} = BlkValueRead({R}-->1, LIST_ITEM_BASE)-->1;
			{-lvalue-by-reference:value} = JSON_Object_Get_Key({R}, {-by-reference:key});
		}
		for ({-my:1} = 0: {-my:1} < {-my:2}: {-my:1} = {-my:1} + 2, {-lvalue-by-reference:key} = BlkValueRead({R}-->1, {-my:1} + LIST_ITEM_BASE)-->1, {-lvalue-by-reference:value} = JSON_Object_Get_Key({R}, {-by-reference:key}))
	-).
2 Likes
26

Is there a technical reason not to have:

To decide what text is (R - JSON reference) as a text: (- 
	JSON_Read((+ JSON string type +), {R})
-).

?

It appears to be working but I imagine there’s a reason text & list are handled differently.

27

That will clone the text. I thought it would be better to access the internal text directly, and most of the time, when used with a new variable, the let phrase will appear the same. It’s only if you wanted to assign it to an existing variable for some reason that the text phrase would trip you up.

When I change JSON to be based on the Collections extension then you’ll be able to clone read it as well as accessing a reference to the internal text. This could be quite confusing…

2 Likes
28

Is there a “bridge” between JSON & Collections?
I’d like to parse some JSON and then use it as a collection map and add some keys/values that are not JSON-compatible.

29

Not yet. I’m working on a third extension that will supersede Collections, and will then rewrite JSON to use it.

1 Like