How does your parser IF system stand up to the Winograd Schema Challenge

Technical Development Development Systems
1

https://www.youtube.com/watch?v=m3vIEKWrP9Q

In other words, what’s your system’s capability in disambiguating pronouns?

2

Is there a description of this online, or just a video?

1 Like
3
4

I found this paper for what it is worth.

A Winograd schema is a pair of sentences differing in only one or two words and containing an ambiguity that is resolved in opposite ways in the two sentences and that requires the use of world knowledge and reasoning for its resolution.

There are 30+ examples at the end of this paper on what a Winograd schema is. For example, this is one of them: “Sid explained his theory to Mark but he couldn’t [convince/understand] him. Who did not [convince/understand] whom? Answers: Sid did not convince Mark/Mark did not understand Sid.”

1 Like
5

I’m basing this on the Wikipedia summary, but I don’t see how that could be applied to an IF parser. Other than something convoluted like >TAKE BOOK AND APPLE, THEN EAT IT which would preferably be understood as eating the apple and >TAKE BOOK AND APPLE, THEN READ IT which would mean reading the book, and I don’t think any system does that kind of analysis.

6

I see you also watch Tom Scott’s videos.

1 Like
7

Actually, Dialog does. It uses the computed likelihood of actions to resolve parser ambiguities. It also allows “it” to refer either to what the player just said (direct object) or what the narrator just said.

The manual gives a small example where this exchange could take place:

> OPEN SAFE
You open the safe, revealing a pearl necklace.

> TAKE IT
You take the pearl necklace.

But so could this one:

> OPEN SAFE
You open the safe, revealing a pearl necklace.

> CLOSE IT
You close the safe.

Likewise, if you “OPEN DOOR” in a room with many doors, it will only consider the currently closed ones, meaning that one word (open) resolves the ambiguity of another (door). I think many IF systems support that scenario.

More uniquely, Dialog can also infer the correct verb based on mentioned objects, but it’s the same basic mechanism at work. So if you have an NPC called Mrs North, then “NORTH, EAST” is ambiguous: It is either a command for the player character to go north, then east, or a command for Mrs North to go east. Dialog will resolve the issue based on the currently available exits. If both interpretations are equally likely, it will ask the player to clarify.

“PUT” is accepted as a synonym for “DROP”, so “PUT CATCHER IN RYE” could be a command to drop a book, or it could be a command to insert a baseball card into a rye-coloured envelope. Again, Dialog will use the world model to infer what action is most likely to be the intended one. So it is possible to conjure up contrived scenarios where an object name resolves the ambiguity of a verb, or vice versa.

But all of this works because we have a limited, well-defined world model, with only a few salient objects in scope at any time. I think the Winograd Schema Challenge is more concerned with resolving ambiguities in arbitrary text, that isn’t tied to a particular, known situation. It’s kind of the opposite problem, figuring out what the situation is based on clues in the text.

8 Likes
8

That’s cool! I remember that there were feature requests for Inform to do similar things, but the pronouns are set to a fixed value during the previous turn so it wasn’t feasible.