I think the following covers your intended cases, and it’s not difficult to add in the functionality to indicate where something was last seen.
Unavailable Things
"Unavailable Things"
[Doesn't use Remembering because you said your actual goal was simpler.]
Include Epistemology by Eric Eve. [for "seen"]
Place is a room.
Other Place is east of Place.
A red ball is in Place.
An open openable container called a box is in Place.
A blue ball is in Other Place.
[An alternative to scope testing via direct object tree inspection]
To decide whether (X - thing) has line of sight to (Y - thing):
let CA be the common ancestor of X with Y;
if CA is nothing, decide no;
let checked be an object;
now checked is X;
while checked is not CA:
now checked is the holder of checked;
if checked is a closed opaque container, decide no;
now checked is Y;
while checked is not CA:
now checked is the holder of checked;
if checked is a closed opaque container, decide no;
decide yes.
To decide whether (X - thing) does not have line of sight to (Y - thing):
if X has line of sight to Y:
decide no;
otherwise:
decide yes.
Definition: a thing is unavailable if it is seen and the player does not have line of sight to it.
After deciding the scope of the player:
repeat with X running through unavailable things:
place X in scope.
[Uses before rules to intervene ahead of visibility checks]
Before doing something when the noun is unavailable:
say "[The noun] [are] nowhere to be seen." instead.
Before doing something when the second noun is unavailable:
say "[The second noun] [are] nowhere to be seen." instead.
test me with "x red ball / x blue ball / e / x red ball / x blue ball / w / put red ball in box / x red ball / close box / get red ball / open box / get ball / e / put red ball in box".
which yields:
Place
You can see a red ball and a box (empty) here.
>test me
(Testing.)
>[1] x red ball
You see nothing special about the red ball.
>[2] x blue ball
You can't see any such thing.
>[3] e
Other Place
You can see a blue ball here.
>[4] x red ball
The red ball is nowhere to be seen.
>[5] x blue ball
You see nothing special about the blue ball.
>[6] w
Place
You can see a red ball and a box (empty) here.
>[7] put red ball in box
(first taking the red ball)
You put the red ball into the box.
>[8] x red ball
You see nothing special about the red ball.
>[9] close box
You close the box.
>[10] get red ball
The red ball is nowhere to be seen.
>[11] open box
You open the box, revealing a red ball.
>[12] get ball
(the red ball)
Taken.
>[13] e
Other Place
You can see a blue ball here.
>[14] put red ball in box
The box is nowhere to be seen.