I’ve thought about making my own “engine” as well. I wrote out the syntax and basic commands that I would need. After all was said and done, I ended up with kind of a mixture of the 4 main Twine story formats. However, the main thing I wanted was to have something as bare bones as possible.
Snowman is a framework that requires you to make your own language on top of it to get anything reasonable out of it. However, it’s got everything you need in its base functionality. Meaning, you can launch it in Twine, but use your own commands to do whatever you want. The documentation is sparse, but so is the base functionality… which is a boon, not a knock against it. It honestly looks like a solid framework and seems like a great thing to read up on to determine if you’ve got your own base functionality covered.
Chapbook is like Snowman (kind of), but with an existing thin language on top of it. It’s straight forward to extend with your own custom commands, but it has some quirks. I’m currently learning it, but I have high hopes. Looks very promising. I want to be able to extend a language more than create one from scratch. Chapbook encourages authors to extend it and works with JavaScript very, very well. You can check out some discussion around figuring out how Chapbook works here, but the official documentation is limited.
The other two are Harlowe and SugarCube. I sense you won’t want to work with them (even though you can do whatever you want in both – see my journey with Harlowe) for the same reasons I’ve shied away. I just like the idea of coding in JavaScript as much as possible. I don’t like overriding someone else’s default CSS if possible. I don’t like the idea of a bloated framework where I’ll never use 95% of what is there. I don’t like having to work around someone else’s interpretation of how things should work.
It honestly sounds like you might want to visit Snowman. It’s got all the base functionality covered to be fully compatible with Twine, yet it needs a language of custom functions to make it feasible. It’s like C++ and you’re building your own library on top of it. That’s where I would start. Snowman, baby. Snowman. 
You’ll have to decide if you enjoy the process of making a great game or making a great framework for a game. Both are worthy endeavours and a lot of fun.
Edit: I have no idea why this post replied to Jim.