Difference between in-line "instead" and "rule fails"?

“instead” and “stop the action” end the rule in failure, which stops the current rulebook. “rule fails” ends the rulebook in failure, which stops it and also sets its outcome value to failed.

No, there’s no good way to remember this.