I mislabeled normal mode, it should be called command mode. For the example you provided I don't see any problem, because when you execute set -o emacs it will simply first change vi mode back to insert mode, then change vi mode to emacs.
The way it works now is a bit weird, because when you run set -o emacs it will first change to emacs mode, and then I assume it will change mode parameter for vi mode to be an insert mode, without any visible change.