You are right on your last thought; assigning the layer eject to the “off” position of your toggle. You can do that using the UI Inspector to specifically have the OSC command trigger when “off”
Also, I would suggest trying to use the Control Surface plugin in VDMX as an intermediary. It can seem redundant, but I think it can give you additional control on certain behaviors or more ease when having a button controlling multiple things. Heck, you can even import your TouchOSC layout so you don’t have to recreate everything. 