Req: Dead-man's switch (reactive alert and response to a VDMX crash)

VDMX
1

Hey, could the brilliant minds of both Vidvox and perhaps forum geniuses help me think of:

A VDMX dead-man’s switch (in-software or in-OS) to automatically swap over to backup system in case of crashes, automatically

VDMX is crashing a lot. This is a problem for shows that really should not crash. It would be great if it could crash less, but I know from multiple sources that the crashes are more the responsibility of Apple than VDMX (an insider on the AV foundation team confirms this), so either way this is not fixed quickly or at all.

So now I’m touring with two laptops and a Blackmagic switcher to potentially quickly switch. I can manually switch over when I notice a crash, and reload the crashde computer. Problem is, I’m touring solo, and in some parts I’m so focused on performance it can take me up to 90 seconds before I can review crash status, and of course that is when the crashes happens.

Is there a way, that we could get a trigger AT THE MOMENT WHEN VDMX CRASHES. At that moment, I want to switch immediatly to backup system, and also possibly get a notification somehow, preferably a large blinking light or whatever, something MIDI or audio or anything so I can review status as soon as possible.

Optimally, the crash alert would also instantly relaunch VDMX on the crashed computer, so in case the BACKUP goes down a minute later, which it inevitable does, the switch can happen back again, and the systems keeps reseting themselves. This would minimise visual downtime.

Of course a more stable VDMX would be preferable but don’t think that is gonna happen.

Maybe this could be doable with a “ping” kind of MIDI / OSC system, where every second a MIDI / OSC routine pings the running system, and if does not respond within a second, the MIDI system tells the switcher to swap and restart. Or maybe VDMX itself could alert by sending out a MIDI signal upon crash?

Anyone else dealing with this and has found a workable solution? Except hiring a person or two…

2

Hi GMM,

I’m not sure about a crash alert, but VDMX has been very stable for me over the years. Needless to say, for larger shows, I typically run a backup computer through an HDMI mixer (with the same project, and show content). If VDMX were to crash on one, I simply use the cross fader on my Roland to move over to the next computer. I can keep both computers in sync using OSC. The Roland HDMI mixer ensures that there is a constant feed to the projector or displays. When I’m at larger venues with built in switcher scalers, I’ll use their capture image option so I can always freeze on a logo or cut to black if need be.

So a few thoughts here.

1: Which version OSX are you running and which version of VDMX?

2: What is causing your crashes? (if you know)

3: If you use OSC or another network protocol to keep two computers in sync you may be able to redirect that through something like OSCulator or make a custom script that says (when this signal stops), switch over to other computer.

The Roland HDMI mixers can be controlled through USB and Midi, so it seems do-able to make an auto switcher. Unless it is an apple script or something similar, I’m not sure how you’d get a dying app to send a switch command over network, but if it was always “talking” (over OSC) to the other computer and then that signal drops off, it might be possibile to script the other computer to switch on?

3

HI @ProjectileObjects thanks for following up, appreciate your work and presence!

VDMX has not been stable for me at all over the years, on multiple computers, your company’s bug reports on my name will tell you that and quite likely scare you away as hit has scared everyone else away from dealing with this :)

at one point ( 2019) VDMX just stopped responding to my crash reports probably because they just gave up, so I’ve been on my own since then. I understand its now a transition to Apple silicon, my estimate based on company presence and support over the years, it is less than probable that this software will be awesome up ahead ;)

1: Mojave, and right now on latest beta x.x.x.x.10 but the crashes has been endless since 2017, on multiple systems, with multiple projects, it crashes, crashes, crashes

2: nobody knows, but through fans and supporters who has observed my live crashes, and i have enough of a crowd that some of them understand what’s going on, eventually then developers at Apple have looked at the crash logs, and agree the problem is with AV foundation, not the app itself. i reached out to VDMX with this information, as always, no response. weird that I get further with Apple than VDMX.

3: both computers are in sync, using NDI cams they both have the same inputs, they both get the same MIDI, they are always presenting the same output. They both crash randomly at random potins. My only challenge is, how to know when to swap over to the backup automatically, and how do it get both back up to speed once a crash happen.

I guess both your and mine setup are kind of similar, regardless of hardware. I’m on a BM Atem Mini, i can program that remotely with MIDI or OSC. i run a duplicate setup, external cams are with NDI, the visual output is the same, so when the primary setup goes down, i can swap over to the backup and it is exactly the same. I could even add a third or fourth computer. what i need to know and handle, is WHEN the primary goes down.

VDMX and/or AV foundation crashes once or twice in every of my live shows, so it’s a cost I need to carry, the software is buggy or built on buggy foundations, I just need to build a solution to handle that. I’ll transition elsewhere as soon as I can but for now I have decenniums of content and setups developed for and dependent on VDMX so I kind of just need to find solutions.

Don’t have a lot of hope for Apple silicon but curious to observe, if throwing money at a new platform would help I am okay with that.

4

What did Apple say the issue with AVFoundation is?

Apple transitioned to AVFoundation and discontinued support for all their “legacy” codecs after 10.12.6.
Maybe the reason VDMX has always been stable for me, is that I keep the computer with the version of OSX that it comes with and as soon as I have a stable version of VDMX, I keep it. I can install newer versions, but I change the name so I can always roll back to the “stable” version in my apps.

The thing I’ll note about older projects, video codecs, and newer versions of VDMX as well as MacOS, is many times there are FX that carry forward in upgrades that are not stable in newer versions.

I’m currently running OSX 12.0.1 on a 2021 M1 Max Macbook Pro. Version 10 is stable and all my FX work, some ISF shaders don’t, but most do. I also have a special beta build that is harnessing this computers architecture and there are a number of old .qtz FX that don’t work and will either need to be updated or removed entirely. If I was to install a new version of VDMX (that does not include those broken .qtz files), VDMX will keep the older presets, settings, and FX (rather than completely remove everything and replace). That’s so you don’t lose custom FX and settings if you’ve created them.

Looking at a years load of user crashes there were two main issues that I’ve seen. There was an Elgato plugin that would crash VDMX at startup (until it was updated) and then there were Vuo users experiencing crashes due to their custom Vuo compositions (that for the most part don’t work properly in newer version of Vuo and OSX).

As you said, AVFoundation is what Apple blames, which is a bummer, because that’s the backend of Madmapper, Modul8, and Resolume if you were to run visuals off a Mac.

I’ll also note, that Bug reports submitted directly to apple do not include user data, the projector, or the system specs, so we can’t get back to users when someone submits through that route.

If a bug comes in through the built in VDMX bug reporter, the dev team more often than not, loads up the same OSX, project, and version of VDMX on a computer and sees if they can recreate the problem.

If they can, it gets fixed in the next version of VDMX, but they don’t always reply back to people (which is why they asked me to help with this at the beginning of 2021).

If you’d like to chat on here or share a bug report with me, at least I’ll be able to look at it any communicate back and forth with you.

Let me know if you’re interested. I’m here to help.

At 51 minutes into this video, is this when VDMX crashes?

Also, during your performance are you using images .tiff, .png, jpeg, etc? Or is everything Hap or Hap Alpha?

Are you NDI cameras pulling in or sending at RGB or is that turned off?

Are you using OBS for your chat and the rest of the overlays? Is that on the same computer as VDMX or separate?

Thanks!

5

(Just noting that we’re currently following up the crash issue in emails - if we find a solution I’ll update the thread)

6

I just use QuickTime player. Have it in ready with a long clip that I recorded the output from
VDMX. Make sure you select the loop option.

When you need it. Enter full screen mode and output to your projector “screen”.

Thankfully it doesn’t take long for VDMX to start up again.

Also if your experience crashes.
Rollback to a more stable version or figure out
Why it’s crashing. Check the logs

2 Likes
7

That’s a good solution too! I’m currently mostly having a custom “blue screen of death” as desktop background, which then instantly apologizes to the audience that the visuals has crashed, this seems to work well with the audience, they laugh and cheer when it happens. But this solution only fits some kind of performances, and also can’t happen too often. For other shows a seamless replacement like yours is more approriate. Though the blue screen of death is highly visible and jarring, so I’m instantly alert to the problem :slight_smile:

1 Like
8

It is possible to write a dameon that would detect the VDMX exiting and then you would have it execute a script (bash / AppleScript) from there.

But guaranteed it won’t be as fast.
Sometime when apps crash you’ll notice the crash reporter may come up right away and sometimes not.

9

Thanks. Indeed crash reporter sometimes is very slow reporting, and a few times there has been crashes that leaves no trace. I finally had some time to test for the problem this weekend, with help from Vidvox support, and over a 12 hour session, got 8 crashes reported, but it crashed twice more seemingly didn’t leave any report at all.

But in the same test I also now got the Universal build to test on my Mac Mini M1, which then did NOT crash at all during the same time span.

Very promising. I can’t know if that is software (the M1 runs Big Sur, the Intel systems run Catalina) or hardware (M1 vs Intel) but later in December I will have time to test a Big Sur upgrade on my Intel hardware, and early next year update to Macbook Pro M1 Max. Based on the Mac Mini tests it is likely that the crashes won’t happen on that.

10

Yes, and to be clear GMM you are using a good number of layers and still images of various resolutions that could have been asking too much for your previous computers. I’m also impressed by the Apple native hardware and how well it performs with the latest version of VDMX (which does have its own bugs! :slight_smile: )

11

just on the option of an actual deadman’s switch, this is how I would solve it:

  1. send out an OSC heartbeat from VDMX to a program managing the switch. (I would use MaxMsp, but PD or any Program that can handle UDP will be fine)
  2. write a script that checks for the heartbeat and whenever it is not receiving a message for one or two beats, send a command to atemOSC to make the switch
  3. execute an AppleScript to restart VDMX on the crashed device.

This would be a reliable solution. More reliable would of course be troubleshooting your VDMX setup to make it not crash :)

edit: if you need help with this. hit me up. maybe I can write up something simple for you to try.

1 Like