Background Removal with macOS Vision API

HI everyone

I recently used this plug in installed through OBS software to do a very impressive job of live background removal on incoming camera feeds - was seriously impressed with its results and ran well on my M1 Max laptop

Would there be any way to integrate this as an EFX into VDMX ? as to work with the camera feeds in VDMX I routed the image through NDI which is fine, but be nice not to have to rely on OBS for the filter

Any thoughts or direction welcome

Thanks
Dan

2 Likes

Hi Dan,

Looks like there was already some internal testing with this, but we were seeing 22fps max with the video output, and if two videos ran the same filter, then it dropped to 11fps. It’s possible this will change with M3 & M4+ macs, but right now, it doesn’t make sense for the general body of users. M1, M2, and Intel macs.

1 Like

Also,
There’s already a standard CoreImage version of this effect that people can try out, listed under Stylize > Person Segmentation FX (macOS 13+). This outputs a mask that you can sort of get working with the Layer Mask FX.

1 Like

Re: the internal vision API - I wouldn’t mind this, even at 22 fps! ;)

Re: the core image version - holy shit i wasn’t aware of that! Was that always there? For me it outputs a pretty good mask, but the mask is “shifted” somehow to a slightly different position on-screen and stretched. Any idea why that happens, is it broken? I can re-hack it back to cover the person quite okay with a bunch of geometry adjustments by hand, but its not very elegant and gets trouble the further away from the center of the screen.

The coreimage version is for OSX 14+, I’m not sure about the details (still running OSX 12 to ensure backward compatability, but I’ll be updating to OSX 14 within the next Month.

1 Like