Background Removal with macOS Vision API

1

HI everyone

I recently used this plug in installed through OBS software to do a very impressive job of live background removal on incoming camera feeds - was seriously impressed with its results and ran well on my M1 Max laptop

Would there be any way to integrate this as an EFX into VDMX ? as to work with the camera feeds in VDMX I routed the image through NDI which is fine, but be nice not to have to rely on OBS for the filter

Any thoughts or direction welcome

Thanks
Dan

2 Likes
2

Hi Dan,

Looks like there was already some internal testing with this, but we were seeing 22fps max with the video output, and if two videos ran the same filter, then it dropped to 11fps. It’s possible this will change with M3 & M4+ macs, but right now, it doesn’t make sense for the general body of users. M1, M2, and Intel macs.

1 Like
3

Also,
There’s already a standard CoreImage version of this effect that people can try out, listed under Stylize > Person Segmentation FX (macOS 13+). This outputs a mask that you can sort of get working with the Layer Mask FX.

2 Likes
4

Re: the internal vision API - I wouldn’t mind this, even at 22 fps! ;)

Re: the core image version - holy shit i wasn’t aware of that! Was that always there? For me it outputs a pretty good mask, but the mask is “shifted” somehow to a slightly different position on-screen and stretched. Any idea why that happens, is it broken? I can re-hack it back to cover the person quite okay with a bunch of geometry adjustments by hand, but its not very elegant and gets trouble the further away from the center of the screen.

5

The coreimage version is for OSX 14+, I’m not sure about the details (still running OSX 12 to ensure backward compatability, but I’ll be updating to OSX 14 within the next Month.

2 Likes
6

Thanks for sharing …helpful

7

Hi all, sorry to reply here sorry ! been busy with some shows, just starting to play with the Person segmentation on my copy of VDMX - still running OS 12 too, and get some functionality - low frame rates with slider set at default but if increase quality to 1.0 I can get 30fps - although with a shrunk feed

Very interested to see more and to try in OS 14 when take the plunge

Running M1 MacBook Pro btw

1 Like