Have you tried or considered using chroma keying instead? I’ve done this (same as your setup) both in studio and on stage, I had much better and easier results with chroma keying; making the full background green, and lighting the subject from front + sides and the background from below, then the keying worked quite okay. The subject was like you, very close to the wall/greenscreen.
I’m aware that will completely change the aesthetic of the room/stage/environment, it is a technical solution that not always working artistically. It was also for me a nightmare on festival or shared stages where I couldn’t control lighting 100%. I’m currently researching depth tracking and infrared solutions that would work regardless of lighting/stage.
(Can share more knowledge/experience, but you might have already rejected this route and specifically want to do it the way you describe, let me know!)