You can still run VDMX5 on your mac pro. There are still users out there on 2015 Macs running older versions of VDMX, but you will have to optimize for your setup. VDMX can push a machine to its limit, and with 8GB of Ram (and your OS most likely using up 6GB at any given time), you don’t have much to work with.
This was discussed recently (see my reply here) Computer match
As for your machine, there’s information about this in the VDMX docs. But I would run HAP files from your media bin off SSD or fast HDD external drives. No more than 2 layers at a time. (Try the simple mixer template and see how that works). Do not, pre-load media in the media bin (this will cache your media files into the system memory). Avoid heavy .vuo compositions.