Optanes will be good for latency, but not so much for BW which seems to be your major bottleneck if I'm not mistaken?

yeah, the mobo upgrade is something I gotta do anyway, so I'll cover that up more or less, the optane is something I didn't thought about

In those systems you could probably leverage something like Nvidia SCADA or GDS directly.

Actually since they have direct GDS it should perform really well on professional gpus

nvmlib/ssd-gpu-dma and BaM (based on the same code base) are pretty cool as they allow you to initiate disk reads/writes directly from a CUDA kernel (so not only reading/writing directly to GPU memory but also allowing the GPU to initiate IO on its own). Sometimes called GPU-initiated I/O or accelerator-initiated I/O.