Yeah, FPGA+HBM works but it has no advantage over GPU+HBM. If you want to store weights in FPGA LUTs/SRAM for insane speed you're going to need a lot of FPGAs because each one has very little capacity.
Ok, then I may have misunderstood what you were saying. If the only thing we are interested is to store all the weights into the block RAM or LUTs then, yeah, that wouldn't be possible. I understood the OPs question a bit differently too.
