Multi-GPU automated partial offloading

[dumblob]

Describe the feature

Scientific computations often work with matrices or "not so conditionals/branch heavy" algorithms.

Using purely CPUs (which seems what VSL currently does and unfortunately seems to aim for) will "penalize" VSL compared to other hybrid frameworks whose API is made to accommodate for running the designated computations also on GPUs without user intervention.

I.e. in runtime it automatically detects available GPUs with the needed capabilities, quickly benchmarks them incl. the upload & download bus performance - think "bogomips" from Linux, then chooses the parts of the code which make sense to run there, and if there is no compile-time appropriate binary snippet to run on GPU it compiles the snippets by something like JIT and finally plumb everything to offload the chosen work to GPUs.

Use Case

1. To save time to researchers (yes, I often hear complains "I can not use more features/variables in my model as one run with 2500 of rows of input data already takes 2 hours on my modern PC and there are not enough money in the grant to pay for an AWS GPU cluster. So I can not do a full proper analysis but have to instead assume too many things instead of actually analysing them.").

2. To use ubiquitous HW to its max.

Proposed Solution

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

Version used

latest git master

Environment details (OS name and version, etc.)

Any (at least macOS, Windows, Linux).

[ulises-jeremias]

@dumblob hey! thanks for opening the issue! I just moved it to a discussion to make it easier to discuss about it

I agree with that, that is why VSL already has a module to support developing on GPU among some examples using GPU on the examples/ directory

the immediate planned GPU support is not for VSL but for VTL instead. I already started working on a VCL backend there to make it possible to use GPU. It is in a really early stage but will make it possible to define every computation using GPU 👌🏻 You can see an small example of VTL using VCL as backend here

I'll try to create a GitHub Project soon with the complete roadmap soon since VSL and VTL are closely related regarding plans for the future and development

[ulises-jeremias]

the real ML models development is in the scope of VTL, not in VSL. That is why I'm prioritizing adding full GPU support in there using vsl.vcl as backend and not adding GPU to VSL.ml or VSL.la

[ulises-jeremias]

anyways, I'll plan the addition of it on VSL.la in the future, and it won't be a braking change since I'll follow the same pattern we use on VTL for consistency. Something like

import vsl.la
import vsl.vcl

// get all devices if you want
devices := vcl.get_devices(.gpu)!
println('Devices: ${devices}')

// do not create platforms/devices/contexts/queues/...
// just get the device
mut device := vcl.get_default_device()!
defer {
	device.release() or { panic(err) }
}

matrix := la.matrix_deep2[f64]([[0.3, 0.4], [1.0, -0.9], [13.1, 0.67]])
cl := matrix.vcl(device: device)!

println(cl)

[dumblob]

anyways, I'll plan the addition of it on VSL.la in the future, and it won't be a braking change since I'll follow the same pattern we use on VTL for consistency.

OK. What I lack is though the help with "offloading in runtime" (e.g. the small benchmarking, hints, etc. so that the same code can run on your laptop CPUs as well as on the AWS GPU cluster with 0 code lines changed). Do you plan to add APIs for that too?

I mean, this should work also with FPGAs - offloading certain workloads to home-grown DSPs etc. would be great. Of course, the FPGA bitstream has to already exist, but all the plumbing of VSL/VTS to & from the FPGA shall be part of the VSL/VTL API.

I think the snippets you showed go in that direction - but I am really asking about the next steps and the overall planned scope 😉.

I'll try to create a GitHub Project soon with the complete roadmap soon since VSL and VTL are closely related regarding plans for the future and development

Perfect, that will make it more transparent what is (not) planned! Thanks a lot for all the hard work!