[FEA] Programmatic Device Selection

[will-hill]

It would be nice to programmatically select a device for cuML instead of using environment variables as mentioned in #4486

Something akin to XGBoost's ordinal with the device parameter:
https://xgboost.readthedocs.io/en/stable/parameter.html

Or maybe something similar CuPy:
https://docs.cupy.dev/en/stable/reference/generated/cupy.cuda.Device.html

[trivialfis]

Hi, as the maintainer of XGBoost, I suggest against integrating this feature into a ML library.

It's a maintenance burden for this global variable that can be changed by anyone at any time. And it's a surprise to users if cuML changes it internally without restoring it afterwards. (You are likely to see segfault)

We had it before the time of distributed frameworks mature. Given another chance, it wouldn't be here.

For new code, you can advise users to use cuda python or cupy to set it, in a way that aligns with its global variable semantic.

It's okay if cuML wants to provide a helper/wrapper, but just don't make it look like a local variable.

[csadorf]

@will-hill I tend to agree with @trivialfis here. Do you have a case as to why you would prefer this path over using cuda python or cupy?

[will-hill]

Enhancing the developer experience of cuML and the 'zero code change' philosophy leans towards not needing CuPy or some other API to select a device. I think @trivialfis suggestion of "a helper/wrapper, but just don't make it look like a local variable" is a good idea.

[csadorf]

I don't think that the use case outweighs the additional complexity required to support such a feature. There a standard methods for device selection that are already well supported. If anything, we could consider improving our documentation in that regard.