Nvidia Spark / ARM64

[daz-williams]

Having waited the best part of half a year to receive an Nvidia Spark, imagine my dismay when I realized that TGI will not work as the Spark uses ARM64 architecture.

Super sad that my only option is to use one of the alternatives such as Ollama or vLLM, instead of TGI which we've been happily using for the last 2 years and has been rock solid, serving us well.

-Daz

[daz-williams]

But I love TGI and will not give up on this awesome project!

Any plans in the pipeline for ARM64 compatibility?

[aniruddhaadak80]

From my point of view, ARM64 support is probably blocked more by the compiled inference stack and container build matrix than by TGI''s serving API itself. That is why users can see alternatives running on the same hardware while one otherwise mature server is missing.

A published support matrix that separates architecture blockers from model or kernel blockers would help people plan around this a lot better.