Skip to content

Upgrade xllamacpp to Vulkan on Windows & Linux #300

@iwr-redmond

Description

@iwr-redmond

XLlamaCPP now has Vulkan wheels for Windows and Linux. They still include the standard CPU optimizations that are present in the standard build. Note that while there is also a Vulkan OSX wheel, this is for Intel Macs that lack Metal acceleration.

Some code for matching available VRAM to GGUF layers is available here and may be of use when added to this GGUF layer reader code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions