Benchmark ?

[grigio]

Can you share how many seconds or it/s you do with your hardware (CPU/GPU/RAM) ?

[grigio]

ah ok, maybe #1

[mjkrakowski]

it is rather slow, q8 is the fastest i guess

sdcpptest.ipynb.txt

[mjkrakowski]

with 256x256px image size the q4_1 took about 8-9 minutes.

[leejet]

it is rather slow, q8 is the fastest i guess

Currently, it only supports running on the CPU. The CPU performance on Colab is not very strong, which results in slower processing. I'm currently working on optimizing its CPU performance and adding support for GPU acceleration.

[h3ndrik]

My old Skylake PC takes about 38s per step for the 8bit model. (OpenBLAS doesn't seem to make a difference)
the f32 model about 40s per step.

My old Laptop from 2016 needs 90s per step with the 8bit model.

[czkoko]

Sample Picture test on M1 16G, 5-bit, 512x768, 15 steps, euler a
The picture quality of q5_1 is quite good.

16-bit: memory < 3GB , 23 s/step
5-bit: memory < 2GB , 22.5 s/step

[juniofaathir]

@czkoko are you using sd 1.5 ggml base model? I think your result is just too good for just an base model

[czkoko]

@juniofaathir SD 1.5 base model can't generate such portrait, i use epicrealism

[juniofaathir]

@czkoko you can use that model?? I've been trying some civitai model and converting it, but it didn't work like at #8

[czkoko]

@juniofaathir There is no problem for me to use it. You can try the model I mentioned, or other training models, and filter merge models.

[mjkrakowski]

@czkoko
i was able to convert "reliberate" but not realisticvision5.1 with baked VAE. if the civit ai model has a vae free version, you should be able to convert any of them. all major models have a huggingface link you should prefer over civit ai.

[klosax]

Linking my tests using cuda acceleration (cublas) here #6 (comment)

[leejet]

@czkoko you can use that model?? I've been trying some civitai model and converting it, but it didn't work like at #8

@juniofaathir Most of the SD 1.x models from Civitai are working fine, except for a few that include control model weights. I'm currently researching how to adapt these models.

[ClashSAN]

@leejet hey, this implementation seems to use a very low amount of ram, lower and faster than using onnx f16 models. Thank you for your efforts!

It seems like the peak RAM usage stays at the minimum 1.4gb, when doing 256×384 images, using the current "q4_0" method!

Are you choosing a specific "recipe"?

like explained here: https://huggingface.co/blog/stable-diffusion-xl-coreml

The current composition of the model:

using these mixed quantization methods seems better than creating distilled models, they can be tailored and optimized for individual models..

[ClashSAN]

Here's something interesting: I almost got a full generation on a 2gb 32bit mobile phone, before running out of ram. If someone has a better 32bit arm device, please see if the generation is successful.

 ~/stable-diffusion.cpp $ ./sd -m anything-v3-1-ggml-model-q4_0.bin -W 64 -H 64 -p "frog" --steps 1
WARNING: linker: /data/data/com.termux/files/home/stable-diffusion.cpp/sd: unsupported flags DT_FLAGS_1=0x8000001
[INFO]  stable-diffusion.cpp:2191 - loading model from 'anything-v3-1-ggml-model-q4_0.bin'
[INFO]  stable-diffusion.cpp:2216 - ftype: q4_0
[INFO]  stable-diffusion.cpp:2261 - params ctx size =  1431.26 MB
[INFO]  stable-diffusion.cpp:2401 - loading model from 'anything-v3-1-ggml-model-q4_0.bin' completed, taking 21.55s
[INFO]  stable-diffusion.cpp:2482 - condition graph use 4.30MB of memory: static 1.37MB, dynamic = 2.93MB
[INFO]  stable-diffusion.cpp:2482 - condition graph use 4.30MB of memory: static 1.37MB, dynamic = 2.93MB
[INFO]  stable-diffusion.cpp:2824 - get_learned_condition completed, taking 15.42s
[INFO]  stable-diffusion.cpp:2832 - start sampling
[INFO]  stable-diffusion.cpp:2676 - step 1 sampling completed, taking 180.52s
[INFO]  stable-diffusion.cpp:2691 - diffusion graph use 11.46MB of memory: static 2.82MB, dynamic = 8.63MB
[INFO]  stable-diffusion.cpp:2837 - sampling completed, taking 180.62s
Killed
~/stable-diffusion.cpp $