I tried to create a similar site myself a while back, but the biggest challenge was that the margin of error was just too wide. I checked various benchmarks for different GPUs, but there were cases where the results differed, and there were just too many factors that influenced the results—such as the graphics card generation and whether quantization was enabled.
Very useful project for quick comparisons! One question: in the "recommended settings" where the quantisation are shown, e.g., "Q6_K", are Unsloth dynamic quantisations also considered or available to compare with?
I tried to create a similar site myself a while back, but the biggest challenge was that the margin of error was just too wide. I checked various benchmarks for different GPUs, but there were cases where the results differed, and there were just too many factors that influenced the results—such as the graphics card generation and whether quantization was enabled.
Very useful project for quick comparisons! One question: in the "recommended settings" where the quantisation are shown, e.g., "Q6_K", are Unsloth dynamic quantisations also considered or available to compare with?
[flagged]
It's incredibly useful! Takes all the guesswork out of provisioning servers for local agent deployments. Thanks for open-sourcing this.
[dead]
[flagged]