12 points | by targetbridge 3 hours ago
3 comments
If a prefix cache is good, why not keep it in primary mem?
The persistent KV cache is interesting; I’d love to see how much of the speedup remains when requests share less structure.
Why qwen 2.5 everywhere? Why not 3.5?
If a prefix cache is good, why not keep it in primary mem?
The persistent KV cache is interesting; I’d love to see how much of the speedup remains when requests share less structure.
Why qwen 2.5 everywhere? Why not 3.5?