Ignoring the fact that this is clearly not written by a human, it's untrustworthy and the claims are dubious at best.
1. The comparison of vLLM to vLLM.cpp never shows more improvement than a handful of tokens per second. That's less than 0.05x improvement on every run, and the gap doesn't grow as concurrency increases. The comparison doesn't show any real net improvement, let alone justify the project.
2. The depth anything comparison isn't apples to apples. Of course a q8_0 quant is faster than f32. It's 4x less data to chew on.
3. This point is silly, it again fights against the argument that writing your own c++ engines are worth it. It's a bug, just fix it:
> The reason it is faster has nothing to do with writing better matmul kernels than PyTorch. Two positional embeddings, the DPT head’s UV embedding and the backbone’s bicubic position embedding, were being recomputed on every forward pass with single-threaded scalar sin, cos and bicubic loops, even though they depend only on the input geometry and are identical every call.
They argue against their own point again just after:
> For a biometric pipeline, matching the reference exactly matters more than being faster than it.
Okay, then don't rewrite it! It's not faster anyway!
4. If the argument is that the venv is large, then rewriting it in C++ doesn't seem like the answer, it seems like a lot of work and maintenance to avoid having to cull unreachable files in your venv. In a past life I maintained a simple denylist for files in node_modules. The low hanging fruit is plentiful and generally very safe.
Getting any performance improvement over vLLM is just a bonus although the RAM reductions are very nice. The main point is that we find deploying Python based projects to be a pain. I can't imagine how you know what files are unreachable in a dynamic language, but in any case that doesn't begin to solve issues like PyTorch requiring a C++ compiler at runtime to generate optimised CPU code. This isn't exactly embedded device friendly.
Having said that if you know of some magic bullet to fix the problems with the Python ecosystem (that vLLM and Pytorch are apparently unaware of) then I will be happy to try it because our CI is constantly blocked by Python related issues.
> Deploying a Python inference stack means resolving a dependency tree at install time, on the target machine, against whatever CUDA and glibc that machine has. Deploying a ggml port means copying a shared library and a GGUF file.
Take a look at the code. It is a conglomeration of python, rust, go, cpp etc. I was expecting lean C or C++ but all I see is a kitchen sink of code, the AI generated kind. No problem with that, but it is inconsistent with the messaging. I appreciate the intention behind this, but dont like the execution. My prediction is that this will enter the graveyard of ai generated git repos.
For open-source code to get adoption and support, the theory of its construction needs to be understood by humans because at the end of the day, some human will be responsible for its operation. I would call this the Terrence Tao principle: Output of AI is only as useful as how well it can be digested by humans.
I can only assume you are talking about LocalAI (not vllm.cpp) which is modular and contains adapters for many different AI engines written in various languages. LocalAI's core is written in pure Go and you choose what engines to install at runtime.
vLLM.cpp's runtime is pure C++, but AOT Triton compilation is done at build time in Python.
> Should have started with writing your own blog posts.
While the page looks vibe-coded[1], the content itself does not have any AI tells. What are the tells you are seeing?
[1] Too many sites I find on HN frontpage these days slow my PC to a crawl. I assume they are all using the same autogenerated HTML, Javascrip and CSS to make animated backgrounds :-( On this specific site scrolling is laggy.
Just from the subtitle, you can already tell it's likely AI-generated. Maybe you're lucky enough not to be exposed to claude/codex output your whole working day.
I stopped reading almost immediately. The stylistic choices in the writing just felt like LLM to me. Examples:
"depth estimation that beats PyTorch on CPU in half the memory" — "…beats X in Y…"
"Most LocalAI backends wrap somebody
else’s engine, and that is the right default." — "…and that is the right"
"MLX and the rest are maintained by people who are better at those models than we are" — "better at those models than we are" — it's this thing that LLMs do where they are kind of weirdly confident but overly deferential
"This post is about what those ports buy" — "…buy" used in this context
"Same model, 1.31x the speed" — "Same X, something Y" — it's this overconfident yet deferential writing style
The further I read, the more tells there are. I find it incredibly tiring to read LLM generated prose and I'm not sure why. Is it because I'm aware it's not human written and have an unconscious bias? Or is it because the style is just full-on, "Not X but Y. Those performance gains are bought, not earned. This stops, that starts. Read on, or don't, that's the follow-up"
For me, it's that AI writing is always trying to be clever for every single point it makes (and constantly uses the same language patterns when doing so).
Its like listening to an insufferable clever dick, who is not as bright as they think they are. You would also find it incredibly irritating if a human talked like that
I've always used em-dash. I got familiar with the hotkeys when I started using Mac OS X in 2002. Option+Hyphen for en-dash, Option+Shift+Hyphen for em-dash. It's unfortunate that it's an LLM tell. I'm glad the LLMs haven't subsumed proper ellipses yet…
This has many of the writing tropes of Claude. I'd bet money that much of it is ai-generated, if not all of it. This paragraph alone feels almost entirely Claude's self-congratulatory style of writing:
> We are ahead at all six points, and five of those six are ties. Our run-to-run noise band is 0.5%, and concurrency 2 through 32 land between 0.7% and 1.7%, so the honest reading is that only the single-stream case (4.5%) is clearly outside noise. Output is token-for-token identical to vLLM at every point on that curve. Peak host memory is 24.88 GiB against 28.18 GiB.
I found the post insightful and interesting. I'm not sure it was written with AI assistance, but even if it was, I don't see that as a reason to dismiss it. For what it's worth, I spend hours everyday reading AI output and summaries.
I get the motives but it's tiring. I myself sometimes check my own (unpublished) writing or have friends preview it and the same feedback comes out, it's all hand written.
Yeah, weirdly enough, I was testing gptzero on some of my own writings, it suggested they're 100% written by a human, tried this article, it says 100% written by AI. I would prefer for any blog post to be human sourced as much and as often as possible, but there's no true way to enforce or incentivize people to do this sadly.
People on HN have accused me of sounding like an AI one time or another, English is my second language and sometimes my ADD goofs my writing into something that sounds like gibberish even though I fully understand what I wrote others might not, so yeah.
I've had this debate before on HN. The excuses are generally "well it can write better code than most developers, but an LLM can't write better prose than most people" (I strongly disagree with this) and, what I think is at the heart of the matter, "text is for the reader to read directly, code is hidden." Or in other words, "as long as I can't tell it's AI, it's fine."
The thing is... although the writing is unmistakably full of LLMisms, I can't fault the `author` for having produced a slop readme.
The content earns its keep, it only grates because of the robotic personality. We need another word than "slop" for this.
I went through the post because of your comment but it really doesn't look like AI slop. Can you please share why you feel like its slop and not written by a human? I can also say "should have started writing your own comments" to you and its unfalsifiable. Blanket accusations with no proof is not a good move really.
"This post is about what those ports buy, measured, and what they cost."
if you've done any amount of optimizaation or hill-climbing work (kernel optimization, autoresearch type shit), you would know that these models LOVE the terms "buy" and "cost" in this sense. they use them nearly constantly (along with "budget", and even "credit" (even when working with RL/credit assignment!))
What do you mean you could go on and on? Why do you think those sentences are AI written.
And okay, your second argument is that I just don't know slop because I am not exposed to it? But you don't know anything about me or what I am exposed.
You're just making random claims and stating they are correct without any evidence or arguments.
What kind of evidence do you expect beyond "random claims" here?
This post is incredibly obviously AI generated, to the extent that I doubt a human author edited it at all. Not "written with AI assistance" but full on "give Claude some bullets and hit publish." It contains tons of tropes that show up in all AI writing and which people are highlighting here.
I don’t think it would be surprising that people want to write their own kernels.
A big problem with the existing engines like llama or sd is that they don’t support optimal graph compilation. Usually this means about a real 2 or 3x multiplier loss relative to optimal. Cuda graphs do okay but they still leave a lot on the floor
It’s usually worth it to optimize in that context if you are willing to peer into the mechanics.
Of course that’s expensive. You need to know how to appropriately pipeline and merge your kernels.
I did took a native c++ approach when writing a relational transformers engine (RelativeDB). My journey was pytorch -> c++ -> Triton (lang). While C++ was more performant than Triton, I couldn't afford to optimize on every gpu. I just accepted the ~15% throughput loss for my cloud service, which honestly wasn't bad for the amount of flexibility I got out of it.
But the cpp port of vllm looks great, that'd be great if you'll maintain that. I hit the same limitations with vllm.
Could this vllm port be faster to install? Im starting gpu machine multiple times a day and it takes 5 minutes to set vllm up. If Inise this port that time is minimized?
This is on my list to evaluate, I absolutely do not want to download 9gb of supply chain risk into prod every time we upgrade, when I can compile 70mb of binary. We run vLLM in a container with hardware passthrough for gitops, having the entire environment in a single container would drastically improve things and move local LLM into a pattern that more closely follows our other CI/CD systems, rather than this hulking behemoth snowflake deployment.
Not sure what's worse, if the AI slop content, or the AI code slop that makes the site consume 100% of a single CPU core (and if you enable GPU rendering on the browser, CPU usage is still significantly above zero.)
People are critiquing whether or not AI wrote the article.
Well, it's clearly not 1-shot / low effort.
(can we agree there's a difference between low effort 1-shotting a prompt and those who use LLMs interactively?)
The one reason why I can't take a mega hardline stance against AI for writing: Not everyone speaks English as a first language.
People with awful English speaking and writing skills are using LLMs to contribute content in English. This is a good thing. They used to try their best and suffix every post with "sorry for my english".
The author's name is Ettore Di Giacinto - I put 2 and 2 together on this one.
I give a pass when non-native English speakers are just using it to write better English and keep up with the rest of us. It wasn't super obvious anyway, it wasn't interruptive.
The content is beyond that, and I more than welcome stuff like this here.
Ignoring the fact that this is clearly not written by a human, it's untrustworthy and the claims are dubious at best.
1. The comparison of vLLM to vLLM.cpp never shows more improvement than a handful of tokens per second. That's less than 0.05x improvement on every run, and the gap doesn't grow as concurrency increases. The comparison doesn't show any real net improvement, let alone justify the project.
2. The depth anything comparison isn't apples to apples. Of course a q8_0 quant is faster than f32. It's 4x less data to chew on.
3. This point is silly, it again fights against the argument that writing your own c++ engines are worth it. It's a bug, just fix it:
> The reason it is faster has nothing to do with writing better matmul kernels than PyTorch. Two positional embeddings, the DPT head’s UV embedding and the backbone’s bicubic position embedding, were being recomputed on every forward pass with single-threaded scalar sin, cos and bicubic loops, even though they depend only on the input geometry and are identical every call.
They argue against their own point again just after:
> For a biometric pipeline, matching the reference exactly matters more than being faster than it.
Okay, then don't rewrite it! It's not faster anyway!
4. If the argument is that the venv is large, then rewriting it in C++ doesn't seem like the answer, it seems like a lot of work and maintenance to avoid having to cull unreachable files in your venv. In a past life I maintained a simple denylist for files in node_modules. The low hanging fruit is plentiful and generally very safe.
Getting any performance improvement over vLLM is just a bonus although the RAM reductions are very nice. The main point is that we find deploying Python based projects to be a pain. I can't imagine how you know what files are unreachable in a dynamic language, but in any case that doesn't begin to solve issues like PyTorch requiring a C++ compiler at runtime to generate optimised CPU code. This isn't exactly embedded device friendly.
Having said that if you know of some magic bullet to fix the problems with the Python ecosystem (that vLLM and Pytorch are apparently unaware of) then I will be happy to try it because our CI is constantly blocked by Python related issues.
Then the article should have talked about those things and not pointed at nonsense.
It's literally the third paragraph:
> Deploying a Python inference stack means resolving a dependency tree at install time, on the target machine, against whatever CUDA and glibc that machine has. Deploying a ggml port means copying a shared library and a GGUF file.
[dead]
Take a look at the code. It is a conglomeration of python, rust, go, cpp etc. I was expecting lean C or C++ but all I see is a kitchen sink of code, the AI generated kind. No problem with that, but it is inconsistent with the messaging. I appreciate the intention behind this, but dont like the execution. My prediction is that this will enter the graveyard of ai generated git repos.
For open-source code to get adoption and support, the theory of its construction needs to be understood by humans because at the end of the day, some human will be responsible for its operation. I would call this the Terrence Tao principle: Output of AI is only as useful as how well it can be digested by humans.
I can only assume you are talking about LocalAI (not vllm.cpp) which is modular and contains adapters for many different AI engines written in various languages. LocalAI's core is written in pure Go and you choose what engines to install at runtime.
vLLM.cpp's runtime is pure C++, but AOT Triton compilation is done at build time in Python.
Should have started with writing your own blog posts.
> Should have started with writing your own blog posts.
While the page looks vibe-coded[1], the content itself does not have any AI tells. What are the tells you are seeing?
[1] Too many sites I find on HN frontpage these days slow my PC to a crawl. I assume they are all using the same autogenerated HTML, Javascrip and CSS to make animated backgrounds :-( On this specific site scrolling is laggy.
Just from the subtitle, you can already tell it's likely AI-generated. Maybe you're lucky enough not to be exposed to claude/codex output your whole working day.
I stopped reading almost immediately. The stylistic choices in the writing just felt like LLM to me. Examples:
"depth estimation that beats PyTorch on CPU in half the memory" — "…beats X in Y…"
"Most LocalAI backends wrap somebody else’s engine, and that is the right default." — "…and that is the right"
"MLX and the rest are maintained by people who are better at those models than we are" — "better at those models than we are" — it's this thing that LLMs do where they are kind of weirdly confident but overly deferential
"This post is about what those ports buy" — "…buy" used in this context
"Same model, 1.31x the speed" — "Same X, something Y" — it's this overconfident yet deferential writing style
The further I read, the more tells there are. I find it incredibly tiring to read LLM generated prose and I'm not sure why. Is it because I'm aware it's not human written and have an unconscious bias? Or is it because the style is just full-on, "Not X but Y. Those performance gains are bought, not earned. This stops, that starts. Read on, or don't, that's the follow-up"
For me, it's that AI writing is always trying to be clever for every single point it makes (and constantly uses the same language patterns when doing so).
Its like listening to an insufferable clever dick, who is not as bright as they think they are. You would also find it incredibly irritating if a human talked like that
"Parity is what makes the replacement a drop-in rather than a migration."
It always reads like a 14 year old arguing.
Also, “honest reading” — without any context explaining why one would contemplate a dishonest reading.
In "biometrics that match insightface bit for bit" - the "bit for bit" thing is also something I have encountered more than once.
Now that you point it out, there are quite a few tells, still not as many as most of the slop that gets posted here.
I think it's because of the laggy scrolling that I didn't read the whole thing anyway, just the first few screens.
[dead]
Why are you using an em-dash though?
I've always used em-dash. I got familiar with the hotkeys when I started using Mac OS X in 2002. Option+Hyphen for en-dash, Option+Shift+Hyphen for em-dash. It's unfortunate that it's an LLM tell. I'm glad the LLMs haven't subsumed proper ellipses yet…
Opened this page in Firefox and my CPU fan started spinning like crazy. All this for a background that just renders the whole page unreadable.
Stop. Seek help.
This has many of the writing tropes of Claude. I'd bet money that much of it is ai-generated, if not all of it. This paragraph alone feels almost entirely Claude's self-congratulatory style of writing:
> We are ahead at all six points, and five of those six are ties. Our run-to-run noise band is 0.5%, and concurrency 2 through 32 land between 0.7% and 1.7%, so the honest reading is that only the single-stream case (4.5%) is clearly outside noise. Output is token-for-token identical to vLLM at every point on that curve. Peak host memory is 24.88 GiB against 28.18 GiB.
Use lynx, links or netsurf. Or vibe code your noscript/basic HTML web engine?
[flagged]
I found the post insightful and interesting. I'm not sure it was written with AI assistance, but even if it was, I don't see that as a reason to dismiss it. For what it's worth, I spend hours everyday reading AI output and summaries.
This witch hunting is getting tiring.
I get the motives but it's tiring. I myself sometimes check my own (unpublished) writing or have friends preview it and the same feedback comes out, it's all hand written.
Yeah, weirdly enough, I was testing gptzero on some of my own writings, it suggested they're 100% written by a human, tried this article, it says 100% written by AI. I would prefer for any blog post to be human sourced as much and as often as possible, but there's no true way to enforce or incentivize people to do this sadly.
People on HN have accused me of sounding like an AI one time or another, English is my second language and sometimes my ADD goofs my writing into something that sounds like gibberish even though I fully understand what I wrote others might not, so yeah.
Same could be said for all that talk about having Claude do their work.
I've had this debate before on HN. The excuses are generally "well it can write better code than most developers, but an LLM can't write better prose than most people" (I strongly disagree with this) and, what I think is at the heart of the matter, "text is for the reader to read directly, code is hidden." Or in other words, "as long as I can't tell it's AI, it's fine."
I would rather read faulty English, succinct sentences and getting to the point, than the generic filler LLMs produce.
I would also code review code that people actually put some effort learning on how to write it, even if it had one bug or two.
Came here to say the same. Really tiring to read these slop-infested posts, where everything has the “right shape”.
The thing is... although the writing is unmistakably full of LLMisms, I can't fault the `author` for having produced a slop readme. The content earns its keep, it only grates because of the robotic personality. We need another word than "slop" for this.
"blland", "llame",... ?
Vapid
While I am tired to see slop-infested pull requests being celebrated.
I went through the post because of your comment but it really doesn't look like AI slop. Can you please share why you feel like its slop and not written by a human? I can also say "should have started writing your own comments" to you and its unfalsifiable. Blanket accusations with no proof is not a good move really.
"This post is about what those ports buy, measured, and what they cost."
if you've done any amount of optimizaation or hill-climbing work (kernel optimization, autoresearch type shit), you would know that these models LOVE the terms "buy" and "cost" in this sense. they use them nearly constantly (along with "budget", and even "credit" (even when working with RL/credit assignment!))
The post is full of signs. Here's only a couple of examples:
> The method, the measurements, and what it costs us.
> That is the general shape of these wins.
> Parity is the gate, speed is the follow-up
I could go on and on, but you probably get the point. If you don't find anything funny with the above, you might have not been enough-exposed to slop.
What do you mean you could go on and on? Why do you think those sentences are AI written.
And okay, your second argument is that I just don't know slop because I am not exposed to it? But you don't know anything about me or what I am exposed.
You're just making random claims and stating they are correct without any evidence or arguments.
What kind of evidence do you expect beyond "random claims" here?
This post is incredibly obviously AI generated, to the extent that I doubt a human author edited it at all. Not "written with AI assistance" but full on "give Claude some bullets and hit publish." It contains tons of tropes that show up in all AI writing and which people are highlighting here.
What would convince you of that?
https://www.pangram.com/history/125358dc-9f80-4fe2-8540-85a9...
They follow the tropes I get when I ask AI to do docs or summaries. Very Opus style, this one.
[dead]
It's always hyping up something and throwing punch lines in every sentence. Normies love this shit.
Yes, it’s basically business-as-usual but on speed.
its so toxic when people show this kind of confidence in their ability to tell when something is ai or not.
It's so toxic when people prevent others from avoiding wasting their time reading blatant fluff that no one spent their own time writing.
I don’t think it would be surprising that people want to write their own kernels.
A big problem with the existing engines like llama or sd is that they don’t support optimal graph compilation. Usually this means about a real 2 or 3x multiplier loss relative to optimal. Cuda graphs do okay but they still leave a lot on the floor
It’s usually worth it to optimize in that context if you are willing to peer into the mechanics.
Of course that’s expensive. You need to know how to appropriately pipeline and merge your kernels.
I had a similar success with Model2Vec static embedder and NER inference (both GGUF, compiled for WASM), ported to plain C from ONNX Runtime.
Wasm size from 30Mb to 300kb and 1.5x speedup. It's definitely worth it for performance or distribution size.
I did took a native c++ approach when writing a relational transformers engine (RelativeDB). My journey was pytorch -> c++ -> Triton (lang). While C++ was more performant than Triton, I couldn't afford to optimize on every gpu. I just accepted the ~15% throughput loss for my cloud service, which honestly wasn't bad for the amount of flexibility I got out of it.
But the cpp port of vllm looks great, that'd be great if you'll maintain that. I hit the same limitations with vllm.
What you get: X is the A, Y is the B.
Thanks for the depth-anything.cpp port. Saves us a couple of kernel OOM's, causing connection disruptions.
Could this vllm port be faster to install? Im starting gpu machine multiple times a day and it takes 5 minutes to set vllm up. If Inise this port that time is minimized?
This is on my list to evaluate, I absolutely do not want to download 9gb of supply chain risk into prod every time we upgrade, when I can compile 70mb of binary. We run vLLM in a container with hardware passthrough for gitops, having the entire environment in a single container would drastically improve things and move local LLM into a pattern that more closely follows our other CI/CD systems, rather than this hulking behemoth snowflake deployment.
Not sure what's worse, if the AI slop content, or the AI code slop that makes the site consume 100% of a single CPU core (and if you enable GPU rendering on the browser, CPU usage is still significantly above zero.)
[dead]
[flagged]
[flagged]
[dead]
People are critiquing whether or not AI wrote the article.
Well, it's clearly not 1-shot / low effort.
(can we agree there's a difference between low effort 1-shotting a prompt and those who use LLMs interactively?)
The one reason why I can't take a mega hardline stance against AI for writing: Not everyone speaks English as a first language.
People with awful English speaking and writing skills are using LLMs to contribute content in English. This is a good thing. They used to try their best and suffix every post with "sorry for my english".
The author's name is Ettore Di Giacinto - I put 2 and 2 together on this one.
I give a pass when non-native English speakers are just using it to write better English and keep up with the rest of us. It wasn't super obvious anyway, it wasn't interruptive.
The content is beyond that, and I more than welcome stuff like this here.