"Inference" is just software running. It has always cost money to run software, it's just that it is generally too cheap to matter. If a client makes a regular API call to your server, you pay for that compute, probably in the form of a flat hosting fee. If too many calls come in and workload goes up, you pay for a more expensive hosting tier to handle it (or do dynamic scaling which is per-unit of compute).
Right now the "hosting" cost for inference is per-unit because it's new and expensive, but that won't last.
There is a lot of inefficiency right now keeping prices elevated. That will change very fast and soon paying for inference will likely resemble paying for hosting your app.
The bigger problem for SaaS is that the floor has risen - people can build their own solutions for things that they used to buy SaaS for. So the industry needs to level up and solve harder problems.
Exactly my point in another comment. Just to illustrate this further: a rough ballpark of how the cost of intelligence fell since 2022 could be about 1000x, and continues to fall. Unfortunately, it’s really hard to measure.
It’s so cheap that companies choose to spend more on AI inference (more reasoning, more capabilities, longer context), not less - see Jevons paradox.
>Users began expecting something fundamentally different from software: not just tools that store and retrieve, but products that reason, generate, and respond.
Not really.
>Every inference call costs money.
Not really, either. If you buy your own GPU, rack it, and run an open model, there is no unit cost. This is just expensive hosting infra. You also pay unit costs for SaaS that your software uses (things like SMS etc).
> If you buy your own GPU, rack it, and run an open model, there is no unit cost.
No. There is economic opportunity cost (borrowing), energy cost, infra cost, depreciation / risk of failure with each unit of work, bandwidth, maintenance, and lots more. Small, but not zero, and often overlooked - especially the opportunity cost.
These are basically all fixed costs, not unit costs. You buy the GPU once and use it for as many calls as you have traffic for, and depreciate it over a fixed lifetime. You have to power it regardless of whether it's fully utilized or not. You have to maintain it by virtue of owning it, not really based on how many queries it has served. Bandwidth is the only one that really scales as a unit cost.
Open question whether this model is actually more economical than using the cloud AI service. The whole reason the industry moved to cloud computing in the first place was because computing had very high fixed costs, and the more these could be amortized over a fully-loaded query stream, the lower the unit costs.
I don't think it's at all certain this won't land back on the same unit economics as the old way. The cost of serving a user doesn't have to be free - it never has been - it just has to not be the dominating factor in your costs. I'm guessing there are still quite a lot of per-user costs that aren't easily visible. Like how many of your users are logging support requests, or suing you, or demanding bug fixes or custom integrations or a myriad of other things. And how much are you having to invest in security updates, regulatory compliance, marketing etc. Not to mention, users are getting well acclimatised to the idea of quotas and paying for increased limits.
I think part of the new equation may also become; "Why even pay for the SaaS in the first place if you can just forge the service exactly how you want it?" The benefits of unlimited access to the tool you forge are still there, its just a lot easier to make whatever tool you want.
Are there any examples of products containing ai inference that are successful? Products that are beyond just direct access to frontier LLM's, I mean.
As if making all the decisions that go into that product, hosting it, and maintaining it are free! Now do that for all the SaaS you subscribe to. Would you even get any real work done?
> Meeting that expectation means making LLM calls, and LLM calls cost money.
Of course, and so does everything in the software world. The point is getting the cost so low that it’s basically free. The new DS V4 Flash or the smaller Qwen3.6 models are still really expensive compared to what we were used to in the economics of software, but it’s not unreasonable to expect these costs to continue falling down.
Rough chatgpt estimate says 3-5 orders of magnitude of difference compared to a typical user interaction with a SPA (db/cache lookup, CDN…)
Yeah, my point is that LLMs can reach that point too. Especially if you do it clientside. Copying was also way more expensive back in the 60s (accounting just for machine time and electricity, not storage cost, about 100 million times more expensive than today). Everything has a cost.
Definitely, of course. But I think the logic of the author’s article is based on there being a huge difference between the two, or rather a high cost of inference in absolute terms. And that can change and we have seen it change. Which breaks the entire premise of the article going forward, no?
Inference will be always more expensive than db operations or copying, sure. But how much more expensive is the question.
This is one reason I've been trying to figure out tasks (and products based on those tasks) that can be pushed to the edge, either via small specialized models or small general purpose open models. I suspect the same desire to keep unit costs low is part of why Google is falling behind on the "frontier", but seemingly at the lead, or near it, on models that run on-device. I think they're just focused on making models for tasks that don't require boiling the ocean.
But, it's a hard problem. The models that run locally on normal computers/phones are pretty terrible compared to the frontier, without specialization and fine-tuning. And, even with specialization and fine-tuning, often a high-end general purpose model is going to do a better job and people don't need a bunch of local tools installed to do their various tasks.
> And, even with specialization and fine-tuning, often a high-end general purpose model is going to do a better job and people don't need a bunch of local tools installed to do their various tasks.
I think this is the critical point that would be interesting to see if it holds. Technology seemingly tends towards increased specialization.
If I'm any kind of indicator of where Youtube users are headed, their AI chatbots in the video pages are going to kill their business model.
There are so many videos with hooks/teasers/'you won't believe what we discovered!!1', and now I just pause the video in the first second, ask "what's the tldr" and get the value from the video without a single ad impression (and likely racking up far more opex for Youtube than if I just streamed the video).
It depends on the solution. If AI is generating value, you can charge for the value.
Most SaaS already works this way. M365 or Adobe Creative Cloud are great examples. They value it like a life insurance policy and find ways to make you sticky. It’s easier to just buy it.
The first round of AI products suck because they are not well defined. Copilot only makes sense if you do shit in office and SharePoint isn’t a dumpster fire. In my large O365 environment the bottom 50% of users use less storage than the top 2%. So why would i buy copilot for my janitor?
When M365 E9 reconciles invoices automatically with Excel, I’ll pay $150/mo and fire a bunch of people.
"Users began expecting something fundamentally different from software: not just tools that store and retrieve, but products that reason, generate, and respond."
Absolutely not. Customers want systems for sales, reservations, accounting, and taking stock. That's where almost all the SaaS money is and none of it benefits from AI - and never will.
"Inference" is just software running. It has always cost money to run software, it's just that it is generally too cheap to matter. If a client makes a regular API call to your server, you pay for that compute, probably in the form of a flat hosting fee. If too many calls come in and workload goes up, you pay for a more expensive hosting tier to handle it (or do dynamic scaling which is per-unit of compute).
Right now the "hosting" cost for inference is per-unit because it's new and expensive, but that won't last.
There is a lot of inefficiency right now keeping prices elevated. That will change very fast and soon paying for inference will likely resemble paying for hosting your app.
The bigger problem for SaaS is that the floor has risen - people can build their own solutions for things that they used to buy SaaS for. So the industry needs to level up and solve harder problems.
Exactly my point in another comment. Just to illustrate this further: a rough ballpark of how the cost of intelligence fell since 2022 could be about 1000x, and continues to fall. Unfortunately, it’s really hard to measure.
It’s so cheap that companies choose to spend more on AI inference (more reasoning, more capabilities, longer context), not less - see Jevons paradox.
>Users began expecting something fundamentally different from software: not just tools that store and retrieve, but products that reason, generate, and respond.
Not really.
>Every inference call costs money.
Not really, either. If you buy your own GPU, rack it, and run an open model, there is no unit cost. This is just expensive hosting infra. You also pay unit costs for SaaS that your software uses (things like SMS etc).
> If you buy your own GPU, rack it, and run an open model, there is no unit cost.
No. There is economic opportunity cost (borrowing), energy cost, infra cost, depreciation / risk of failure with each unit of work, bandwidth, maintenance, and lots more. Small, but not zero, and often overlooked - especially the opportunity cost.
These are basically all fixed costs, not unit costs. You buy the GPU once and use it for as many calls as you have traffic for, and depreciate it over a fixed lifetime. You have to power it regardless of whether it's fully utilized or not. You have to maintain it by virtue of owning it, not really based on how many queries it has served. Bandwidth is the only one that really scales as a unit cost.
Open question whether this model is actually more economical than using the cloud AI service. The whole reason the industry moved to cloud computing in the first place was because computing had very high fixed costs, and the more these could be amortized over a fully-loaded query stream, the lower the unit costs.
I don't think it's at all certain this won't land back on the same unit economics as the old way. The cost of serving a user doesn't have to be free - it never has been - it just has to not be the dominating factor in your costs. I'm guessing there are still quite a lot of per-user costs that aren't easily visible. Like how many of your users are logging support requests, or suing you, or demanding bug fixes or custom integrations or a myriad of other things. And how much are you having to invest in security updates, regulatory compliance, marketing etc. Not to mention, users are getting well acclimatised to the idea of quotas and paying for increased limits.
I think part of the new equation may also become; "Why even pay for the SaaS in the first place if you can just forge the service exactly how you want it?" The benefits of unlimited access to the tool you forge are still there, its just a lot easier to make whatever tool you want.
Are there any examples of products containing ai inference that are successful? Products that are beyond just direct access to frontier LLM's, I mean.
As if making all the decisions that go into that product, hosting it, and maintaining it are free! Now do that for all the SaaS you subscribe to. Would you even get any real work done?
So many pivot opportunities, we’ll live remotely from office.
> Meeting that expectation means making LLM calls, and LLM calls cost money.
Of course, and so does everything in the software world. The point is getting the cost so low that it’s basically free. The new DS V4 Flash or the smaller Qwen3.6 models are still really expensive compared to what we were used to in the economics of software, but it’s not unreasonable to expect these costs to continue falling down.
Rough chatgpt estimate says 3-5 orders of magnitude of difference compared to a typical user interaction with a SPA (db/cache lookup, CDN…)
> and so does everything in the software world.
Well, no. Copying is free, or so near free it makes zero sense to charge. LLMs are just papering over the damage caused by profit.
Yeah, my point is that LLMs can reach that point too. Especially if you do it clientside. Copying was also way more expensive back in the 60s (accounting just for machine time and electricity, not storage cost, about 100 million times more expensive than today). Everything has a cost.
LLMs will never be as cheap or reliable as copying.
Definitely, of course. But I think the logic of the author’s article is based on there being a huge difference between the two, or rather a high cost of inference in absolute terms. And that can change and we have seen it change. Which breaks the entire premise of the article going forward, no?
Inference will be always more expensive than db operations or copying, sure. But how much more expensive is the question.
This is one reason I've been trying to figure out tasks (and products based on those tasks) that can be pushed to the edge, either via small specialized models or small general purpose open models. I suspect the same desire to keep unit costs low is part of why Google is falling behind on the "frontier", but seemingly at the lead, or near it, on models that run on-device. I think they're just focused on making models for tasks that don't require boiling the ocean.
But, it's a hard problem. The models that run locally on normal computers/phones are pretty terrible compared to the frontier, without specialization and fine-tuning. And, even with specialization and fine-tuning, often a high-end general purpose model is going to do a better job and people don't need a bunch of local tools installed to do their various tasks.
> And, even with specialization and fine-tuning, often a high-end general purpose model is going to do a better job and people don't need a bunch of local tools installed to do their various tasks.
I think this is the critical point that would be interesting to see if it holds. Technology seemingly tends towards increased specialization.
If I'm any kind of indicator of where Youtube users are headed, their AI chatbots in the video pages are going to kill their business model.
There are so many videos with hooks/teasers/'you won't believe what we discovered!!1', and now I just pause the video in the first second, ask "what's the tldr" and get the value from the video without a single ad impression (and likely racking up far more opex for Youtube than if I just streamed the video).
It depends on the solution. If AI is generating value, you can charge for the value.
Most SaaS already works this way. M365 or Adobe Creative Cloud are great examples. They value it like a life insurance policy and find ways to make you sticky. It’s easier to just buy it.
The first round of AI products suck because they are not well defined. Copilot only makes sense if you do shit in office and SharePoint isn’t a dumpster fire. In my large O365 environment the bottom 50% of users use less storage than the top 2%. So why would i buy copilot for my janitor?
When M365 E9 reconciles invoices automatically with Excel, I’ll pay $150/mo and fire a bunch of people.
Customers expect more, so they'll pay more. It really isn't more complicated than that.
"Users began expecting something fundamentally different from software: not just tools that store and retrieve, but products that reason, generate, and respond."
Absolutely not. Customers want systems for sales, reservations, accounting, and taking stock. That's where almost all the SaaS money is and none of it benefits from AI - and never will.
Wonder how much this cost the author to write, as it’s just AI slop…