You can tell when this deal started to come together by looking at the history of the website on Wayback Machine. In fall of 2024, the website had a checklist comparing SDF to dbt and claiming SDF had a better feature set than dbt Core (page rendering is hit and miss right now for whatever reason):
https://web.archive.org/web/20240919110243/https://www.sdf.c...
Alas, dbt Labs has developed a reputation for rug pulling functionality from dbt Core and gating most of their differentiating features behind dbt Cloud. I cannot see this type of consolidation being in the best interest of the dbt community.
dbt Labs is a Series D company with hundreds of millions in funding and a 4.2 billion USD valuation at their last round.
Their CEO and founder spoke of an IPO in 2022.
Let's not pretend they are still remotely close to their humble beginnings or were able to get this far without credibly demonstrating they have a plan for how to make enterprises bleed through their nose for their product.
That's the future.
On the flipside, building a dbt adjacent product enhancing or complementing capabilities is basically a sure way of how to get bought.
I've been on the lookout for a lighter, faster version of dbt and I was hoping sdf might be it.
For our (https://www.definite.app/) use case, I'd love to have something that compiles client-side, but in general dbt just feels like a lot of work to set up for what most of our customers actually need (simple transform to create tables and views).
Yeah my experience is much closer to that, I generally point my clients to core over cloud even if they're indifferent to the cost. (Sorry dbt guys, love your product, but somebody read the strategy memo backwards and you've got lock-out not lock-in. Replacing my IDE, ci/cd, or orchestration are "dealbreakers", not "features")
Could you point to some functionalities removed from dbt Core? I love dbt and use it where applicable but I have not yet encountered a loss of features upon upgrade yet - it would be useful to be aware what kind of features get removed
A brief list of features withheld or removed from Core:
- The dbt docs functionality is no longer maintained in favor of dbt Explorer in dbt cloud. A natural consequence is that larger dbt Core projects simply cannot leverage local docs due to performance defects.
- Multi-project support was widely discussed in the core repo w/ tooling contributions from the community, but that was locked behind enterprise-tier dbt cloud accounts
- Metricflow was a full OSS application that used to work in tandem with dbt Core. Post-acquisition, the original code was re-licensed and the functionality added to Cloud only (and you have to pay per semantic layer query now).
dbt needs to play catch up against SQLMesh feature-wise, so they bought their other competitor SDF. SQLMesh seems to have more development velocity, and dbt will need to execute a smooth transition and integration to catch up.
For context, the team behind SQLMesh also develops SQLGlot, which powers the features dbt attempts to implement
To expand on this, dbt uses Jinja templating in your SQL to allow the developer to modify the query so it can be expanded at "compile-time" into the target database SQL dialect. (uni-directional, write once deploy anywhere). The key features are CICD, test automation, and transformation sequence automation.
SQLGlot is a Abstract Symbol Tree SQL Dialect transpiler that could (in theory) convert from one dialect to another (bi-directional).
SQLMesh appears to combine both of the above into one tool that sounds like it's even better.
I'm curious if this product actually works because I was beating my head against almost exactly this very recently.
I haven't checked yet, but is SDF schema-dependent? For some reason, all sql comprehending/transforming tools I can find are either too trivial to be useful or require an exact schema to operate, both of which are too brittle to be useful when I try to use them in anger.
It compiles to SQL, so in since sense yes it is schema dependent. As mentioned in the article DBT itself has no understanding of semantics, so it’s not enforced by DBT directly but if it doesn’t compile to working SQL expressions it’s obviously not going to work.
Seems like a great pair. Especially the bit about static analysis instead of using string parsing.
Frankly, the dbt product hasn't really evolved much. I've been a bit disappointed with its lack of evolution toward this stuff organically. The "modern data stack" is in kind of in a magic position where they are working at very technical companies but the people using it are not SWEs who can build out the tooling themselves so they are just getting buckets of money without a really big value proposition. My team self hosts a dbt core workflow and it's been almost trivial to build out dbt's paid product ourselves
I guess if they integrate this then that will be the case, I'm not sure I'm convinced that much stuff is going on in dbt cloud that I can't do in dbt core.
A comparison that's obvious to my team is the release cadence of Metabase, which we also self-host. The frequent rollout of new features in Metabase is great and gives me more confidence in the product. But I'm not in the position to decide whether to cough up the $ for the paid version so I suppose it's moot.
I briefly thought this was about sdf.org and was very, very sad. Congrats on the exit to the sdf.com people though, and relief to the sdf.org userbase :)
Looker is mostly a reporting tool. LookML let's you define some transformations that happen at query time. There is a little bit of materialization functionality with the PDTs, but that doesn't seem intended as a replacement for other "ELT" tools.
You can tell when this deal started to come together by looking at the history of the website on Wayback Machine. In fall of 2024, the website had a checklist comparing SDF to dbt and claiming SDF had a better feature set than dbt Core (page rendering is hit and miss right now for whatever reason): https://web.archive.org/web/20240919110243/https://www.sdf.c...
In December 2024 the page had been updated to now compare "dbt Core" against "SDF with dbt": https://web.archive.org/web/20241217172451/https://www.sdf.c...
Little marketing switcharoo there to avoid pissing off their future owners.
Congrats to the SDF team for their exit.
Alas, dbt Labs has developed a reputation for rug pulling functionality from dbt Core and gating most of their differentiating features behind dbt Cloud. I cannot see this type of consolidation being in the best interest of the dbt community.
dbt Labs is a Series D company with hundreds of millions in funding and a 4.2 billion USD valuation at their last round.
Their CEO and founder spoke of an IPO in 2022.
Let's not pretend they are still remotely close to their humble beginnings or were able to get this far without credibly demonstrating they have a plan for how to make enterprises bleed through their nose for their product.
That's the future.
On the flipside, building a dbt adjacent product enhancing or complementing capabilities is basically a sure way of how to get bought.
I agree with you 100%, and we may both be correct!
I've been on the lookout for a lighter, faster version of dbt and I was hoping sdf might be it.
For our (https://www.definite.app/) use case, I'd love to have something that compiles client-side, but in general dbt just feels like a lot of work to set up for what most of our customers actually need (simple transform to create tables and views).
A lot of work to set up?
I'm quite surprised to hear that.
It's literally pip install, a single file for your DB config and that's it. 30-40 seconds.
I'm in no way affiliated with dbt but have worked with the tool since 2018.
Lighter, faster, sure, but hard to set up?
I'm not sure where you'd want to cut corners on setup.
I think they mean setting up in production.
Even that... the beauty of it and why it took off as much as it did is simplicity.
Dockerfile, env var injection and you're done.
I'm sure you've heard of SQLMesh but that seems like a potential fit. Or is it still too heavy handed?
Not to mention the sudden pricing change at the end of 2022 that doubled costs for most cloud customers.
Have they introduced any interesting features outside of core? Most of it seems like fluff, or there are better alternatives.
Yeah my experience is much closer to that, I generally point my clients to core over cloud even if they're indifferent to the cost. (Sorry dbt guys, love your product, but somebody read the strategy memo backwards and you've got lock-out not lock-in. Replacing my IDE, ci/cd, or orchestration are "dealbreakers", not "features")
Could you point to some functionalities removed from dbt Core? I love dbt and use it where applicable but I have not yet encountered a loss of features upon upgrade yet - it would be useful to be aware what kind of features get removed
A brief list of features withheld or removed from Core:
- The dbt docs functionality is no longer maintained in favor of dbt Explorer in dbt cloud. A natural consequence is that larger dbt Core projects simply cannot leverage local docs due to performance defects.
- Multi-project support was widely discussed in the core repo w/ tooling contributions from the community, but that was locked behind enterprise-tier dbt cloud accounts
- Metricflow was a full OSS application that used to work in tandem with dbt Core. Post-acquisition, the original code was re-licensed and the functionality added to Cloud only (and you have to pay per semantic layer query now).
Checking their own post, https://www.getdbt.com/product/dbt-core-vs-dbt-cloud, stuff such as the semantic layer and column-level lineage are Cloud exclusive
those were never in the core though.
We're using Dagster cloud with integrated DBT core and I don't really see what the draw of DBT cloud is - perhaps a bit easier to get set up?
Column level lineage for one
I first thought this was SDF (https://sdf.org/) and thought how could this happen.
Again shows we have run out of 3 letter acronyms :)
Glad I wasn't the only one! ;)
Heh, I was just starting to look at SDF and SQL Mesh to see if they actually addressed any of the dbt pain points.
dbt needs to play catch up against SQLMesh feature-wise, so they bought their other competitor SDF. SQLMesh seems to have more development velocity, and dbt will need to execute a smooth transition and integration to catch up.
For context, the team behind SQLMesh also develops SQLGlot, which powers the features dbt attempts to implement
To expand on this, dbt uses Jinja templating in your SQL to allow the developer to modify the query so it can be expanded at "compile-time" into the target database SQL dialect. (uni-directional, write once deploy anywhere). The key features are CICD, test automation, and transformation sequence automation.
SQLGlot is a Abstract Symbol Tree SQL Dialect transpiler that could (in theory) convert from one dialect to another (bi-directional).
SQLMesh appears to combine both of the above into one tool that sounds like it's even better.
What is your initial impression and pros and cons?
(Asking as I kinda wish my company’s opinionated dbt had made some different choices)
con is some ppl just go crazy with jinja and create a mess. Its fine if you just stick to the basics.
Please watch out for Server Side Template Injection.
Same. dbt feels like angularjs, I want the svelte alternative.
I'm curious if this product actually works because I was beating my head against almost exactly this very recently.
I haven't checked yet, but is SDF schema-dependent? For some reason, all sql comprehending/transforming tools I can find are either too trivial to be useful or require an exact schema to operate, both of which are too brittle to be useful when I try to use them in anger.
It compiles to SQL, so in since sense yes it is schema dependent. As mentioned in the article DBT itself has no understanding of semantics, so it’s not enforced by DBT directly but if it doesn’t compile to working SQL expressions it’s obviously not going to work.
Seems like a great pair. Especially the bit about static analysis instead of using string parsing.
Frankly, the dbt product hasn't really evolved much. I've been a bit disappointed with its lack of evolution toward this stuff organically. The "modern data stack" is in kind of in a magic position where they are working at very technical companies but the people using it are not SWEs who can build out the tooling themselves so they are just getting buckets of money without a really big value proposition. My team self hosts a dbt core workflow and it's been almost trivial to build out dbt's paid product ourselves
It hasn't evolved 'cause there is no free lunch.
It has evolved quite a bit behind the paywall - as has the paywall.
Most if not all of this stuff will land behind, not before said wall.
I guess if they integrate this then that will be the case, I'm not sure I'm convinced that much stuff is going on in dbt cloud that I can't do in dbt core.
A comparison that's obvious to my team is the release cadence of Metabase, which we also self-host. The frequent rollout of new features in Metabase is great and gives me more confidence in the product. But I'm not in the position to decide whether to cough up the $ for the paid version so I suppose it's moot.
Congrats to everyone, very excited to see how these come together
I briefly thought this was about sdf.org and was very, very sad. Congrats on the exit to the sdf.com people though, and relief to the sdf.org userbase :)
Interesting that their pricing page is already 404'ing: https://www.sdf.com/pricing
they were charging for compute: $0.16/vCPU min
https://web.archive.org/web/20241217210804/https://www.sdf.c...
older version:
Professional
For development teams looking to scale fast.
$1,250/mo
Annually
Everything in Plus
Up to 1,500 models
Unlimited Team Members
Direct Slack Connect support
Unlimited compile, test, & runs
What does Dbt Labs do? I've read their website, but I need a concrete example.
In simplest of terms, it's a templating and orchestration layer for data pipelines which which compiles purely into SQL expressions.
How is it different from Looker and LookML?
Looker is mostly a reporting tool. LookML let's you define some transformations that happen at query time. There is a little bit of materialization functionality with the PDTs, but that doesn't seem intended as a replacement for other "ELT" tools.
Think of dbt as yarn for SQL, only the SQL is built from jinja templates.
dbtlabs also sells dbt cloud, which provides a runtime for your dbt projects.
It takes your data operation from 1995 to 2025
Given replies to this comment, I cannot think of a more useless thing. Yarn for sql in cloud, why?? Why is that needed???
These days I get to know about services only when they get acquired.
For a second I thought that this was about sdf.org ... but it is about sdf.com (which I have never heard of before)
As someone who has been MetaARPA for about 20 years at this point I would be _pissed_ if they sold out.
Same here, small panic attack ensued ;P