Looks like Andy's here, so if you see this - please also try to convince ClickHouse to consider funding DB research in academia. With all the money being poured into AI and the chaos in government funding, there is almost nothing for DB research anymore.
Remy - Good to hear from you. Yes things are mess right now. Let me get settled with setting up the lab and we'll figure out how we want to engage with academia.
I'm very curious about the convergence of the best in class fast OLAP products (StarRocks, ClickHouse) with Trino. It sounds like everybody is going for decoupled compute/storage, using S3 or similar as the storage layer, and thus forgoing colocated joins (ok I know ClickHouse joins suck)...
So what does this mean for ingestion (and indexing)? Iceberg V3? Paimon? Bespoke ingestion through the DB engine to do the indexing?
It's also interesting how Clickhouse / Starrocks can now also act as a query planner and executor on top of non-native formats (ex. Iceberg).
I assume the native formats will always be faster / more optimized but the need for Trino as a separate executor while running either of these databases seems to be close to gone.
The benefit of StarRocks/ClickHouse over Trino is that you get secondary indices, but that means you have to do the indexing somehow.
Native format is faster (especially for colocated joins), but it's way more expensive if you have to run a bunch of separate storage nodes vs just using S3, especially your query volume isn't that high.
I liken it to the BigQuery cost model, where storage is effectively free.
> I assume the native formats will always be faster / more optimized
Actually query on Iceberg is end-to-end faster on ClickHouse than MergeTree native format on disk. It's mostly a matter of how much compute you throw at it. Native MergeTree still wins in cases that depend on use of indexes to reduce I/O but scan speed is no longer an issue.
Opinionated take: ingestion should be treated as a streaming reorganization workload, separate from whatever a "database" is.
You do not even need to change the Iceberg spec, although you could. Just rejig things a bit to produce better statistics (think liquid clustering but on the write path). You still write the same dataset queried the same way, but automatically get better query performance. Workload owners should be able to decide the tier at which this streaming reorganization happens: they know their data staging and bandwidth constraints the best.
I’ve historically read this as ‘open format compatible’ but ‘native preferred’ - where this opens up market space and dev velocity - but it’ll be interesting to see if native storage differentiation gets dumped entirely. It just seems like ‘fork and optimize for our engine’ would always be tempting enough that you’d want a native play for when you don’t need the decoupling.
I “audited” CMU 15-445 in April 2026, a few months ago. I then applied to Azure Hyperscale DB team, and as my pitch to get hired I cold DM’d the hiring manager whatever I finished in the assignments (BPM, Concurrent B-tree, Transaction support etc).
With that, I was able to get the interview although I ended up failing it. My lack of experience in C++ was probably one of the main reasons. Your course helped me stand my own during the interview and even though I had zero database experience apart from finishing the course, I felt adequately prepared.
Thank you for making the course open source. As a working professional, it was helpful to be able to do the course on my own time. Suffice to say, I am a big fan of your work and thank you for introducing me (and other fellow learners) to the interesting world of Databases!
I know it’s a troll, This was no MOOC. I didn’t even get a certificate.
I saw the job opening in the careers portal and just cold DM’d the HM. As to why I DM’d, what do I have to lose ? I was already working at MSFT in a different team. The worst that could happen was that he would just ignore the message. Also, I didn’t nag the HM, it was just one DM where I summarised the things I did in the course. He then asked me to send me my resume and that was it.
Its refreshing to see corporate research labs in a non AI area. Clickhouse has been a huge beneficiary of the AI wave and its good to see some of the value going into advancing fundamental research in infrastructure
Congrats, that's incredible news. Watched apavlo@'s lectures while studying in the university and finished my bachelor thesis implementing features and doing research at ClickHouse. Surprised to see these worlds being together now!
There are several out of date entries in there. We used to have students help with writing entries. I just did a major refresh of the site this summer. I need to go through all the entries and figure out how to update them.
> The goal of ClickHouse Labs is to establish a best-in-class industry research organization focused on databases. It will not operate as an isolated research organization that throws ideas over the wall to engineering. Instead, we will work closely with ClickHouse engineers, customers, collaborators, and industry partners to develop and disseminate new ideas that keep ClickHouse at the bleeding edge.
This is cool, though bittersweet that the public research infrastructure (universities) is not really configured to support this kind of high-impact research any more.
There are areas of academia where I definitely worry about this kind of thing, but not really database research (and in general CS is a lot less worrisome than most fields).
It's a good thing when industry is a competitively attractive environment for research, which is how I read this.
Deep tech fields have a more complex relationship with academia than other industries. Engineers often need license to test ideas that may not see adoption for years, or require tens of millions to build-out.
A "research" arm is exactly this license, although it comes at the cost of potentially killing innovation in the rest of the company.
C'mon, databases are great but they are not in any way a "deep tech" field. Databases exist today in a zillion production forms as commercial products and free software.
Deep tech includes things like nuclear fusion, solid state batteries, quantum computers. I know everyone wants to feel cool, but just because your new javascript framework will be in beta for the next ten years doesn't make it "deep tech".
The databases we use today in production have severe limitations and are not even close to what is theoretically possible. Many traditional parts of a database (indexing, caching, scheduling, et al) are AI-complete algorithm problems. Entire sub-classes of database (e.g. graph or spatial) famously have persistently poor scalability and performance because of open questions in the foundational computer science.
Just the fact that increasing the generality, scalability, and performance of databases asymptotically converges on designing AGI suggests that it is, in fact, "deep tech". And this property has to mesh with other practical constraints on database behavior. Many problems in databases are hard with little forward progress in decades.
It is true that most database research is not deep tech but there is ample room for it to be if one is sufficiently ambitious.
Could you explain what practical research there is to be done? The heavy theory I know does not seem to be very useful in practice. Optimal join algorithms, Yannakakis adjacent algorithms, tree decomposition of queries all seem to be worse than well implemented naive algorithms. But maybe the implementations of the new algorithms just are not good? I really don’t know.
There are several interesting foundational problems around scalability, generality, efficiency, and adaptability. Historically use cases haven't always been there to justify focusing on these problems but that is changing. In my opinion, some of the more important open problems:
Ideally a table should be index-organized across all relevant columns. No public system works anything like this. We don't have single indexing structures that work for a collection of arbitrary types each with possibly unpredictable distributions, never mind ones that mix temporal, geometric, and other difficult types. The AI-complete nature of indexing becomes evident when you dig into this. Downstream from this is an implication of extremely granular and adaptive storage management that current storage engines aren't designed for.
Tractable cache replacement algorithms are broken for many workloads and data models. These algorithms need to be very fast for search, update, and eviction selection but they are also AI-complete; improvements to generality have impractically high computational cost. Storage growth is decoupled from RAM availability thanks to disaggregation, aggravating the problem even for workloads that worked well under tractable cache replacement. In theory we know that cache admission (read: fancy latency-hiding schedules) is more robust and scales better but is so difficult to implement in non-trivial real systems that I don't think anyone has figured out how to reduce that concept to practice yet.
At exabyte scales, conventional database internals have embedded assumptions that no longer hold true. For example, you cannot guarantee even "small" internal control structures are resident in memory. What used to be fairly boring internals bits in databases suddenly have to be redesigned from first principles. This is more applied than theoretical but it suggests a major change in the way we do internal architecture.
Traditionally we've treated spatial and temporal locality as architecturally separate concerns. This is extremely convenient from a building real systems standpoint. Optimizing either one in isolation is adversarial to the efficiency of the other, which becomes increasingly visible as you scale up. Converging these concerns into a single "thing" almost certainly has solving the above problems as a prerequisite. If you squint, you can kind of see this as the last step before databases become literal AI.
All of these have really broad scope. If we could solve even half of these open research problems the resulting database engines would be unrecognizable. There are ton of other narrower interesting research problems around data layouts, compression, join parallelism, etc that still have potential for substantial improvement.
It is a great time to be doing database research, we've barely scratched the surface.
There's been a lot of fun, practical work recently on optimal join algorithms! (Full disclosure, this advisor on this paper is one of Andy and my former students, so I'm slightly biased)
Like most academic work, this one builds on some work that's been done over the last few years on ways to make the Yannakakis algorithm actually practical.
how do you optimally balance code generation and virtual function calling patterns for the best performance? Is there a standardized OLTP format on object storage? How do you performantly implement differential privacy policies within the db engine?
Are there re-usable query primitives for extremely large scale multi-modal data? how do you scale such queries or make them efficient?
I personally built quite several specialized hyper-performant DB engines, and studied most popular OSS projects, and believe most DB questions are theoretically answered long ago(decade back). The puzzle is mostly to assemble pieces together to fit specific tradeoff of performance/simplicity/functionality and not overengineer system.
I prefer to not study databases through academic papers, I think major OSS projects(PG, Cassandra, RockDb, duckdb, ClickHouse, etc) have all/most important data structures and algorithms implemented and described way better.
I'd ascribe "deep tech" to anything that you can reasonably get a PhD in and have it not be unusual. There are dozens of academic conferences on DBs pushing the frontier forward.
I know that many practical databases already exist and where I come from refining or improving something that already exists hardly qualifies as "deep tech". Is PepsiCo doing deep tech research on Crystal Pepsi?
However, the commenter I was responding to apparently considers just about _any_ research in any significant field to be deep tech so w/e I give up and I'll go update my CV to include the phrase "deep tech research"
Maybe only an OS can be close to how MUCH deep you can go with a DB engine.
And I bet DB engines can go far than normal OS.
You don't know how much is still waiting for somebody to try, and how much is not applied. And how many of that DBs are not even doing, because all are constrained be being "apps" with so poor interface (sql).
Fun fact: Not exist a viable true relational DBs implemented, neither exist one with a viable programming language AND apis that is for developers.
At the risk of arguing about semantics (and not interesting database semantics) it seems half the commenters here think "deep tech" just means "like, technology you go really super DEEP into, like really dig into that DEEP DEEP HARDCORE TECHNOLOGY"
> Deep tech innovations are often radical and may create new markets or disrupt existing ones. Deep tech companies often address big societal and environmental challenges and have potential to impact everyday life. Silicon chips are an example of innovation that enabled calculation at previously unimaginable speed and scale.
Database research is good, important, critical, even! But it's not creating something new that has never existed before. It's not inventing the transistor or the integrated circuit.
>> It's not inventing the transistor or the integrated circuit.
Integrated circuit on its own isn't doing anything unless used for something. Better chips enable more use-cases, more electronics etc etc. Databases in similar way are enabling more applications for users. None of the apps or services we reliably use everyday at the scale (e.g. Google Search, Online shopping, Messaging) would not exist without solid storage software.
> Integrated circuit on its own isn't doing anything unless used for something
YES! YES EXACTLY!
That's part of what makes things like ICs "deep tech" - they are transformative innovations that have no immediate application so they need government or a very well-funded far-thinking private org (e.g. Bell Labs) to bring them into being. They are not like making a new, better database.
You have just reiterated my point very concisely, so thank you.
I don't think I did reiterate your point. You were saying unlike ICs, databases -- "But it's not creating something new that has never existed before".
The point I was making was like newer/better ICs pave way for new electronics/use-cases; so do better databases/storage software pave way for more new applications that wouldn't have existed before.
I guess I get what you are saying. Creating the first of its kind is "deep tech".
I really don't care about the definition or semantics here personally. I think creating first one is great, but then depending what it is creating 10th generation can also be quite a breakthrough. First IC had like 10 transistors, modern ones do billions and getting there probably require way more work, ideas and research than the original IC did. Things just don't improve in silos.
I'm not sure what you think "research lab" means? As I mentioned in the article, we look at IBM Research (Almaden) and Microsoft Research as inspiration.
Many database companies have research labs. It allows you to explore ideas, unorthodox concepts, and parts of the design space that may never be reflected in an actual product. Or to figure out how to solve specific hard problems that you come across with neither a good solution nor proof of impossibility in literature.
This is essential for a company that wants to stay on the frontier of database tech.
Which database companies have research labs? AFAIK amazon, snowflake, databricks, google and SAP don’t have a research lab dedicated to databases. They have some people that are paid to do research but there does not seem to be an IBM Almaden anywhere in the world.
> ClickHouse had features that at the time were only found in a handful of closed-source, commercial analytical DBMSs. For example, ClickHouse was written in C++ and supported vectorized query execution using SIMD in 2016. Most prominent open-source analytical DBMSs in 2016 were JVM-based and did not support SIMD optimizations until years later.
Performance is a feature. "Written in C++" is a strange idea of a feature.
Looks like Andy's here, so if you see this - please also try to convince ClickHouse to consider funding DB research in academia. With all the money being poured into AI and the chaos in government funding, there is almost nothing for DB research anymore.
Remy - Good to hear from you. Yes things are mess right now. Let me get settled with setting up the lab and we'll figure out how we want to engage with academia.
I'm very curious about the convergence of the best in class fast OLAP products (StarRocks, ClickHouse) with Trino. It sounds like everybody is going for decoupled compute/storage, using S3 or similar as the storage layer, and thus forgoing colocated joins (ok I know ClickHouse joins suck)...
So what does this mean for ingestion (and indexing)? Iceberg V3? Paimon? Bespoke ingestion through the DB engine to do the indexing?
It's also interesting how Clickhouse / Starrocks can now also act as a query planner and executor on top of non-native formats (ex. Iceberg).
I assume the native formats will always be faster / more optimized but the need for Trino as a separate executor while running either of these databases seems to be close to gone.
The benefit of StarRocks/ClickHouse over Trino is that you get secondary indices, but that means you have to do the indexing somehow.
Native format is faster (especially for colocated joins), but it's way more expensive if you have to run a bunch of separate storage nodes vs just using S3, especially your query volume isn't that high.
I liken it to the BigQuery cost model, where storage is effectively free.
> I assume the native formats will always be faster / more optimized
Actually query on Iceberg is end-to-end faster on ClickHouse than MergeTree native format on disk. It's mostly a matter of how much compute you throw at it. Native MergeTree still wins in cases that depend on use of indexes to reduce I/O but scan speed is no longer an issue.
Opinionated take: ingestion should be treated as a streaming reorganization workload, separate from whatever a "database" is.
You do not even need to change the Iceberg spec, although you could. Just rejig things a bit to produce better statistics (think liquid clustering but on the write path). You still write the same dataset queried the same way, but automatically get better query performance. Workload owners should be able to decide the tier at which this streaming reorganization happens: they know their data staging and bandwidth constraints the best.
I’ve historically read this as ‘open format compatible’ but ‘native preferred’ - where this opens up market space and dev velocity - but it’ll be interesting to see if native storage differentiation gets dumped entirely. It just seems like ‘fork and optimize for our engine’ would always be tempting enough that you’d want a native play for when you don’t need the decoupling.
Clickhouse joins have been improving almost every month for the last couple of years. Maybe they still suck but a lot less
Always enjoyed his lecture series from CMU, hopefully those continue in a sponsored format from Clickhouse.
They will continue. New seminar series starts next month (announcement coming this week).
I “audited” CMU 15-445 in April 2026, a few months ago. I then applied to Azure Hyperscale DB team, and as my pitch to get hired I cold DM’d the hiring manager whatever I finished in the assignments (BPM, Concurrent B-tree, Transaction support etc).
With that, I was able to get the interview although I ended up failing it. My lack of experience in C++ was probably one of the main reasons. Your course helped me stand my own during the interview and even though I had zero database experience apart from finishing the course, I felt adequately prepared.
Thank you for making the course open source. As a working professional, it was helpful to be able to do the course on my own time. Suffice to say, I am a big fan of your work and thank you for introducing me (and other fellow learners) to the interesting world of Databases!
Why did you think it was appropriate to nag the hiring manager while you finished a MOOC?
I know it’s a troll, This was no MOOC. I didn’t even get a certificate.
I saw the job opening in the careers portal and just cold DM’d the HM. As to why I DM’d, what do I have to lose ? I was already working at MSFT in a different team. The worst that could happen was that he would just ignore the message. Also, I didn’t nag the HM, it was just one DM where I summarised the things I did in the course. He then asked me to send me my resume and that was it.
What's the hiring manager's job? Hiring people? Why would that be inappropriate?
I read it as "After finishing the course, I DM'd the hiring manager with a list of the stuff I did for the course."
Will you maintain a connection to CMU? If so, what will you continue doing and at what percent of effort?
Your DB course is phenomenal. Thank you very much for making it available to all of us!
Its refreshing to see corporate research labs in a non AI area. Clickhouse has been a huge beneficiary of the AI wave and its good to see some of the value going into advancing fundamental research in infrastructure
Congrats, that's incredible news. Watched apavlo@'s lectures while studying in the university and finished my bachelor thesis implementing features and doing research at ClickHouse. Surprised to see these worlds being together now!
Clickhouse just became the hottest talent-attraction on the market.
Congrats Andy, hope you enjoy the ride =)
Andy taught me a lot about trolling. I also hear he's banned from every post office within 50 miles of Baltimore.
Ok now I need to know.
Andy was a PhD student at Brown when I was an undergrad.
Once he hired a magician off of Craigslist for a departmental event. The guy showed up drunk and lost his bird in the building.
Or the time he ran BitTorrent-over-MediaWiki and made a number of people very angry.
https://pdfs.semanticscholar.org/01e4/ecb5a12b6735dd0344d888...
Andy, go ahead and correct the https://dbdb.io/db/bigquery How did you come to the conclusion that BigQuery's data model is "Document / XML" :joy:
This entry is not only wrong, it's way outdated, therefore misinforming. I'm wondering about the overall quality of this dbdb.io database at large.
There are several out of date entries in there. We used to have students help with writing entries. I just did a major refresh of the site this summer. I need to go through all the entries and figure out how to update them.
> The goal of ClickHouse Labs is to establish a best-in-class industry research organization focused on databases. It will not operate as an isolated research organization that throws ideas over the wall to engineering. Instead, we will work closely with ClickHouse engineers, customers, collaborators, and industry partners to develop and disseminate new ideas that keep ClickHouse at the bleeding edge.
This is cool, though bittersweet that the public research infrastructure (universities) is not really configured to support this kind of high-impact research any more.
What do you mean? There’s plenty of high impact research on databases happening in academia. (Including, for example, Andy’s group at CMU)
Also keep in mind that the part you quoted is partially marketing copy.
I take this announcement to mean he is leaving. So he's a perfect example
There are areas of academia where I definitely worry about this kind of thing, but not really database research (and in general CS is a lot less worrisome than most fields).
It's a good thing when industry is a competitively attractive environment for research, which is how I read this.
That's great news for Clickhouse and for Andy I guess, they are both obsessed with databases and are at the cutting edge.
Hi Andy, that’s very cool news. Will you be at the HPTS workshop in October?
What has been your experience with HPTS? It will be my first time presenting.
TBD
Hope it goes well for you Andy. Have loved your lectures over the years.
Best of luck!
Andy, you had me at KillerMike + El-P super group. Best of luck running the jewels on the dbase world with CH
Congrats! Andy is incredible!
Yooo is this the CMU funny guy?
[dead]
[dead]
[flagged]
Clickhouse Labs is a research lab for ClickHouse’s engineers and their db product?
Whatever floats your boat. Sounds like you just work as an engineer at a db company
Deep tech fields have a more complex relationship with academia than other industries. Engineers often need license to test ideas that may not see adoption for years, or require tens of millions to build-out.
A "research" arm is exactly this license, although it comes at the cost of potentially killing innovation in the rest of the company.
C'mon, databases are great but they are not in any way a "deep tech" field. Databases exist today in a zillion production forms as commercial products and free software.
Deep tech includes things like nuclear fusion, solid state batteries, quantum computers. I know everyone wants to feel cool, but just because your new javascript framework will be in beta for the next ten years doesn't make it "deep tech".
The databases we use today in production have severe limitations and are not even close to what is theoretically possible. Many traditional parts of a database (indexing, caching, scheduling, et al) are AI-complete algorithm problems. Entire sub-classes of database (e.g. graph or spatial) famously have persistently poor scalability and performance because of open questions in the foundational computer science.
Just the fact that increasing the generality, scalability, and performance of databases asymptotically converges on designing AGI suggests that it is, in fact, "deep tech". And this property has to mesh with other practical constraints on database behavior. Many problems in databases are hard with little forward progress in decades.
It is true that most database research is not deep tech but there is ample room for it to be if one is sufficiently ambitious.
Could you explain what practical research there is to be done? The heavy theory I know does not seem to be very useful in practice. Optimal join algorithms, Yannakakis adjacent algorithms, tree decomposition of queries all seem to be worse than well implemented naive algorithms. But maybe the implementations of the new algorithms just are not good? I really don’t know.
There are several interesting foundational problems around scalability, generality, efficiency, and adaptability. Historically use cases haven't always been there to justify focusing on these problems but that is changing. In my opinion, some of the more important open problems:
Ideally a table should be index-organized across all relevant columns. No public system works anything like this. We don't have single indexing structures that work for a collection of arbitrary types each with possibly unpredictable distributions, never mind ones that mix temporal, geometric, and other difficult types. The AI-complete nature of indexing becomes evident when you dig into this. Downstream from this is an implication of extremely granular and adaptive storage management that current storage engines aren't designed for.
Tractable cache replacement algorithms are broken for many workloads and data models. These algorithms need to be very fast for search, update, and eviction selection but they are also AI-complete; improvements to generality have impractically high computational cost. Storage growth is decoupled from RAM availability thanks to disaggregation, aggravating the problem even for workloads that worked well under tractable cache replacement. In theory we know that cache admission (read: fancy latency-hiding schedules) is more robust and scales better but is so difficult to implement in non-trivial real systems that I don't think anyone has figured out how to reduce that concept to practice yet.
At exabyte scales, conventional database internals have embedded assumptions that no longer hold true. For example, you cannot guarantee even "small" internal control structures are resident in memory. What used to be fairly boring internals bits in databases suddenly have to be redesigned from first principles. This is more applied than theoretical but it suggests a major change in the way we do internal architecture.
Traditionally we've treated spatial and temporal locality as architecturally separate concerns. This is extremely convenient from a building real systems standpoint. Optimizing either one in isolation is adversarial to the efficiency of the other, which becomes increasingly visible as you scale up. Converging these concerns into a single "thing" almost certainly has solving the above problems as a prerequisite. If you squint, you can kind of see this as the last step before databases become literal AI.
All of these have really broad scope. If we could solve even half of these open research problems the resulting database engines would be unrecognizable. There are ton of other narrower interesting research problems around data layouts, compression, join parallelism, etc that still have potential for substantial improvement.
It is a great time to be doing database research, we've barely scratched the surface.
To be honest none of these problems feels remotely "solvable" beyond having a ton of knobs and engineering tradeoffs like we already have now.
In a sense they don't seem very exciting because all of this required generality.
There's been a lot of fun, practical work recently on optimal join algorithms! (Full disclosure, this advisor on this paper is one of Andy and my former students, so I'm slightly biased)
RPT: https://people.iiis.tsinghua.edu.cn/~huanchen/publications/r...
Like most academic work, this one builds on some work that's been done over the last few years on ways to make the Yannakakis algorithm actually practical.
There is research as of last year which brings unoptimised impls of those join algorithms to “on-par” with optimized non-optimal industry impls.
There’s still research to be done in optimizing these algorithms in both theory and practice.
how do you optimally balance code generation and virtual function calling patterns for the best performance? Is there a standardized OLTP format on object storage? How do you performantly implement differential privacy policies within the db engine?
Are there re-usable query primitives for extremely large scale multi-modal data? how do you scale such queries or make them efficient?
The list goes on.
I personally built quite several specialized hyper-performant DB engines, and studied most popular OSS projects, and believe most DB questions are theoretically answered long ago(decade back). The puzzle is mostly to assemble pieces together to fit specific tradeoff of performance/simplicity/functionality and not overengineer system.
What is the best resource for finding good academic papers on databases that are actually practical to implement?
I prefer to not study databases through academic papers, I think major OSS projects(PG, Cassandra, RockDb, duckdb, ClickHouse, etc) have all/most important data structures and algorithms implemented and described way better.
I'd ascribe "deep tech" to anything that you can reasonably get a PhD in and have it not be unusual. There are dozens of academic conferences on DBs pushing the frontier forward.
Don't feed the troll.
The dude doesn't know anything about databases.
I know that many practical databases already exist and where I come from refining or improving something that already exists hardly qualifies as "deep tech". Is PepsiCo doing deep tech research on Crystal Pepsi?
However, the commenter I was responding to apparently considers just about _any_ research in any significant field to be deep tech so w/e I give up and I'll go update my CV to include the phrase "deep tech research"
I'm very excited to keep pushing the frontiers of deep tech art history, just as soon as I complete my $150M Series B raise.
They use cyclotrons with some regularity and advanced imaging techniques.
Maybe only an OS can be close to how MUCH deep you can go with a DB engine.
And I bet DB engines can go far than normal OS.
You don't know how much is still waiting for somebody to try, and how much is not applied. And how many of that DBs are not even doing, because all are constrained be being "apps" with so poor interface (sql).
Fun fact: Not exist a viable true relational DBs implemented, neither exist one with a viable programming language AND apis that is for developers.
ZERO.
At the risk of arguing about semantics (and not interesting database semantics) it seems half the commenters here think "deep tech" just means "like, technology you go really super DEEP into, like really dig into that DEEP DEEP HARDCORE TECHNOLOGY"
but that's not what this term has historically meant https://en.wikipedia.org/wiki/Deep_tech
> Deep tech innovations are often radical and may create new markets or disrupt existing ones. Deep tech companies often address big societal and environmental challenges and have potential to impact everyday life. Silicon chips are an example of innovation that enabled calculation at previously unimaginable speed and scale.
Database research is good, important, critical, even! But it's not creating something new that has never existed before. It's not inventing the transistor or the integrated circuit.
>> It's not inventing the transistor or the integrated circuit.
Integrated circuit on its own isn't doing anything unless used for something. Better chips enable more use-cases, more electronics etc etc. Databases in similar way are enabling more applications for users. None of the apps or services we reliably use everyday at the scale (e.g. Google Search, Online shopping, Messaging) would not exist without solid storage software.
> Integrated circuit on its own isn't doing anything unless used for something
YES! YES EXACTLY!
That's part of what makes things like ICs "deep tech" - they are transformative innovations that have no immediate application so they need government or a very well-funded far-thinking private org (e.g. Bell Labs) to bring them into being. They are not like making a new, better database.
You have just reiterated my point very concisely, so thank you.
I don't think I did reiterate your point. You were saying unlike ICs, databases -- "But it's not creating something new that has never existed before".
The point I was making was like newer/better ICs pave way for new electronics/use-cases; so do better databases/storage software pave way for more new applications that wouldn't have existed before.
Exactly: Making new/better ICs is not "deep tech" research.
Creating the first IC, creating the concept of an integrated circuit is deep tech research.
I guess I get what you are saying. Creating the first of its kind is "deep tech".
I really don't care about the definition or semantics here personally. I think creating first one is great, but then depending what it is creating 10th generation can also be quite a breakthrough. First IC had like 10 transistors, modern ones do billions and getting there probably require way more work, ideas and research than the original IC did. Things just don't improve in silos.
I'm not sure what you think "research lab" means? As I mentioned in the article, we look at IBM Research (Almaden) and Microsoft Research as inspiration.
Looking forward to the database analogues of IBM Watson and Tay.ai
I'm partly kidding; there's plenty of space for innovation in databases. You should sponsor https://sled.rs/
A cobbler should stick to his last.
Many database companies have research labs. It allows you to explore ideas, unorthodox concepts, and parts of the design space that may never be reflected in an actual product. Or to figure out how to solve specific hard problems that you come across with neither a good solution nor proof of impossibility in literature.
This is essential for a company that wants to stay on the frontier of database tech.
Which database companies have research labs? AFAIK amazon, snowflake, databricks, google and SAP don’t have a research lab dedicated to databases. They have some people that are paid to do research but there does not seem to be an IBM Almaden anywhere in the world.
There is MongoDB Labs FWIW (also Oracle Labs but I think that's just rebranded Sun Labs and not sure how much DB research happens there).
One example of what they do there is GraalVM, which started as MaximeVM at Sun Labs, although they don't share code per se only the research.
Or how Solaris eventually became the first UNIX with hardware memory tagging.
I see a lot of concurrency papers from Oracle Labs but not DB papers FWIW.
Secret sauce.
Contrary to most beliefs, there are still features on Oracle, SQL Server, DB 2, and similar, not yet available on Postgres.
Not having DB papers is certainly one way for Oracle to keep it that way.
> ClickHouse had features that at the time were only found in a handful of closed-source, commercial analytical DBMSs. For example, ClickHouse was written in C++ and supported vectorized query execution using SIMD in 2016. Most prominent open-source analytical DBMSs in 2016 were JVM-based and did not support SIMD optimizations until years later.
Performance is a feature. "Written in C++" is a strange idea of a feature.
zero cost abstraction is one of the many features of C++, its just you are not familiar with the language
I am familiar with C++.