> For loads with over 20k rows/s and people pushing changes, you may need a full-time person to handle the cluster and take a look at the crazy queries people are going to write.
I think this was a benefit of DBA culture in previous eras. Not that the DBAs were specifically necessary to write good queries (often they'd need to work with application teams to guide them towards schemas/behavior that worked well) or to maintain the database (managed DB offerings obsolete a lot of this work), but because they functioned as gatekeepers and rate-limiters of what queries and schemas could exist.
In that mode, DBAs functioned a bit like a human/process version of a thin microservice wrapping database access functionality. A big benefit was that the rate of change of queries/schema changes/access patterns was controlled and had a higher probability of being reviewed and thought about by humans before it went live. This also resulted in an increased end-database-user culture of trying to make existing schemas/query patterns work before jumping straight to bespoke access patterns. That culture's not what you want as e.g. a startup or pro-rapid-big-refactors shop, but it is what you want when your DB reliability needs or query rate/dataset size are high.
I don't think it's a given that a gatekeeper team is worth the overhead and cost; that's situational. I do think that the code version of that team (aforementioned microservice that wraps DB accesses/schema changes and nothing else) is usually not worth the cost. In my experience, that pretty much always reduces reliability and free performance gains that come from using direct DB clients from user code.
Most startups can just scale your traditional separation of compute & storage here though. You’d be shocked how well duckdb against s3 scales for 99.9% of use cases
It's one thing to be able to scale database compute/storage; it's another thing to be able to partition it. It's extremely common for bad queries/access patterns to cause noisy-neighbor effects on other simultaneous accesses to the database, to the extreme of knocking the whole database over with timeouts/OOMs/etc.
Scaling out DB compute can only help with that to a (expensive) point; eventually, you end up wanting to either prevent the bad queries from being added to the system (DBA culture) or ensure that the bad query runs on database infrastructure that doesn't affect other queries. That's why partitioning DB compute (and storage: noisy-neighbor effects from a bad query at the storage layer don't require storage to be running e.g. a BookKeeper or whatever on a server; they can manifest as hot S3 keys or cloud object/block store rate limiting) is a necessary capability if your plan for dealing with a culture of "anyone can add any access pattern they want" is to scale the DB.
Iceberg addresses the hot s3 key prefixes. It places entropy in the prefix path per object so reads/writes are randomized over the bucket key prefix space.
It’s much less of an issue. Also, 99%+ of Aws customers likely don’t have enough QPS to hit the per prefix limits anyway
Iceberg & Delta Lake do partition the data. You can add as many DuckDB servers processing them as you want, each app can have as many as you need. You pay Amazon for all the I/O and let them worry about it scaling it.
I'm not saying there is no point of contention - depending upon your write patterns you have practical limits, but for reading data that is already stored you pretty much just pay more to do more and the rate of that cost is close to linear.
I'm sure there is a scale at which that pattern breaks, and I'm sure 99.9% of people reading this will never operate a system at that scale.
Most software developers now are absolutely ignorant of performance concerns. Just throw more compute at it until it works, and someone else will pay the AWS bill.
Instead we're... listen to this... we're going to take a software developer right. Just a normal developer right. We're going to make them be the database expert right. And the cloud expert. And we're going to put them on call. We're going to have them debug linux logs, and optimize our AWS costs. They'll be there for client escalation work. And big sales calls. From time to time we'll even have them do front end work.
I think a DBA/ops/infrastructure person as an imposed bottleneck is a useful capability in some environments.
But I won't follow you as far as "expecting developers to have expertise in how and where their software runs is unreasonable".
Like, yeah, it sucks that added DevOps responsibilities etc. don't come with adjusted compensation/time allocation expectations. I'm with you there.
But it's simultaneously true that a ton of "just regular developer" people are significant liabilities because they don't understand anything about the environment where their software runs. That liability manifests operationally (if someone's just running integration tests on Windows for their Java business logic changes and don't have any familiarity with e.g. the Linux, container, or cloud environments where their code runs, they're going to be useless when their code breaks in production and operations staff needs context), and it also makes them less effective when writing code--this culture of "developers should just live in business logic and not have to context-switch or fill their brains with other levels of the stack" is what leads to full table scans, lack of awareness of memory use, N+1 query hell, looping microservice dependencies, misunderstanding of what HTTP fields are set on requests that are mutated by load balancers, mistaken assumptions about how many instances of code can run and what concurrency/thread/coroutine behaviors are present, and so on. Those are very common problems, and it's incumbent on developers in every specialty to gain familiarity with how and where their code runs in order to write and maintain that code effectively.
If your code runs on Linux in Kubernetes, all of your developers should know how to read Linux system logs, check database sessions/queries issued by parts of the application, ls/grep/cat/strace/ps their way around, interpret k8s/application dashboards, check application logs both in log storage and as they're emitted from a process, exec into a container, restart pods, check deployment liveness, etc. Even if they don't have permission to do those things in production.
That was true in 2005 when they deployed their code to IIS on Windows Server/MSSQL, too--just with different operational specifics.
That's a low bar that's often unmet, and all sorts of teams suffer from that failure. Those skills can be trained, kept up to date, and hired for; I don't think there's a great excuse for not expecting them.
> But I won't follow you as far as "expecting developers to have expertise in how and where their software runs is unreasonable".
In my professional experience, management tends to treat this as a dichotomy rather than a range. And its lead to situations where we're choosing to route every decision through a small number of 'experts' and expecting that to scale, or have people make very bad decisions about things like indices, data layout, etc.
I align with you on this. If you are using some sort of system, there is a sort-of minimum bar of expertise one should acquire. When using analytical platforms, having a basic understanding of how data is stored and the levers it gives you to optimize data processing should be standard.
Some examples:
MySQL - I would expect developers to be able to come up with a list of columns to index and why, understand why UUIDv4 is a bad choice for primary keys, know how to write performant SQL by hand and understand when their ORM is going to do the wrong thing, etc.
Redshift - Understand dist and sort keys and have some mental models on how data shuffles and is pruned based on each choice.
Iceberg + Spark/Trino - Understand that data tends to be stored/loaded as batch files. Understand how deletes work. Understand partitions and sort as options. Understand generally how bloom filters work and when you should decide to place a bloom filter. Understand why UUIDv7 is better than UUIDv4 when its part of a partition key.
Apologies my comment was very tongue and cheek. I understand what you're getting at with regards to dedicated DB style teams being a potential bottleneck and not effective for every organization, as for example most orgs get by fine with a simple RDS instance with minimal management and optimization.
What I'm calling out practically is a de-specialization in roles and the trade-offs it imposes.
For example, the "jack of all trades master of one" that was already propagating under the guise of "Full stack engineer" or "DevOps" is further exasperated when those consolidated roles are also delegating critical knowledge acquisition to LLMs.
Further, as we consolidate roles and delegate knowledge acquisition to llms, we're also paying people less, giving less job security, lowering the general quality of life not just for developers, but for operations staff and users, and generally promoting total consolidation within big tech.
No apology needed; I understand what you're getting at, and I broadly agree. It's a spectrum between "extremely easy-to-hire people that operate in such a narrow niche that they're an operational liability with limited capabilities" and "expect everyone to be an expert at every level of the stack". The right point on that spectrum is different depending on context, but I do think that a majority of software shops would be better served by moving their required skillset more towards the generalist end of that spectrum, because the default is often far too niche (driven by poor tradeoffs and short-termism in service of growth/hiring, usually).
I wanna re-emphasize that this is not a new problem. It's not because of DevOps culture or cloud complexity or scale or whatever. Very limited-specialty people were always operational liabilities and had limited positive impact on feature delivery once you accounted for the help they needed to do anything that spanned multiple levels of the stack. There are just more engineers working on more systems with tighter timeline expectations now, so it seems like the complexity incumbent on the engineering role went up in general. It didn't (it went up in some situations and down in some situations), we just started noticing operational pain more often.
I definitely do agree that there's widespread ignorance of the velocity and difficulty-of-work tradeoffs that arise from requiring a wider range of specialties from engineers, and a similarly widespread failure to adjust compensation and timeline expectations accordingly.
What's worked really well for us is not having the DBAs as gate keepers, but rather having them as guardians and experts for devs to rely on.
Our DBAs are the first line of defense when a DB starts to struggle. They crack the database open, see what's going on, and troubleshoot and sometimes fix the problems. If it's just a simple stored proc change or table change, our DBAs are empowered to go in and fix them. If there's a code change needed, they work with the devs to fix the code.
But by and large the devs still write most of the queries and tables. We as devs get DB training, but we also have a very open door policy with the DBAs so the devs can lean on their expertise and ask advice before sending stuff out. Our devs get a much better sense of "This seems like it could be expensive, let's talk to the DBAs and see what they'd suggest".
It's worked great.
I think any company would benefit from this sort of subject matter expert employment.
I usually see full-stack and DevOps roles as being a creation of the startup and early growth phases of a company. In those phases, the market and product definition is changing rapidly. Whatever you lose in reduced technical depth of these engineers, you gain in reduced communication costs, faster ability to make changes, and additional context and ability to spot efficiencies because they understand the full stack. And these companies are usually default-dead anyway, so if they screw things up and cause an outage, it's no worse than if they build the wrong product and have no users.
I've never seen a DevOps role in Big Tech (I've ping-ponged back and forth between the startup and FANG worlds a few times), and usually a full-stack engineer is someone very senior who designed a lot of the product when it was young and stuck with it for a decade. Instead there's usually a clear SRE/SWE split, and hyper-specialization, enough that someone might actually become the expert on one particular screen of a mobile app.
If you're frustrated with the de-specialization of roles, it's entirely possible you're working at companies at the wrong stage of their lifecycle. Move to a big company and the struggle is to avoid becoming pigeon-holed into one very specific task (and then laid off when the structure of the product changes so that task is no longer necessary). Such a career change can be challenging at first because your career can get specialized to "working for startups and early growth companies", and then big companies won't even look at your resume. But if you try enough places and say in the interview that you're looking for a place where you can really specialize and develop expertise, you can often find a manager desperate enough to take a chance.
LLMs are another issue. I'm worried that they're going to make the economy very unstable because the whole reason a firm exists is to capture institutional knowledge and the LLM captures that institutional knowledge in the model, where a.) it's not specific to a differentiated business and b.) it's subject to model-collapse issues. Could easily see the whole technological world collapse through lack of expertise, but if that happens it's a bigger problem that you can only fix by developing the expertise yourself.
> Such a career change can be challenging at first because your career can get specialized to "working for startups and early growth companies", and then big companies won't even look at your resume.
How would you suggest to approach such a career change successfully?
Eerily accurate how it works these days, I wish you weren't correct. I met a DBA wizard (he looked like the creator of c++) at a banking IT dept and this guy intuitively sensed what you needed and how its done.
And don't forget contract management with the supplier, L1-L2-L3 support, all in one, and integrating as the supplier is useless and your contract is shit.
Oh and you will do also business analysis with the business as claude is too complex for them (read any version of the nocode initiative).
Why are you complaining? You should be grateful of the learning opportunity not everyone gets to have. Sure, we pay you peanuts for it. But, it's ultimately for your own good. Don't mind my yacht and Ferrari, though.
That kinda how moving from programmers + sysadmins to devops looks like.
Managers went happy coz now they don't need to have hire sysadmins, while in reality they hire sysadmins, call them devops, and have them know some programming on the side.
And the "savings" from not having onprem infrastructure are burned on expensive cloud and debugging cloud blackboxes
As an aside, I was stuck when turning on the Fulham v Crystal Palace game last week to find that Fulham have ClickHouse on their shirts this year, and Palace have Temporal AI. Talk about my worlds colliding.
From the article:
> Let me clarify something: we created Tinybird ClickHouse because we wanted to build an analytics application without all the pain I'm describing. We do not offer ClickHouse® hosting; we solve the analytics problem
EDIT: I'm wrong, their home page clearly states: "Ship fast over a Managed ClickHouse®", though I don't really understand the difference between a managed clickhouse and clickhouse hosting...
It's defensive language for sure, i don't know how much it adds of protection in reality, but i nonethelesss sympathize with the author if he feels the need to protect himself that way or signaling the risk he takes.
> Tinybird is not affiliated with, associated with, or sponsored by ClickHouse, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc.
Yeah, it sucks they need to do this. If I was a visitor to their website, I'd immediately want to know what ClickHouse, Inc. is and you'd realize ---> it's managed clickhouse, direct competitor... why would I use the one that needs all the ®'s
A couple weeks ago I was scanning Github for repos with frequent commits to identify so-called "software factories" and was surprised to see clickhouse. I wouldn't touch it with a 10ft pole considering how quickly they're merging code into main. I'm talking 50+ commits per day and thousands of AI generated issues and triages. Check it out for yourself https://github.com/clickhouse/clickhouse
It's multiple records per user over time. See something like posthog. One interaction with a form can generate 10-15 events. eg enter page, click link with text "settings", change input, clicked input, etc. So active users can generate 15-150 events per session, each ~1kb to 2kb in size.
The reason to store that data is to be able to build analytics and funnels you didn't anticipate and construct them retrospectively. You can of course save massive amounts of storage by building rollups and aggregating, but that prevents you from being able to eg change the funnels and see the funnel in the past.
10M users × 100 events/day × 1KB ≈ 1TB/day. A couple years gets you into PB range.
The other obvious thing is logs. Being able to have a vulnerability then go back in time and see if you were exploited is super valuable. See eg log4shell: if you kept records, you probably could go back and see if you'd been exploited.
I'm aware; unfortunately that then leads to "must have a vendor approval" and a lot more process. If it were RDS, then it'd just be provision and done.
No, because the issue is navigating legal agreements with your vendors, not the ease of deploying. With AWS Marketplace you are making a new legal agreement with the vendor of the product, not with Amazon.
Do any two teams actually operate it in anything like the same way though?
What I saw of it, especially some years ago, was it was highly particular, and everyone had their own odd habits built around running it, ingestion, querying, everything, to the point I suspect there are a non trivial number of companies using it where it is actually the core operational expertise of the company, despite them all appearing to be in totally different domains.
Using BYOC management reduces the costs significantly. The big cost in analytic SaaS offerings is generally compute, which vendors mark up significantly. (They keep margins low on storage.)
Disclosure: My company Altinity offers BYOC management of ClickHouse.
Complex systems have complex operations and problems. I think isolating the read and write workloads and separating the read queries by departments or even users is a distinct possibility especially with technologies like DuckDB.
That is exactly where all the high scale stuff like Trino and StarRocks have gone (and ClickHouse can use S3 as storage too). However, the problem you run into is when you want to index stuff- secondary indices are tightly coupled to the query engine, so in practice you arent going to use StarRocks to write and index data to the object store, and then use ClickHouse or Trino to query it. I think it would be a useful development to decouple writes and reads, but this requires a common indexing scheme, and then once you have this standard, in principle lots more indexing plugins can be written and all compatible query engines can use them. Apache DataFusion is probably the right framework to build off of.
I don’t think it’s necessarily the use case changed but maybe the constraints. Streaming directly to Clickhouse is a great solution. Recent approaches I’ve seen though buffer data as parquet in object storage first, and read data from there. Im not sure if CH would be the preference
Where does ClickHouse fit between ElasticSearch, Pinot, TrinoDB, or just plain Spark? These are very different tools but I’m curious to know if any one has already compared them and can share some thoughts regarding their maintainability, QPS, latency, etc...
You have to make a distinction betweet query engines, datastores and batch compute.
Trino is a federated query engine and lacks secondary indices, but can do all sorts of big scale stuff like distributed merge sort and spool to disk. The typical use case is querying your data lake (say Iceberg or Hive catalog with a bunch of Parquet format files in S3, and a separate ingestion system). It is extremely mature and well understood with lots of extensions.
ClickHouse, Pinot, StarRocks, and Druid are full on OLAP datastores that can handle varying rates of ingestion (Druid is by far the slowest, the rest are fast like 100kqps writes per table is nothing fast). ClickHouse is the most used but sucks at joins and stateful data. Druid, Pinot and StarRocks can do joins and handle stateful data. In my experience Druid is the worst, Pinot is relatively immature and has minimal industry traction, and StarRocks is the most mature and has lots of traction, both in China and in ethnic Chinese analytics teams in US companies. They can all support high qps for trivial queries (thousands of simple queries in tens of ms, given enough hardware), but depending on data partitioning it can get slow quick handling a bunch of concurrent queries that are scanning the same physical servers. But people have PostGres vertically scaled to 100kqps plus and 100+TB too. In principle the use case is dashboards and charts for your real time UI; tier down to S3 with Trino for more flexible, bigger queries.
The cool thing about all these systems (the fast OLAP) systems is that they are all converging with Trino where they are moving their storage layer to object storage, which is way more flexible. No more hot storage nodes, and infinite storage. And then use Paimon or Iceberg v3 as your data lake and you get near real time stateful storage.
ElasticSearch is amazing but is really optimized for full text search and aggregations, and while it can scale to huge sizes, it does not give you joins and forces you into a very particular approach to materialized views. Also, not columnar... and nested documented dont scale well in my experience.
Spark on the other hand is just your old batch compute framework at this point. It is very flexible, and you are writing a series of SQLish transformations, but for many simpler use cases Trino is much faster and easier to use. Way bigger learning curve than just writing SQL and hoping your query engine has a good optimizer!
The relationship depends on the use case and is not linear. For web analytics and log management ClickHouse is a great replacement for ElasticSearch, for example.
I would put it closest to Pinot though pinot makes different architectural choices about storage as a difference that stands out to me. Then i’d rank trino as a second closest in only some ways. Spark the furthest for sure.
I would like to see some sources on your claim about it being Russian. It is incorporated in the US with most developers in Amsterdam.
After the invasion of Ukraine they stayed silent for a while but that was because they needed to allow there developers to get out of Russia. Many of them are Ukrainians including the CTO and founder. As soon as it was safe for there team they took a very firm stance against Russia with Ukrainian flags on the website and written statement from the team.
I am not aware of any Russian influence currently.
It was created at Yandex, the biggest techcorp in Russia, by Alexey Milovidov among others. He is still the CTO there. So I will stay away. There are many good alternatives to pick.
We’ve come at a time in which we fear exploring projects like Apache Doris because we fear that OSS might be used in malicious ways by state-backed actors.
When replaying customer queries against the next version, how do you compare results for queries using now() or approximate aggregates? Curious how you separate expected differences from actual regressions.
> A quick note about HTTP: ClickHouse® offers a TCP connector with a native protocol, but we don't use it. It does not offer many advantages for the type of application we build
This needs more elaboration. One of the major goals of running a ClickHouse cluster is to provide low-latency queries; a persistent TCP connection removes the need to re-establish a new connection for each query and thus reduces overall latency in line with CH goals. So I really didn't understand this.
> ClickHouse® open source faces a significant challenge: limited support for cloud storage. Modern OLAP databases and data systems should leverage cloud storage for cost efficiency and independent scaling of compute and storage resources. Snowflake established this standard over a decade ago, and ClickHouse® (open source) lags behind
ClickHouse writing to NVMEs is exactly how they provide their latency and performance advtanges. Writing and reading to object buckets is fundamentally slower with multiple network hops to reach what is, in this architectural context, a storage server for your storage server. If you really need far more storage, and are willing to sacrifice query latency to get it... why not architect for one of the OLAP databases, like Snowflake, where that was part of their architecture from day one?
> Because you are testing your analytics queries, right?
No? Half the point of an OLAP database is to let users write their own queries. If we knew the queries ahead of time, we probably wouldn't need an OLAP database, and instead use a less-flexible streaming architecture storing intermediate calculations so as not to need to pay for petabyte-scale storage. The expected value from paying for all of that storage is to support not knowing which queries will be written by users.
> Every single company handling ClickHouse® struggles with ingestion... Backpressure mechanism: Some people put Kafka before ClickHouse®. This does the job
The whole trade-off that you make with column-store databases like ClickHouse (instead of row-store databases like Postgres) is that inserts are slow for column stores (whereas they are fast for row stores). Inserts happen slowly, asynchronously, in the background. It is the price you pay for fast analytics queries. This is why OLAP databases have a latency lag and do not show real-time results. This is why stores like Kafka are usually a good fit, you let Kafka hold onto new data until batch insertions can catch up. If you do need real-time queries, you don't write to an OLAP directly; you write to a stateful frontend that answers the query itself, then streams out historical data from the OLAP that was successfully written there. And the first thing you do in a "I want to have my cake, and eat it too, and yes I'm willing to pay for the privilege" architecture like that is... to keep the persistent TCP connections, because that's really low-hanging fruit.
My dude, you only need to put the registered trademark symbol after the first occurrence of the mark. You® don't® need® to® put® it® after® every® mention® of® Click®House®™™®™®™®™®™.
About 20 years ago the titans of the database fields told us they wanted to put ML in the brain of databases. I think agentic databases are the future.
> For loads with over 20k rows/s and people pushing changes, you may need a full-time person to handle the cluster and take a look at the crazy queries people are going to write.
I think this was a benefit of DBA culture in previous eras. Not that the DBAs were specifically necessary to write good queries (often they'd need to work with application teams to guide them towards schemas/behavior that worked well) or to maintain the database (managed DB offerings obsolete a lot of this work), but because they functioned as gatekeepers and rate-limiters of what queries and schemas could exist.
In that mode, DBAs functioned a bit like a human/process version of a thin microservice wrapping database access functionality. A big benefit was that the rate of change of queries/schema changes/access patterns was controlled and had a higher probability of being reviewed and thought about by humans before it went live. This also resulted in an increased end-database-user culture of trying to make existing schemas/query patterns work before jumping straight to bespoke access patterns. That culture's not what you want as e.g. a startup or pro-rapid-big-refactors shop, but it is what you want when your DB reliability needs or query rate/dataset size are high.
I don't think it's a given that a gatekeeper team is worth the overhead and cost; that's situational. I do think that the code version of that team (aforementioned microservice that wraps DB accesses/schema changes and nothing else) is usually not worth the cost. In my experience, that pretty much always reduces reliability and free performance gains that come from using direct DB clients from user code.
Most startups can just scale your traditional separation of compute & storage here though. You’d be shocked how well duckdb against s3 scales for 99.9% of use cases
It's one thing to be able to scale database compute/storage; it's another thing to be able to partition it. It's extremely common for bad queries/access patterns to cause noisy-neighbor effects on other simultaneous accesses to the database, to the extreme of knocking the whole database over with timeouts/OOMs/etc.
Scaling out DB compute can only help with that to a (expensive) point; eventually, you end up wanting to either prevent the bad queries from being added to the system (DBA culture) or ensure that the bad query runs on database infrastructure that doesn't affect other queries. That's why partitioning DB compute (and storage: noisy-neighbor effects from a bad query at the storage layer don't require storage to be running e.g. a BookKeeper or whatever on a server; they can manifest as hot S3 keys or cloud object/block store rate limiting) is a necessary capability if your plan for dealing with a culture of "anyone can add any access pattern they want" is to scale the DB.
Iceberg addresses the hot s3 key prefixes. It places entropy in the prefix path per object so reads/writes are randomized over the bucket key prefix space.
It’s much less of an issue. Also, 99%+ of Aws customers likely don’t have enough QPS to hit the per prefix limits anyway
Iceberg & Delta Lake do partition the data. You can add as many DuckDB servers processing them as you want, each app can have as many as you need. You pay Amazon for all the I/O and let them worry about it scaling it.
I'm not saying there is no point of contention - depending upon your write patterns you have practical limits, but for reading data that is already stored you pretty much just pay more to do more and the rate of that cost is close to linear.
I'm sure there is a scale at which that pattern breaks, and I'm sure 99.9% of people reading this will never operate a system at that scale.
Most software developers now are absolutely ignorant of performance concerns. Just throw more compute at it until it works, and someone else will pay the AWS bill.
And later complain AWS was too expensive after they kept throwing hardware at it rather than writing an efficient query
Instead we're... listen to this... we're going to take a software developer right. Just a normal developer right. We're going to make them be the database expert right. And the cloud expert. And we're going to put them on call. We're going to have them debug linux logs, and optimize our AWS costs. They'll be there for client escalation work. And big sales calls. From time to time we'll even have them do front end work.
And get this. We pay them the exact same.
I think a DBA/ops/infrastructure person as an imposed bottleneck is a useful capability in some environments.
But I won't follow you as far as "expecting developers to have expertise in how and where their software runs is unreasonable".
Like, yeah, it sucks that added DevOps responsibilities etc. don't come with adjusted compensation/time allocation expectations. I'm with you there.
But it's simultaneously true that a ton of "just regular developer" people are significant liabilities because they don't understand anything about the environment where their software runs. That liability manifests operationally (if someone's just running integration tests on Windows for their Java business logic changes and don't have any familiarity with e.g. the Linux, container, or cloud environments where their code runs, they're going to be useless when their code breaks in production and operations staff needs context), and it also makes them less effective when writing code--this culture of "developers should just live in business logic and not have to context-switch or fill their brains with other levels of the stack" is what leads to full table scans, lack of awareness of memory use, N+1 query hell, looping microservice dependencies, misunderstanding of what HTTP fields are set on requests that are mutated by load balancers, mistaken assumptions about how many instances of code can run and what concurrency/thread/coroutine behaviors are present, and so on. Those are very common problems, and it's incumbent on developers in every specialty to gain familiarity with how and where their code runs in order to write and maintain that code effectively.
If your code runs on Linux in Kubernetes, all of your developers should know how to read Linux system logs, check database sessions/queries issued by parts of the application, ls/grep/cat/strace/ps their way around, interpret k8s/application dashboards, check application logs both in log storage and as they're emitted from a process, exec into a container, restart pods, check deployment liveness, etc. Even if they don't have permission to do those things in production.
That was true in 2005 when they deployed their code to IIS on Windows Server/MSSQL, too--just with different operational specifics.
That's a low bar that's often unmet, and all sorts of teams suffer from that failure. Those skills can be trained, kept up to date, and hired for; I don't think there's a great excuse for not expecting them.
> But I won't follow you as far as "expecting developers to have expertise in how and where their software runs is unreasonable".
In my professional experience, management tends to treat this as a dichotomy rather than a range. And its lead to situations where we're choosing to route every decision through a small number of 'experts' and expecting that to scale, or have people make very bad decisions about things like indices, data layout, etc.
I align with you on this. If you are using some sort of system, there is a sort-of minimum bar of expertise one should acquire. When using analytical platforms, having a basic understanding of how data is stored and the levers it gives you to optimize data processing should be standard.
Some examples:
MySQL - I would expect developers to be able to come up with a list of columns to index and why, understand why UUIDv4 is a bad choice for primary keys, know how to write performant SQL by hand and understand when their ORM is going to do the wrong thing, etc.
Redshift - Understand dist and sort keys and have some mental models on how data shuffles and is pruned based on each choice.
Iceberg + Spark/Trino - Understand that data tends to be stored/loaded as batch files. Understand how deletes work. Understand partitions and sort as options. Understand generally how bloom filters work and when you should decide to place a bloom filter. Understand why UUIDv7 is better than UUIDv4 when its part of a partition key.
Apologies my comment was very tongue and cheek. I understand what you're getting at with regards to dedicated DB style teams being a potential bottleneck and not effective for every organization, as for example most orgs get by fine with a simple RDS instance with minimal management and optimization.
What I'm calling out practically is a de-specialization in roles and the trade-offs it imposes.
For example, the "jack of all trades master of one" that was already propagating under the guise of "Full stack engineer" or "DevOps" is further exasperated when those consolidated roles are also delegating critical knowledge acquisition to LLMs.
Further, as we consolidate roles and delegate knowledge acquisition to llms, we're also paying people less, giving less job security, lowering the general quality of life not just for developers, but for operations staff and users, and generally promoting total consolidation within big tech.
No apology needed; I understand what you're getting at, and I broadly agree. It's a spectrum between "extremely easy-to-hire people that operate in such a narrow niche that they're an operational liability with limited capabilities" and "expect everyone to be an expert at every level of the stack". The right point on that spectrum is different depending on context, but I do think that a majority of software shops would be better served by moving their required skillset more towards the generalist end of that spectrum, because the default is often far too niche (driven by poor tradeoffs and short-termism in service of growth/hiring, usually).
I wanna re-emphasize that this is not a new problem. It's not because of DevOps culture or cloud complexity or scale or whatever. Very limited-specialty people were always operational liabilities and had limited positive impact on feature delivery once you accounted for the help they needed to do anything that spanned multiple levels of the stack. There are just more engineers working on more systems with tighter timeline expectations now, so it seems like the complexity incumbent on the engineering role went up in general. It didn't (it went up in some situations and down in some situations), we just started noticing operational pain more often.
I definitely do agree that there's widespread ignorance of the velocity and difficulty-of-work tradeoffs that arise from requiring a wider range of specialties from engineers, and a similarly widespread failure to adjust compensation and timeline expectations accordingly.
What's worked really well for us is not having the DBAs as gate keepers, but rather having them as guardians and experts for devs to rely on.
Our DBAs are the first line of defense when a DB starts to struggle. They crack the database open, see what's going on, and troubleshoot and sometimes fix the problems. If it's just a simple stored proc change or table change, our DBAs are empowered to go in and fix them. If there's a code change needed, they work with the devs to fix the code.
But by and large the devs still write most of the queries and tables. We as devs get DB training, but we also have a very open door policy with the DBAs so the devs can lean on their expertise and ask advice before sending stuff out. Our devs get a much better sense of "This seems like it could be expensive, let's talk to the DBAs and see what they'd suggest".
It's worked great.
I think any company would benefit from this sort of subject matter expert employment.
I usually see full-stack and DevOps roles as being a creation of the startup and early growth phases of a company. In those phases, the market and product definition is changing rapidly. Whatever you lose in reduced technical depth of these engineers, you gain in reduced communication costs, faster ability to make changes, and additional context and ability to spot efficiencies because they understand the full stack. And these companies are usually default-dead anyway, so if they screw things up and cause an outage, it's no worse than if they build the wrong product and have no users.
I've never seen a DevOps role in Big Tech (I've ping-ponged back and forth between the startup and FANG worlds a few times), and usually a full-stack engineer is someone very senior who designed a lot of the product when it was young and stuck with it for a decade. Instead there's usually a clear SRE/SWE split, and hyper-specialization, enough that someone might actually become the expert on one particular screen of a mobile app.
If you're frustrated with the de-specialization of roles, it's entirely possible you're working at companies at the wrong stage of their lifecycle. Move to a big company and the struggle is to avoid becoming pigeon-holed into one very specific task (and then laid off when the structure of the product changes so that task is no longer necessary). Such a career change can be challenging at first because your career can get specialized to "working for startups and early growth companies", and then big companies won't even look at your resume. But if you try enough places and say in the interview that you're looking for a place where you can really specialize and develop expertise, you can often find a manager desperate enough to take a chance.
LLMs are another issue. I'm worried that they're going to make the economy very unstable because the whole reason a firm exists is to capture institutional knowledge and the LLM captures that institutional knowledge in the model, where a.) it's not specific to a differentiated business and b.) it's subject to model-collapse issues. Could easily see the whole technological world collapse through lack of expertise, but if that happens it's a bigger problem that you can only fix by developing the expertise yourself.
> Such a career change can be challenging at first because your career can get specialized to "working for startups and early growth companies", and then big companies won't even look at your resume.
How would you suggest to approach such a career change successfully?
Eerily accurate how it works these days, I wish you weren't correct. I met a DBA wizard (he looked like the creator of c++) at a banking IT dept and this guy intuitively sensed what you needed and how its done.
And don't forget contract management with the supplier, L1-L2-L3 support, all in one, and integrating as the supplier is useless and your contract is shit.
Oh and you will do also business analysis with the business as claude is too complex for them (read any version of the nocode initiative).
> And get this. We pay them the exact same.
Why are you complaining? You should be grateful of the learning opportunity not everyone gets to have. Sure, we pay you peanuts for it. But, it's ultimately for your own good. Don't mind my yacht and Ferrari, though.
I read this in Steve Jobs voice. But maybe I was supposed to use Dr. Ian Malcolm instead?
That kinda how moving from programmers + sysadmins to devops looks like.
Managers went happy coz now they don't need to have hire sysadmins, while in reality they hire sysadmins, call them devops, and have them know some programming on the side.
And the "savings" from not having onprem infrastructure are burned on expensive cloud and debugging cloud blackboxes
A decade ago we would hire them fresh from some Ruby on Rails bootcamp so we could pay them less :shrug:
yes, that was the cloud and "devops" promise. ..or what it just another sham?
>Instead we're... listen to this... we're going to take a software developer right. Just a normal developer right.
Maybe I'm old, and I am, but I just can't get past this point with such annoying writing. Like if you actually spoke like this people would hate you.
I think the writing and the voice are both brilliant, in this case.
I have the same with the "if only there was a ..." sarcastic rebuttals. They belong 15 years ago and even then they were tiresome
hello we still exist :)
I'm going to go out on a limb here and say that having a tuned LLM would probably be able to eliminate the need for a full time DBA query analyst.
Disclaimer: I've never managed a ClickHouse cluster, let alone one of this size.
As an aside, I was stuck when turning on the Fulham v Crystal Palace game last week to find that Fulham have ClickHouse on their shirts this year, and Palace have Temporal AI. Talk about my worlds colliding.
That's a lot of ®, curious how ClickHouse® Inc. is treating the use of its name by others ... Hopes it's not like Oracle with JavaScript
They sell managed ClickHouse so I suspect it’s a precaution
From the article: > Let me clarify something: we created Tinybird ClickHouse because we wanted to build an analytics application without all the pain I'm describing. We do not offer ClickHouse® hosting; we solve the analytics problem
EDIT: I'm wrong, their home page clearly states: "Ship fast over a Managed ClickHouse®", though I don't really understand the difference between a managed clickhouse and clickhouse hosting...
It's defensive language for sure, i don't know how much it adds of protection in reality, but i nonethelesss sympathize with the author if he feels the need to protect himself that way or signaling the risk he takes.
> Tinybird is not affiliated with, associated with, or sponsored by ClickHouse, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc.
Yeah, it sucks they need to do this. If I was a visitor to their website, I'd immediately want to know what ClickHouse, Inc. is and you'd realize ---> it's managed clickhouse, direct competitor... why would I use the one that needs all the ®'s
note to author - whatever you do to add the registered symbol also is screwing up your link to https://github.com/ClickHouse%C2%AE/ClickHouse%C2%AE/issues/...
grep for "they don't like it" and click on that link
A couple weeks ago I was scanning Github for repos with frequent commits to identify so-called "software factories" and was surprised to see clickhouse. I wouldn't touch it with a 10ft pole considering how quickly they're merging code into main. I'm talking 50+ commits per day and thousands of AI generated issues and triages. Check it out for yourself https://github.com/clickhouse/clickhouse
Just wait till you find out how many commits per day to a product are occurring to most of the SaaS you rely on.
Clickhouse has a managed SaaS and it's their primary product. They have a lot of engineers. They're going to do a lot of commits.
What kind of systems are generating enough useful analytics data to ever scale to petabytes in the first place?
It's difficult (for my bird brain, at least) to imagine a scenario where such volume of analytics data would ever be necessary.
It's multiple records per user over time. See something like posthog. One interaction with a form can generate 10-15 events. eg enter page, click link with text "settings", change input, clicked input, etc. So active users can generate 15-150 events per session, each ~1kb to 2kb in size.
The reason to store that data is to be able to build analytics and funnels you didn't anticipate and construct them retrospectively. You can of course save massive amounts of storage by building rollups and aggregating, but that prevents you from being able to eg change the funnels and see the funnel in the past.
10M users × 100 events/day × 1KB ≈ 1TB/day. A couple years gets you into PB range.
The other obvious thing is logs. Being able to have a vulnerability then go back in time and see if you were exploited is super valuable. See eg log4shell: if you kept records, you probably could go back and see if you'd been exploited.
I just wish Amazon would offer it as an RDS DB; it'd make my life so much easier.
You can use actual CH Cloud on AWS?..
I'm aware; unfortunately that then leads to "must have a vendor approval" and a lot more process. If it were RDS, then it'd just be provision and done.
AWS acquired DuckLabs recently, there is a chance they will release a product competing with CH/Snowflake as a new RDS offering
Doesn't aws marketplace help with that?
No, because the issue is navigating legal agreements with your vendors, not the ease of deploying. With AWS Marketplace you are making a new legal agreement with the vendor of the product, not with Amazon.
Do any two teams actually operate it in anything like the same way though?
What I saw of it, especially some years ago, was it was highly particular, and everyone had their own odd habits built around running it, ingestion, querying, everything, to the point I suspect there are a non trivial number of companies using it where it is actually the core operational expertise of the company, despite them all appearing to be in totally different domains.
You can use ClickHouse cloud to host it on AWS.
But given the majority of use-cases of CH, AWS can be quite expensive.
Using BYOC management reduces the costs significantly. The big cost in analytic SaaS offerings is generally compute, which vendors mark up significantly. (They keep margins low on storage.)
Disclosure: My company Altinity offers BYOC management of ClickHouse.
I’m operating a terabyte-scale Clickhouse - but only because I left Langfuse running for a few months on a MacBook :)
But seriously Clickhouse does love disk space.
But on the same time it is super efficient with it compare to other solutions. Kind of efficient for application logs for example.
> "Every single company handling ClickHouse® struggles with ingestion."
Very true. Reading about "too many parts" gave me flashbacks.
(previously owned ingestion into CH at PostHog, no longer)
Scaling during that era was fun. Slack was non-stop:
:oof-1: CH needs more disk to keep up with merges.
We ran CH way too lean in those days.
ofc we meet in the ClickHouse thread
Complex systems have complex operations and problems. I think isolating the read and write workloads and separating the read queries by departments or even users is a distinct possibility especially with technologies like DuckDB.
That is exactly where all the high scale stuff like Trino and StarRocks have gone (and ClickHouse can use S3 as storage too). However, the problem you run into is when you want to index stuff- secondary indices are tightly coupled to the query engine, so in practice you arent going to use StarRocks to write and index data to the object store, and then use ClickHouse or Trino to query it. I think it would be a useful development to decouple writes and reads, but this requires a common indexing scheme, and then once you have this standard, in principle lots more indexing plugins can be written and all compatible query engines can use them. Apache DataFusion is probably the right framework to build off of.
I don’t think it’s necessarily the use case changed but maybe the constraints. Streaming directly to Clickhouse is a great solution. Recent approaches I’ve seen though buffer data as parquet in object storage first, and read data from there. Im not sure if CH would be the preference
For those interested, there's also a second part [2025]: https://www.tinybird.co/blog/what-i-learned-operating-clickh...
(note: the first part was originally published April 2025 according to the date tooltip)
Some entity spoiled the link "they don't like it" by adding (R), correct link is [1]
1. https://github.com/ClickHouse/ClickHouse/issues/49383#issuec...
Where does ClickHouse fit between ElasticSearch, Pinot, TrinoDB, or just plain Spark? These are very different tools but I’m curious to know if any one has already compared them and can share some thoughts regarding their maintainability, QPS, latency, etc...
You have to make a distinction betweet query engines, datastores and batch compute.
Trino is a federated query engine and lacks secondary indices, but can do all sorts of big scale stuff like distributed merge sort and spool to disk. The typical use case is querying your data lake (say Iceberg or Hive catalog with a bunch of Parquet format files in S3, and a separate ingestion system). It is extremely mature and well understood with lots of extensions.
ClickHouse, Pinot, StarRocks, and Druid are full on OLAP datastores that can handle varying rates of ingestion (Druid is by far the slowest, the rest are fast like 100kqps writes per table is nothing fast). ClickHouse is the most used but sucks at joins and stateful data. Druid, Pinot and StarRocks can do joins and handle stateful data. In my experience Druid is the worst, Pinot is relatively immature and has minimal industry traction, and StarRocks is the most mature and has lots of traction, both in China and in ethnic Chinese analytics teams in US companies. They can all support high qps for trivial queries (thousands of simple queries in tens of ms, given enough hardware), but depending on data partitioning it can get slow quick handling a bunch of concurrent queries that are scanning the same physical servers. But people have PostGres vertically scaled to 100kqps plus and 100+TB too. In principle the use case is dashboards and charts for your real time UI; tier down to S3 with Trino for more flexible, bigger queries.
The cool thing about all these systems (the fast OLAP) systems is that they are all converging with Trino where they are moving their storage layer to object storage, which is way more flexible. No more hot storage nodes, and infinite storage. And then use Paimon or Iceberg v3 as your data lake and you get near real time stateful storage.
ElasticSearch is amazing but is really optimized for full text search and aggregations, and while it can scale to huge sizes, it does not give you joins and forces you into a very particular approach to materialized views. Also, not columnar... and nested documented dont scale well in my experience.
Spark on the other hand is just your old batch compute framework at this point. It is very flexible, and you are writing a series of SQLish transformations, but for many simpler use cases Trino is much faster and easier to use. Way bigger learning curve than just writing SQL and hoping your query engine has a good optimizer!
The relationship depends on the use case and is not linear. For web analytics and log management ClickHouse is a great replacement for ElasticSearch, for example.
I would put it closest to Pinot though pinot makes different architectural choices about storage as a difference that stands out to me. Then i’d rank trino as a second closest in only some ways. Spark the furthest for sure.
ClickHouse I would say is more for analytics and data warehouse types of loads. So not a direct competitor to those tools except maybe for Pinot.
It is very easy to ingest data into CH. We connected exchange topics and it just worked with zero code.
But the thing with CH is that it is pretty much a Russian product so you should not use it for production anymore.
I would like to see some sources on your claim about it being Russian. It is incorporated in the US with most developers in Amsterdam.
After the invasion of Ukraine they stayed silent for a while but that was because they needed to allow there developers to get out of Russia. Many of them are Ukrainians including the CTO and founder. As soon as it was safe for there team they took a very firm stance against Russia with Ukrainian flags on the website and written statement from the team.
I am not aware of any Russian influence currently.
It was created at Yandex, the biggest techcorp in Russia, by Alexey Milovidov among others. He is still the CTO there. So I will stay away. There are many good alternatives to pick.
Did you ditch nginx for the same reason?
We’ve come at a time in which we fear exploring projects like Apache Doris because we fear that OSS might be used in malicious ways by state-backed actors.
When replaying customer queries against the next version, how do you compare results for queries using now() or approximate aggregates? Curious how you separate expected differences from actual regressions.
I cocked an eyebrow more than once reading this.
> A quick note about HTTP: ClickHouse® offers a TCP connector with a native protocol, but we don't use it. It does not offer many advantages for the type of application we build
This needs more elaboration. One of the major goals of running a ClickHouse cluster is to provide low-latency queries; a persistent TCP connection removes the need to re-establish a new connection for each query and thus reduces overall latency in line with CH goals. So I really didn't understand this.
> ClickHouse® open source faces a significant challenge: limited support for cloud storage. Modern OLAP databases and data systems should leverage cloud storage for cost efficiency and independent scaling of compute and storage resources. Snowflake established this standard over a decade ago, and ClickHouse® (open source) lags behind
ClickHouse writing to NVMEs is exactly how they provide their latency and performance advtanges. Writing and reading to object buckets is fundamentally slower with multiple network hops to reach what is, in this architectural context, a storage server for your storage server. If you really need far more storage, and are willing to sacrifice query latency to get it... why not architect for one of the OLAP databases, like Snowflake, where that was part of their architecture from day one?
> Because you are testing your analytics queries, right?
No? Half the point of an OLAP database is to let users write their own queries. If we knew the queries ahead of time, we probably wouldn't need an OLAP database, and instead use a less-flexible streaming architecture storing intermediate calculations so as not to need to pay for petabyte-scale storage. The expected value from paying for all of that storage is to support not knowing which queries will be written by users.
> Every single company handling ClickHouse® struggles with ingestion... Backpressure mechanism: Some people put Kafka before ClickHouse®. This does the job
The whole trade-off that you make with column-store databases like ClickHouse (instead of row-store databases like Postgres) is that inserts are slow for column stores (whereas they are fast for row stores). Inserts happen slowly, asynchronously, in the background. It is the price you pay for fast analytics queries. This is why OLAP databases have a latency lag and do not show real-time results. This is why stores like Kafka are usually a good fit, you let Kafka hold onto new data until batch insertions can catch up. If you do need real-time queries, you don't write to an OLAP directly; you write to a stateful frontend that answers the query itself, then streams out historical data from the OLAP that was successfully written there. And the first thing you do in a "I want to have my cake, and eat it too, and yes I'm willing to pay for the privilege" architecture like that is... to keep the persistent TCP connections, because that's really low-hanging fruit.
You can also keep persistent connections with HTTP to avoid the overhead, which is what Tinybird does. (I used to work there)
My dude, you only need to put the registered trademark symbol after the first occurrence of the mark. You® don't® need® to® put® it® after® every® mention® of® Click®House®™™®™®™®™®™.
At first when I read this, I was like "Why does The Onion's ClickHole site need so many servers?"
About 20 years ago the titans of the database fields told us they wanted to put ML in the brain of databases. I think agentic databases are the future.
Yeah that's cool but your blog won't load on my hardened Firefox:
- Uncaught TypeError: navigator.sendBeacon is not a function
- Uncaught (in promise) ReferenceError: WebAssembly is not defined
Not sure why a blog entry needs WebAssembly but I'll give it a pass.
Sometimes I think you guys run your browser like this just to make these performative comments