We recently migrated from RDS to Iceberg and the savings have been mind-blowing. That said, it's like a deconstructed database and everything you took for granted is now on the outside. Sometimes it's cool, like instead of a Cron inside the database you have a cloud Cron on the outside which is more natively monitor able. Some of it sucks like write contention on inserts so you need a write coordinator lock, and external vacuums.
However, the savings are extreme. Better cost per byte stored. Better compression. Less table bloat.
The universal compatibility has not materialised yet. We are still waiting for duckdb and click house predicate pushdown but if that appears we will be able to have specialized query engines over a generic storage layer which would be amazing. Tools like Nimtable underline what compatability unlocks.
Nim is a bad name for a language TBF. Its too short and a common sound and letter sequence (e.g. in words like "nimble", "numbus", "denim"). It is going to be used elsewhere.
Ah this looks great. Whence been doing a lot of work with Iceberg lately and a lot of the basics are still missing in the ecosystem. Like, for instance, a really easy way to explore your tables.
It’ll be interesting to see how table maintenance works out. Everyone is implementing their own. Turns out storage engines have been doing a lot of work for us for the past 50 years…
Anyway, seems like pairing this with Lakekeeper would be great.
We recently migrated from RDS to Iceberg and the savings have been mind-blowing. That said, it's like a deconstructed database and everything you took for granted is now on the outside. Sometimes it's cool, like instead of a Cron inside the database you have a cloud Cron on the outside which is more natively monitor able. Some of it sucks like write contention on inserts so you need a write coordinator lock, and external vacuums. However, the savings are extreme. Better cost per byte stored. Better compression. Less table bloat.
The universal compatibility has not materialised yet. We are still waiting for duckdb and click house predicate pushdown but if that appears we will be able to have specialized query engines over a generic storage layer which would be amazing. Tools like Nimtable underline what compatability unlocks.
Thought this was about the programming language nim.
Nim is a bad name for a language TBF. Its too short and a common sound and letter sequence (e.g. in words like "nimble", "numbus", "denim"). It is going to be used elsewhere.
I far preferred the original name.
Same. Why not 'Icetable'?
Me too!!
Ah this looks great. Whence been doing a lot of work with Iceberg lately and a lot of the basics are still missing in the ecosystem. Like, for instance, a really easy way to explore your tables.
It’ll be interesting to see how table maintenance works out. Everyone is implementing their own. Turns out storage engines have been doing a lot of work for us for the past 50 years…
Anyway, seems like pairing this with Lakekeeper would be great.
Well, what query engine do you use?
We use normal RDB tool like DBeaver against Trino JDBC and it works fine.
Amazing work! An interactive demo of how it work, before investing time to set things up would be really cool