Creator of Apache CouchDB here. Fireproof is a very clever system for building secure, collaborative apps. Your apps can work offline and when connected automatically sync changes to local storage and display them immediately to the UI. It has much of that same CouchDB magic with almost no complexity or headache on the backend.
User of Couchdb for similar use case...been keeping up to date on a parallel branch of my solution using fireproof instead of couch. Still early, but great potential...stumbled onto fireproof because I was about to write an adapter for couch (well Pouch actually, but same same) to replicate in a similar manner to fireproof.
Thanks for reading — Fireproof creator here, happy to answer any questions.
We are in-flight on our cloud launch, so consider it a preview of the experience we are building. We’ll soon be shipping more complete authorization with UCAN capability delegation, and we are working on mature key rotation. I can't wait to hear what people want to build with it.
I wrote a demo with Firebase to sync a bunch of music data between N web clients. EZ-PZ. All of the integrity stuff was icing on the cake, but basically it was an ultra-simple way to replicate data btwn web app users, without worrying about the server side at all.
The website and example, and usage looks clean. Kudos! I have some questions around what's happening under the hood that werent evident from an initial read of both your website as well as GitHub.
1. Does subscribe listen for new changes on a transient server(just a queue). Or from a more persistent store?
2. Where do the events persist? I didn't see a connector to postgres. I did see one for s3.
3. What is the default persistence layer you are advocating?
4. Let's say you run 3 instances of the self hosted server. And a random one of them gets a teacher. And 2 random other students gets load balanced to two other servers. How does the teacher get all messages? What's the thread in a distributed setting
5. How do you filter only messages. Eg: only since time T.
6. Pagination / limits to avoid any avalanche?
7. Auth? Custom auth/jwt?
8. REST API to produce?
9. Are consumers restricted to browsers? What about one in node?
10. BONUS: Have you tested if this works embedded as an iframe or embedded in an native/react native mobile app?
I answered 1-5 in my other reply, hit save and so here's more:
6. There is a limit parameter on the query API, and the underlying data structures utilize async iterator patterns so we have a go-forward path to a streaming (data / query larger than memory) implementation. But for now the decrypt implementation is eager instead of lazy, so that's the first place we'd want to focus to make data > memory workloads no problem.
7. Other embedded databases don't have auth, but we are network-aware so it's a different ballgame. Our next step is read/write access control on a per-ledger basis.
UCAN capability delegation allows us to keep an embedded mindset here, in that authorization becomes a matter of data validity, not something that has to be fetched from a centralized resources. How it works: client device agents generate non-extractable keypairs (like Passkeys) and can link them to account principals via any signing endpoint Fireproof trusts (for starters just the one we run, to an end user it looks like clicking a validation link in an email.) Agents create a new cloud database clock register by locally generating an ephemeral keypair that signs itself over to the principal. Our centralized clock register endpoint only allows updates to the resource identified by the clock's public key ID, from agents which have a valid signed delegation chain to the ephemeral key.
To a developer it will look something like `db.share("bob@example.com")` and now Fireproof Cloud will let Bob read and/or write the db also.
What's cool about this is that access control changes are just data manipulations, so they can happen offline. And the valid delegations can be safely delivered over any channel. In fact there are no secrets in this system except for the non-extractable keypairs.
If you are thinking to yourself "what about revocation?" -- we are hiring.
8. The sync endpoint has the minimal blob k/v (no list) and register APIs. And can all be floated on top of any raw kv with check-and-set semantics if needed.
We have plans for a REST API in Fireproof Cloud, where if you allow the cloud to decrypt and process your data, we can give you raw queries instead of you replicating and then querying locally. I am thinking a CSV output here would be a good place to start.
9. Runs great anywhere JS runs. We have examples (like CatBot linked above) that subscribe to the ledger on the backend and operate locally, often responding the user events. So the DB is acting as an RPC bus... this is a common pattern in CouchDB so I made sure Fireproof works great like that.
To run in an edge function, you usually aren't gonna replicate to local filesystem, instead you can configure the database to read and write directly with the cloud store. Because of the eager decrypt we do, this is actually pretty fast and not that chatty.
10. The CodePen demo on our homepage is an iframe, works great. We have a contributor (I think I see in the thread here) who is working on React Native -- most of the heavy lift is done, but our gateway interface is only now settling down to where it makes sense to finalize the integration. I have also done Socket Supply for mobile and that works great.
These are awesome questions, I'll try to fold the answers into the docs also.
1. The embedded database subscribes to the remote sync endpoint when it is connected. This subscription might be polling, websocket, or anything else. The local embedded database will try to keep up with changes anyone pushes to the remote endpoint. This is more a backend mechanical thing than an API you'll see.
Your code can subscribe to the local database -- this is a JavaScript event loop, and any updates, local or remote, will cause your callback to run. The upshot is all you have to do is connect your database to the sync endpoint and it will stay up to date, and you can also connect your UI to the database via `db.subscribe()`
2. Updates are written to local storage (indexed db or the filesystem) as encrypted blobs. These are then replicated to the cloud (without being parsed by the cloud). We have SQL connectors also, but we haven't done the Postgres specific stuff (just started designing it). That is the data side. There is also the clock register, which the client updates to point to the most recent blob. This register is multi-writer safe, and can occasionally point to more than one "head" blob, in which case the client does the deterministic merge on read.
3. In my experience most people use the defaults, so we have Fireproof Cloud which uses R2 and durable objects. We also have a SAM template for AWS, and a connector for Netlify, in addition things that are more like parts for building your own backend (file and http endpoints).
4. Each ledger replicates 100% when it syncs, so all hosts have the same data (no sharding within a ledger.) Typically you have one centralized endpoint to sync via. (p2p is possible but you'd end up contributing some plumbing to the project I bet). So in this case the class would have a URL that is the sync point, and everyone would pull from it periodically or via streaming.
Merges are idempotent, deterministic, associative, and commutative, so it doesn't matter what order the teacher and students apply updates to their local instance, once all updates are applied, they have the same state.
5. The e2e encryption means you'd have to give the keys to the server to allow it to create subsets for sync, so we haven't done that yet. Our next optimization is to sync the readonly current dataset first, then any extra data needed for writing, and only when necessary, the historical log. This still doesn't solve the subset sync issue, but will benefit all use cases immediately.
wrt (3) Being able to self-host is extremely important. I noticed a lot of focus on the docs on the Quickstart/client usage. But things like default storage engine as a ENV, path for storage as an ENV. These are very important.
hmm. Replicate state to all clients. Ok.
Seems like an opinionated but well thought through project. Godspeed!
Thanks, and thanks for the encouragement to fully document the gateway interface. We have been flux-ing it lately but as soon as it settles down we’ll do that.
The vision is many small ledgers, so the full replication per ledger makes sense, but we have work to do on cross-ledger queries
Fireproof brings really solid DX with a passionate team supporting it. It's deceptively powerful as an embedded db, but delivers the kind of experience you'd typically expect from a good frontend stack. Since modern web development keeps pushing the limits of browsers, I love local-first projects like this.
Fireproof engineer here, I come from the backend engineer perspective, so I thought I'd share some of my personal hacks on Fireproof:
Fireplace - tooling to deploy Fireproof apps and sync data across your Tailscale network. Once all the computers you care about are on your tailnet, of course you want all the browsers on the tailnet to easily sync with one another.
Go Implementation - Fireproof bills itself as a realtime database that runs anywhere, and I want to make sure that includes inside your Go applications. This will allow your Go application to become a full-fledged reader/writer of the Fireproof ledger.
I'm excited to see what other people want to build and answer any questions.
Love this! I've been following and contributing to Fireproof for a while now and it's super exciting to see how far it's come - excited to use it on a project soon!
Fireproof has come a long way and If you’re looking to cut down on backend complexity while having a reliable user experience, Fireproof is worth a serious look.
I'm happy working on fireproof. The people are very nice. In terms of technology it's a new kind on block. Yes a bit of database and a bit of Blockchain with git rebase.That's why we moving from naming it database to ledger lately.
I didn't say enough about the ledger-like aspects above. Some databases have collections, some have tables, Fireproof has ledgers. These are the unit of sharing and access control, and also the unit of provable history.
Congrats on launching Fireproof! You mentioned syncing content-addressed encrypted files via S3 or Cloudflare. For developers already using existing cloud-first setups, how seamless is it to integrate Fireproof as a hybrid solution?
Fireproof embeds in the browser, so you can add it to frontend code just like any other JavaScript module. It's useful for app data even without a backend.
When you are ready to connect multiple users, that's when the backend comes in, which can be as simple as Fireproof Cloud or your existing AWS account.
One of my favorite use cases for Fireproof is storing and searching across vector embeddings. It can act as a RAG for LLMs, operating on the backend for all clients, and/or the frontend for customized or offline scenarios.
Yeah, that seems like an ideal application for Fireproof. I just built a RAG application for cannabis retail data and we could have benefited from some of the realtime support.
Nice stuff! It's extremely fast and a breeze to use with cooperative applications that are shared by multiple users in realtime. Pretty sick work on the part of J Chris Anderson and the Fireproof team.
This looks really promising, this should run in any web-like environment right? Specifically it might be interesting to build a Socket App with this (https://github.com/socketsupply/socket)
Thanks -- we got hello world with Socket, but there's some interesting open source work we'll need to do to support peers acting as servers and take full advantage of the network.
If anyone wants to join the effort, the semantics we are talking about are also aligned with WebRTC delivery, so you'd enable a bunch of p2p topologies with your contribution.
I had recently been experimenting with pouch/couchdb, and got it working locally, then came across a post on reddit about Fireproof. I really liked how it sounded - like the evolution of pouch/couch.
So I set it up on a node app - it works great locally -
import { fireproof } from '@fireproof/core'
const db = fireproof(dbname)
db.put(doc)
async read(id) {
if (id) return await db.get(id)
return await db.allDocs()
}
Then I wanted to sync the data to the cloud - just the simplest thing possible, but I got lost in all the connector options and descriptions. I tried setting up PartyKit, but got bogged down in it all and eventually went on to something else.
So it would be great if the home page included a simple demo with a connector - Amazon S3 or PartyKit - including setting up the cloud db.
Thanks, and good luck with everything - it looks amazing...
(side note: HackerNews doesn't let you format code with ``` ??)
Thanks for the feedback. That is a bridge we are crossing now. We started with open source backends for maximum coverage, and are standing up Fireproof Cloud now as the easy default. If you click "inspect data" in the demo CodePen app, it will show you a live replication in our dashboard.
We'll be adding auth and sharing controls soon, so you should be able to build basic apps with just Fireproof and an HTML host.
Have been developing something similar (local first data tooling) for a project of my own.
The one thing stopping me from swapping my incomplete implementation with this is that I am unable to find how to connect to a db I control.
In another thread you self described your backend options as "maybe too many backend implementions", but I am only seeing established cloud providers (meh) and ipfs (cool, but sluggish).
The default gateway style doesn’t have to parse the encrypted files so the endpoint could be anything like an S3 bucket and anything like a register to redirect to the most recent file.
Someone has a similar question in our discord, where I described what we’re planning with Postgres and unencrypted sync.
Creator of Apache CouchDB here. Fireproof is a very clever system for building secure, collaborative apps. Your apps can work offline and when connected automatically sync changes to local storage and display them immediately to the UI. It has much of that same CouchDB magic with almost no complexity or headache on the backend.
Very coo…errr…hot!
User of Couchdb for similar use case...been keeping up to date on a parallel branch of my solution using fireproof instead of couch. Still early, but great potential...stumbled onto fireproof because I was about to write an adapter for couch (well Pouch actually, but same same) to replicate in a similar manner to fireproof.
We are working on transparent replication to Postgres JSON which will make it a lot easier to adopt.
Thanks for pointing out the key management potential!
We have never met. But great to see you here!
Thanks for reading — Fireproof creator here, happy to answer any questions.
We are in-flight on our cloud launch, so consider it a preview of the experience we are building. We’ll soon be shipping more complete authorization with UCAN capability delegation, and we are working on mature key rotation. I can't wait to hear what people want to build with it.
I wrote a demo with Firebase to sync a bunch of music data between N web clients. EZ-PZ. All of the integrity stuff was icing on the cake, but basically it was an ultra-simple way to replicate data btwn web app users, without worrying about the server side at all.
Here's Mykle's project https://msl.mykle.com/2024/10/bloopernet-fp-a-serverless-web...
We'll be doing a drum machine bike ride in Atlanta in December. Ping me on discord if you want to join.
Here is the code sample I mentioned, example React usage (see our homepage for Vanilla JS)
The website and example, and usage looks clean. Kudos! I have some questions around what's happening under the hood that werent evident from an initial read of both your website as well as GitHub.
1. Does subscribe listen for new changes on a transient server(just a queue). Or from a more persistent store?
2. Where do the events persist? I didn't see a connector to postgres. I did see one for s3.
3. What is the default persistence layer you are advocating?
4. Let's say you run 3 instances of the self hosted server. And a random one of them gets a teacher. And 2 random other students gets load balanced to two other servers. How does the teacher get all messages? What's the thread in a distributed setting
5. How do you filter only messages. Eg: only since time T.
6. Pagination / limits to avoid any avalanche?
7. Auth? Custom auth/jwt?
8. REST API to produce?
9. Are consumers restricted to browsers? What about one in node?
10. BONUS: Have you tested if this works embedded as an iframe or embedded in an native/react native mobile app?
I answered 1-5 in my other reply, hit save and so here's more:
6. There is a limit parameter on the query API, and the underlying data structures utilize async iterator patterns so we have a go-forward path to a streaming (data / query larger than memory) implementation. But for now the decrypt implementation is eager instead of lazy, so that's the first place we'd want to focus to make data > memory workloads no problem.
7. Other embedded databases don't have auth, but we are network-aware so it's a different ballgame. Our next step is read/write access control on a per-ledger basis.
UCAN capability delegation allows us to keep an embedded mindset here, in that authorization becomes a matter of data validity, not something that has to be fetched from a centralized resources. How it works: client device agents generate non-extractable keypairs (like Passkeys) and can link them to account principals via any signing endpoint Fireproof trusts (for starters just the one we run, to an end user it looks like clicking a validation link in an email.) Agents create a new cloud database clock register by locally generating an ephemeral keypair that signs itself over to the principal. Our centralized clock register endpoint only allows updates to the resource identified by the clock's public key ID, from agents which have a valid signed delegation chain to the ephemeral key.
To a developer it will look something like `db.share("bob@example.com")` and now Fireproof Cloud will let Bob read and/or write the db also.
What's cool about this is that access control changes are just data manipulations, so they can happen offline. And the valid delegations can be safely delivered over any channel. In fact there are no secrets in this system except for the non-extractable keypairs.
If you are thinking to yourself "what about revocation?" -- we are hiring.
8. The sync endpoint has the minimal blob k/v (no list) and register APIs. And can all be floated on top of any raw kv with check-and-set semantics if needed.
We have plans for a REST API in Fireproof Cloud, where if you allow the cloud to decrypt and process your data, we can give you raw queries instead of you replicating and then querying locally. I am thinking a CSV output here would be a good place to start.
9. Runs great anywhere JS runs. We have examples (like CatBot linked above) that subscribe to the ledger on the backend and operate locally, often responding the user events. So the DB is acting as an RPC bus... this is a common pattern in CouchDB so I made sure Fireproof works great like that.
To run in an edge function, you usually aren't gonna replicate to local filesystem, instead you can configure the database to read and write directly with the cloud store. Because of the eager decrypt we do, this is actually pretty fast and not that chatty.
10. The CodePen demo on our homepage is an iframe, works great. We have a contributor (I think I see in the thread here) who is working on React Native -- most of the heavy lift is done, but our gateway interface is only now settling down to where it makes sense to finalize the integration. I have also done Socket Supply for mobile and that works great.
These are awesome questions, I'll try to fold the answers into the docs also.
1. The embedded database subscribes to the remote sync endpoint when it is connected. This subscription might be polling, websocket, or anything else. The local embedded database will try to keep up with changes anyone pushes to the remote endpoint. This is more a backend mechanical thing than an API you'll see.
Your code can subscribe to the local database -- this is a JavaScript event loop, and any updates, local or remote, will cause your callback to run. The upshot is all you have to do is connect your database to the sync endpoint and it will stay up to date, and you can also connect your UI to the database via `db.subscribe()`
2. Updates are written to local storage (indexed db or the filesystem) as encrypted blobs. These are then replicated to the cloud (without being parsed by the cloud). We have SQL connectors also, but we haven't done the Postgres specific stuff (just started designing it). That is the data side. There is also the clock register, which the client updates to point to the most recent blob. This register is multi-writer safe, and can occasionally point to more than one "head" blob, in which case the client does the deterministic merge on read.
3. In my experience most people use the defaults, so we have Fireproof Cloud which uses R2 and durable objects. We also have a SAM template for AWS, and a connector for Netlify, in addition things that are more like parts for building your own backend (file and http endpoints).
4. Each ledger replicates 100% when it syncs, so all hosts have the same data (no sharding within a ledger.) Typically you have one centralized endpoint to sync via. (p2p is possible but you'd end up contributing some plumbing to the project I bet). So in this case the class would have a URL that is the sync point, and everyone would pull from it periodically or via streaming.
Merges are idempotent, deterministic, associative, and commutative, so it doesn't matter what order the teacher and students apply updates to their local instance, once all updates are applied, they have the same state.
5. The e2e encryption means you'd have to give the keys to the server to allow it to create subsets for sync, so we haven't done that yet. Our next optimization is to sync the readonly current dataset first, then any extra data needed for writing, and only when necessary, the historical log. This still doesn't solve the subset sync issue, but will benefit all use cases immediately.
There is some cool research we might use for subset sync: https://g-trees.github.io/g_trees/
But more practical is probably to finish the Postgres backend and then build subsetting at the global (multi-ledger) dataset level.
wrt (3) Being able to self-host is extremely important. I noticed a lot of focus on the docs on the Quickstart/client usage. But things like default storage engine as a ENV, path for storage as an ENV. These are very important.
hmm. Replicate state to all clients. Ok.
Seems like an opinionated but well thought through project. Godspeed!
Thanks, and thanks for the encouragement to fully document the gateway interface. We have been flux-ing it lately but as soon as it settles down we’ll do that.
The vision is many small ledgers, so the full replication per ledger makes sense, but we have work to do on cross-ledger queries
Fireproof brings really solid DX with a passionate team supporting it. It's deceptively powerful as an embedded db, but delivers the kind of experience you'd typically expect from a good frontend stack. Since modern web development keeps pushing the limits of browsers, I love local-first projects like this.
Fireproof engineer here, I come from the backend engineer perspective, so I thought I'd share some of my personal hacks on Fireproof:
Fireplace - tooling to deploy Fireproof apps and sync data across your Tailscale network. Once all the computers you care about are on your tailnet, of course you want all the browsers on the tailnet to easily sync with one another.
Go Implementation - Fireproof bills itself as a realtime database that runs anywhere, and I want to make sure that includes inside your Go applications. This will allow your Go application to become a full-fledged reader/writer of the Fireproof ledger.
I'm excited to see what other people want to build and answer any questions.
Love this! I've been following and contributing to Fireproof for a while now and it's super exciting to see how far it's come - excited to use it on a project soon!
Fireproof has come a long way and If you’re looking to cut down on backend complexity while having a reliable user experience, Fireproof is worth a serious look.
Thanks and thanks for the early feedback last year when I was still working on the alpha version.
I'm happy working on fireproof. The people are very nice. In terms of technology it's a new kind on block. Yes a bit of database and a bit of Blockchain with git rebase.That's why we moving from naming it database to ledger lately.
I didn't say enough about the ledger-like aspects above. Some databases have collections, some have tables, Fireproof has ledgers. These are the unit of sharing and access control, and also the unit of provable history.
Congrats on launching Fireproof! You mentioned syncing content-addressed encrypted files via S3 or Cloudflare. For developers already using existing cloud-first setups, how seamless is it to integrate Fireproof as a hybrid solution?
Fireproof embeds in the browser, so you can add it to frontend code just like any other JavaScript module. It's useful for app data even without a backend.
When you are ready to connect multiple users, that's when the backend comes in, which can be as simple as Fireproof Cloud or your existing AWS account.
One of my favorite use cases for Fireproof is storing and searching across vector embeddings. It can act as a RAG for LLMs, operating on the backend for all clients, and/or the frontend for customized or offline scenarios.
Yeah, that seems like an ideal application for Fireproof. I just built a RAG application for cannabis retail data and we could have benefited from some of the realtime support.
I wrote a "smart book" project with vector search in the browser with Fireproof. Here is the vector integration: https://github.com/jchris/smart-book/blob/e2ab5d3ee94846e6ac...
Nice stuff! It's extremely fast and a breeze to use with cooperative applications that are shared by multiple users in realtime. Pretty sick work on the part of J Chris Anderson and the Fireproof team.
Here are some apps people have built, for inspiration:
* Bloopernet Drum Machine: https://news.ycombinator.com/item?id=42177005
* Slack style team chat https://firehouse-chat.jchris.partykit.dev/
* PartKit Cloudflare https://blog.partykit.io/posts/fireproof-database-connector
It has been quite inspiring watching you grow this from a concept into a fully functioning next gen DB. Cheers!
This looks really promising, this should run in any web-like environment right? Specifically it might be interesting to build a Socket App with this (https://github.com/socketsupply/socket)
Thanks -- we got hello world with Socket, but there's some interesting open source work we'll need to do to support peers acting as servers and take full advantage of the network.
https://github.com/fireproof-storage/hello-socket-fireproof
If anyone wants to join the effort, the semantics we are talking about are also aligned with WebRTC delivery, so you'd enable a bunch of p2p topologies with your contribution.
Does it support custom backends?
There are maybe too many backend implementions: https://github.com/fireproof-storage/connect/tree/main/src
And writing a custom one is super-easy (docs coming soon, for now here is the interface): https://github.com/fireproof-storage/fireproof/blob/c4462c8a...
Thing is.. Fireproof is brilliant and has a fun culture.
Are there tutorials available?
The best starter is the React Tutorial in our docs: https://use-fireproof.com/docs/react-tutorial/
We are updating this example to the latest right now, it shows how team chat can be implemented with database sharing. https://github.com/fireproof-storage/firehouse-chat
This drum machine is pretty forkable as well: https://github.com/fireproof-storage/bloopernet
You can also get started by clicking the "Edit in CodePen" button on our home page, which is just a JavaScript and HTML app.
I had recently been experimenting with pouch/couchdb, and got it working locally, then came across a post on reddit about Fireproof. I really liked how it sounded - like the evolution of pouch/couch.
So I set it up on a node app - it works great locally -
Then I wanted to sync the data to the cloud - just the simplest thing possible, but I got lost in all the connector options and descriptions. I tried setting up PartyKit, but got bogged down in it all and eventually went on to something else.So it would be great if the home page included a simple demo with a connector - Amazon S3 or PartyKit - including setting up the cloud db.
Thanks, and good luck with everything - it looks amazing...
(side note: HackerNews doesn't let you format code with ``` ??)
Thanks for the feedback. That is a bridge we are crossing now. We started with open source backends for maximum coverage, and are standing up Fireproof Cloud now as the easy default. If you click "inspect data" in the demo CodePen app, it will show you a live replication in our dashboard.
We'll be adding auth and sharing controls soon, so you should be able to build basic apps with just Fireproof and an HTML host.
Thanks, looking forward to all that - will keep an eye out.
fireproof is super cool!
built a lot over couchdb 10y ago and fireproof checks most of the pain points i had with it!
Thank you! I'm excited to see more and more people try it out as it gets more stable and easy to use.
Looks very cool.
Have been developing something similar (local first data tooling) for a project of my own.
The one thing stopping me from swapping my incomplete implementation with this is that I am unable to find how to connect to a db I control.
In another thread you self described your backend options as "maybe too many backend implementions", but I am only seeing established cloud providers (meh) and ipfs (cool, but sluggish).
https://use-fireproof.com/docs/connect
Can I have the persistent data stored in PostgreSQL on a server I am running?
Are fireproof servers involved in mediating the syncing or is it done through the client?
The default gateway style doesn’t have to parse the encrypted files so the endpoint could be anything like an S3 bucket and anything like a register to redirect to the most recent file.
Someone has a similar question in our discord, where I described what we’re planning with Postgres and unencrypted sync.
Great news! I suspect we'll be using this when we have to go back to on-prem hardware.
I'm curious what your environment looks like. We have a bunch of open source options, but standing up a new one is always getting easier.
How does this compare to Dexie js and Dexie cloud?
We focus on provable data integrity and being easy enough that new devs can find success without learning much stack.
Many alternatives start with a backend integration mindset, I started from the React hooks and worked towards the storage engine.
A real solution for real world applications!!!
[dead]