Peter, how does this compare to Azure Durable Functions? (Say, for the sake of argument, that you are comparing the Python version of both) Are there things that fundamentally you can do in one and not in the other?
The main difference is that this is a library you can install and use in any application anywhere, while Durable Functions is (as I understand it) primarily for orchestrating serverless functions in Azure.
The Java version looks pretty danged cool paging through the some of the code.
I was trying to think of a use case for this and I was reminded of a Sun Microsystems demo (yes I'm that old) I saw. The paused a JVM and "slept" it, then kicked up on another machine almost instantly, all over the network. Was a pretty cool party trick, but then they did it when an HTTP request came in (Serverless was invented a LONG time ago!). I kinda wonder if this could be used for that?
Haha yes, one thing you can use this for is "long waits" or "long sleeps" where a program waits hours or days or weeks for a notification (potentially through server restarts, etc) then wakes up as soon as the notification arrives or a timeout is reached. More info in the docs: https://docs.dbos.dev/java/tutorials/workflow-communication
I really wish you guys would change the name since the product has moved so far away from the goals and concepts in the original publication. :). I love the product and what you are doing -- it's definitely needed and valuable.
We’ve considered it, but at this point we’re kind of stuck. We’d have to rebuild our brand from scratch. But also, a name almost never makes or breaks a company. :)
Also we have a backronym now: Durable Backends, Observable and Simple.
We are increasingly using Temporal with Temporal Cloud and soon Nexus to manage numerous workflows. I'm curious what type of observability is avialable for DBOS and how much of that you get for "free". The reason we ended up in Temporal was not that previous job-systems were unreliable, it was simply that nobody wanted to go dig through a database to find out what happened with their job, and nobody has time/energy to build a UI just for that purpose.
You can view your workflows and queues, search/filter them by any number of criteria, visualize graphs of workflow steps, cancel workflows, resume workflows, restart workflows from a specific step--everything you'd want.
Recently added DBOS python to one of my FOSS projects. It's awesome, looking forward to seeing what that self hostable observability would look like. I have been looking in the DB and OTEL logs to debug development.
glad to see the java sdk released, i've been following it for a while.
one of the rough edges i've noticed w/DBOS is for workflows that span multiple services. all of the examples are contained in a single application and thus use a single dbos 'system db' instance. if you have multiple services (as you often do in the real world) that need to participate in a workflow.. you really can't. you need to break them into multiple workflows and enqueue them in each service by creating an instance of the dbos client pointed at the other services system db. aside from the obvious overhead from fragmenting a workflow into multiple (and that you have to push to the service instead of a worker pulling the step), that means that every service needs to be aware of and have access to, every other services system db. also worth noting that sharing a single system db between services was not advised when i asked.
The pattern I would recommend in such cases is having one service be responsible for the overall workflow and then call the other services as steps.
So if you were for example running a website and wanted to have a "cancellation" flow, you'd have the cancellation service with the workflow inside of it, which would have all the steps defined, like
1) disable user account
2) mark user data as archived
3) cancel recurring payments
And then each step would call the service that actually does that work, using an idempotency key. Each service might have its own internal workflows to accomplish each task. In this case step 1 would call the accounts service, step two would call the storage service, and step three would call the payment service.
But then you have a clean reusable interface in each service, as well as a single service responsible for the workflow.
the OP wasn't clear but that's effectively what i settled on by launching workflows (within steps) via the dbos client. keeping that an implementation detail in each service though is probably better + solves the db awareness, just need to do the endpoint/rpc plumbing. thanks!
Thanks for the great feedback! Yeah, for isolation we recommend each service have its own system database and communicate via clients (so service A starts a workflow in service B by creating a client and calling "client.enqueue").
How could we make this experience better while keeping DBOS a simple library? One improvement that comes to mind is to add an "application name" field to the workflows table so that multiple applications could share a system database. Then one application could directly enqueue a workflow to another application by specifying its name, and workflow observability tooling would work cross-application.
yeah i think that's a step in the right direction, but ultimately as long as the workflow executor needs to know 'who runs this step' there will always be some friction compared to other systems like temporal.
We are a happy user of DBOS. I’ve been building out a lightweight TUI for managing our DBOS application internally, since we have workflows with tens of thousands of steps. I know the team is working on improving conductor for this use-case, but our internal TUI handles it pretty well for now. Hoping to open source it, but the code is a wreck atm. Anyways, I say all this to say, DBOS has a good client that can communicate with the instance easily, so building out a UI that fits your specific needs should be fairly simple.
Thanks Peter, but I wonder if there will there be .NET support? Since Temporal includes one, so it is a hard selling point for us .NET developers when MassTransit went commercial.
Are there any plans for supporting other databases? Our company primarily uses and has experience managing MySQL deployments. I evaluated Temporal some time ago as it seemed like a good fit for what we're building so I'm watching this closely. Thanks!
Our primary focus is Postgres. DBOS Python recently added SQLite support, we'll add this to other languages if it proves popular, but no current plans for MySQL.
That said, while DBOS requires Postgres for its own checkpoints, it can (and often is) used alongside other databases like MySQL for application data.
To build on what Peter said, we need to stay focused and make one backend solid before expanding. But architecturally, nothing prevents us from supporting more engines going forward.
It’s not clear what part of the functionality is specific to Postgres, and why. In particular in the Java world, you would expect any JDBC backend to be able to do the job.
There's no technical reason why this couldn't be done with another database, and we may add support for more in the future (DBOS Python already supports SQLite), but we're not working on it right now.
Hey! Not a DBOS Java question but stumbled on this and looking into it for the python client. Wondering what the support looks like for integration w/ gevent?
Peter, how does this compare to Azure Durable Functions? (Say, for the sake of argument, that you are comparing the Python version of both) Are there things that fundamentally you can do in one and not in the other?
The main difference is that this is a library you can install and use in any application anywhere, while Durable Functions is (as I understand it) primarily for orchestrating serverless functions in Azure.
The Java version looks pretty danged cool paging through the some of the code.
I was trying to think of a use case for this and I was reminded of a Sun Microsystems demo (yes I'm that old) I saw. The paused a JVM and "slept" it, then kicked up on another machine almost instantly, all over the network. Was a pretty cool party trick, but then they did it when an HTTP request came in (Serverless was invented a LONG time ago!). I kinda wonder if this could be used for that?
Haha yes, one thing you can use this for is "long waits" or "long sleeps" where a program waits hours or days or weeks for a notification (potentially through server restarts, etc) then wakes up as soon as the notification arrives or a timeout is reached. More info in the docs: https://docs.dbos.dev/java/tutorials/workflow-communication
I really wish you guys would change the name since the product has moved so far away from the goals and concepts in the original publication. :). I love the product and what you are doing -- it's definitely needed and valuable.
We’ve considered it, but at this point we’re kind of stuck. We’d have to rebuild our brand from scratch. But also, a name almost never makes or breaks a company. :)
Also we have a backronym now: Durable Backends, Observable and Simple.
What is the original publication?
https://dbos-project.github.io/ has all of the info
We are increasingly using Temporal with Temporal Cloud and soon Nexus to manage numerous workflows. I'm curious what type of observability is avialable for DBOS and how much of that you get for "free". The reason we ended up in Temporal was not that previous job-systems were unreliable, it was simply that nobody wanted to go dig through a database to find out what happened with their job, and nobody has time/energy to build a UI just for that purpose.
There's an observability and workflow management UI: https://docs.dbos.dev/java/tutorials/workflow-management
You can view your workflows and queues, search/filter them by any number of criteria, visualize graphs of workflow steps, cancel workflows, resume workflows, restart workflows from a specific step--everything you'd want.
Currently, this is available as a managed offering (Conductor - https://docs.dbos.dev/production/self-hosting/conductor), but we're also releasing a self-hostable version of it soon.
Recently added DBOS python to one of my FOSS projects. It's awesome, looking forward to seeing what that self hostable observability would look like. I have been looking in the DB and OTEL logs to debug development.
glad to see the java sdk released, i've been following it for a while.
one of the rough edges i've noticed w/DBOS is for workflows that span multiple services. all of the examples are contained in a single application and thus use a single dbos 'system db' instance. if you have multiple services (as you often do in the real world) that need to participate in a workflow.. you really can't. you need to break them into multiple workflows and enqueue them in each service by creating an instance of the dbos client pointed at the other services system db. aside from the obvious overhead from fragmenting a workflow into multiple (and that you have to push to the service instead of a worker pulling the step), that means that every service needs to be aware of and have access to, every other services system db. also worth noting that sharing a single system db between services was not advised when i asked.
(docs for the above: https://docs.dbos.dev/architecture#using-dbos-in-a-distribut...)
The pattern I would recommend in such cases is having one service be responsible for the overall workflow and then call the other services as steps.
So if you were for example running a website and wanted to have a "cancellation" flow, you'd have the cancellation service with the workflow inside of it, which would have all the steps defined, like
1) disable user account
2) mark user data as archived
3) cancel recurring payments
And then each step would call the service that actually does that work, using an idempotency key. Each service might have its own internal workflows to accomplish each task. In this case step 1 would call the accounts service, step two would call the storage service, and step three would call the payment service.
But then you have a clean reusable interface in each service, as well as a single service responsible for the workflow.
the OP wasn't clear but that's effectively what i settled on by launching workflows (within steps) via the dbos client. keeping that an implementation detail in each service though is probably better + solves the db awareness, just need to do the endpoint/rpc plumbing. thanks!
Thanks for the great feedback! Yeah, for isolation we recommend each service have its own system database and communicate via clients (so service A starts a workflow in service B by creating a client and calling "client.enqueue").
How could we make this experience better while keeping DBOS a simple library? One improvement that comes to mind is to add an "application name" field to the workflows table so that multiple applications could share a system database. Then one application could directly enqueue a workflow to another application by specifying its name, and workflow observability tooling would work cross-application.
yeah i think that's a step in the right direction, but ultimately as long as the workflow executor needs to know 'who runs this step' there will always be some friction compared to other systems like temporal.
I like dbos architecture much more than temporal since it is much easier to operate on small team.
But dbos doesn't have opensource release of web ui, which is most critical part for a workflow management tool.
Since competitor have all of itself opensource I don't think dbos will have a chance.
We are a happy user of DBOS. I’ve been building out a lightweight TUI for managing our DBOS application internally, since we have workflows with tens of thousands of steps. I know the team is working on improving conductor for this use-case, but our internal TUI handles it pretty well for now. Hoping to open source it, but the code is a wreck atm. Anyways, I say all this to say, DBOS has a good client that can communicate with the instance easily, so building out a UI that fits your specific needs should be fairly simple.
We'll be releasing self-hostable Conductor (the web UI) soon--stay tuned!
Thanks Peter, but I wonder if there will there be .NET support? Since Temporal includes one, so it is a hard selling point for us .NET developers when MassTransit went commercial.
.NET support is something we're considering, but aren't actively building right now.
Are there any plans for supporting other databases? Our company primarily uses and has experience managing MySQL deployments. I evaluated Temporal some time ago as it seemed like a good fit for what we're building so I'm watching this closely. Thanks!
Our primary focus is Postgres. DBOS Python recently added SQLite support, we'll add this to other languages if it proves popular, but no current plans for MySQL.
That said, while DBOS requires Postgres for its own checkpoints, it can (and often is) used alongside other databases like MySQL for application data.
To build on what Peter said, we need to stay focused and make one backend solid before expanding. But architecturally, nothing prevents us from supporting more engines going forward.
For the TypeScript version - how one would run it on Supabase Edge Functions / Deno?
Fully compatible--see this discussion on Github: https://github.com/dbos-inc/dbos-transact-ts/issues/1115
It’s not clear what part of the functionality is specific to Postgres, and why. In particular in the Java world, you would expect any JDBC backend to be able to do the job.
We actually wrote a blog post recently on why we chose Postgres! https://www.dbos.dev/blog/why-postgres-durable-execution
There's no technical reason why this couldn't be done with another database, and we may add support for more in the future (DBOS Python already supports SQLite), but we're not working on it right now.
Hey! Not a DBOS Java question but stumbled on this and looking into it for the python client. Wondering what the support looks like for integration w/ gevent?
DBOS Python works with gevent out of the box (sync DBOS uses Python threading APIs and psycopg3 that gevent monkeypatches).
Have you run into any issues using DBOS Python with gevent? Please let us know!
Upvote on this! Currently building out a project using Flask and gevent, and would love to use DBOS python with my Flask gevent project
See the sibling comment, it should work out of the box!
Looks great, shame that due to annotation-based API it's gonna be a pain to use in Clojure.
One thing we're looking at right now is what it would take to support Clojure or Kotlin.
I've been trying something similar https://github.com/seanwevans/pg_os