Hi, I'm the creator of Docker and came across this post. It's always fun to see other engineers think about how you designed a system. I haven't been involved in developing Docker in many years, but it's a fun trip down memory lane.
I'm a big fan of docker and mainly just want to say thanks for bringing it about.
I remember my first time trying to use a container was about 2015 on a beach cafe's wifi in Bocas del Toro, Panama. You can imagine how it went...
After a brief hiatus though I was back to it and since then it's been a core part of many a local dev stack and even production stack (compose) I've set up. I recall the magic moment for me being when I first realised you could mount src to use all your local native dev tools while completely virtualising everything else, including and especially whatever combo of runtimes and dependencies you need.
It was just a lightbulb moment, like it entirely solves 'works on my machine', setting new people up on a project, it just seems like how software engineering ought to work and once you have experienced it it's difficult to go back.
I'm curious, as its creator are you still using it, or as someone who can clearly look ahead to experimental ways of doing things are you on to the next thing whatever that may be? If so, what is it? :-)
Yes, I still use Docker. Although my way of using it has changed over time, like many others I assume.
My new project, Dagger, although it doesn't directly compete with Docker, does implement its own container runtime and orchestrator (for CI pipelines), and we recently added compatibility with Dockerfiles and Docker Compose files... So increasingly I find myself using Dagger instead of Docker, in the context of my CI pipelines.
On the other hand, I use docker compose on almost every personal project - like a minecraft server for my kids, home automation, AI hacks, etc. These days I just let my AI agent set it up for me, though :)
Yeah I've really been enjoying using AI to write docker/compose config and related ad hoc scripts. It was all pretty understandable anyway, but now you can just set up whatever you can think of in literally minutes, even if you just want a particular configuration as a one off. It's a great era for devops!
Thank you for inventing Docker, it truly changed the life of many developers.
I was there when dotCloud announced Docker in 2013 and it was truly pragmatic. I had previously burned my hands on LXC, Juju Charms and Vagrant to containerize the apps, but it was all a house of cards. I remember helping a friend ship his whole Vagrant Box to the interview panel for a job!
It’s incredible to think about how much boilerplate Docker removed right at it's launch. Thank you.
I was in the audience when you gave the 5-minute docker lightning talk at PyCon US Santa Clara in 2013. I remember chatting with a friend/colleague about it (https://nvie.com/about/) and we both agreed it was the most interesting lightning talk that day.
What’s funny, though, is that since it was “just” a lightning talk, it didn’t even make it into the roundup of notes I sent to my team after PyCon. But a mere year later, Docker was how my team (and seemingly everyone else) ran projects locally.
Any interesting memories from that day in 2013? From your perspective, what happened in the year that followed that caused docker to be adopted so widely? Were there other key moments or turning points?
That day is forever engraved in my memory... Off the top of my head:
I signed up for a lightning talk expecting an audience of 10-20 people, for a confidential demo of our unfinished project that we had so far only showed in 1-1 demos. Imagine my surprise when I learned Pycon does lightning talks on the main stage!
The previous speaker didn't show, so I started early. In the recording you can hear me joke that perhaps I should get his time allocation too - the MC was not amused.
I went over time and had to speedtalk through the end.. The MC once again was not amused and practically shoved me off stage :-D
We had a small sponsor booth at the even, where our small marketing team was prepared to pitch our product (DotCloud) but was thoroughly confused when a horde of people showed up after the talk, to ask about "Docker" - which as far as they knew was the founder's side project.
I drove down to Santa Clara on my newly acquired motorcycle - my very first trip on the freeway. I remember telling myself, "today is the day to take all the risks!"
Docker is probably the highest-profile software written in Go. Are you happy with that choice? Would you do it in Rust if it was available at the time, or Zug?
Hi, question from a sysadmin / network engineer. Why localhost with port forwarding? Why not native mechanism like routing and mdns? I’ve gave Docker and other container managers which do networking the same way a try a few times and I always found the network unnecessarily complicated. Eventually I came across OrbStack which uses the aforementioned mechanisms. Every container gets an IP address in the local bridge and responds to mdsns queries. This plays especially well in “compose” scenario when services on containers can all use original port numbers. And containers can be accessed with hostnames, so there won’t be conflict between port numbers especially if I run multiple composed stacks.
I definitely enjoy learning other people's opinions on things I didn't realize I had an opinion on until I learned there was something to opine about.
To me the arbitrarily complex docker networking makes perfect sense. Especially in a compose setup. I want my containers to be able to talk to each other, but I don't want them on my local network or even accessible (sometimes) from the local machine. The various internal networks offer isolation of containers from the rest of the network stack. If I have a rails app in a docker compose configuration with PG, Redis, Elasticsearch, etc, in a development environment, most of the time I don't want the service/accessory containers anywhere near anything other than the app's container. And if I want several compose stacks to work together, I just put them on the same named network. I apologize if I misunderstood your point.
I'm curious what made think of it in the first place? What was your thought process that made you think that these Linux primitives give the opportunity to make it a platform for distributing and deploying packaged applications and eliminating a lot of the issue around "it works my machine" argument?
You may have already answered this somewhere, so feel free to tell me what to search for or if you have a link to hand, that works too :)
Docker in its earlier incarnations was based on openvz, which predates lxc. The lxc project was initially a kernel development project: a collection of kernel patches to add container support. The userland tool was just a companion. When it became apparent that these patches would "win" the race to be upstreamed even though openvz was much more mature, we switched our tooling to lxc. The patches matured over time, but the userland tool (what we now know as "lxc") was finicky.
The original open-source release of Docker wrapped lxc for low-level container operations. But it was just too unreliable and we swapped it out for an in-house Go implementation called "libcontainer".
As for "commercializing": Docker was famously bad at commercializing anything, we open sourced so much (maybe too much) and still people complained that we were not open enough.
“Docker was famously bad at commercializing anything, we open sourced so much (maybe too much) and still people complained that we were not open enough”
Developers are a tough market to please, especially when your business revolves around open source software.
Unless you have an option to do private cloud hosting (like all these databases and other platforms do), it’s incredibly difficult to monetize.
In the case of docker, I think it’s in the nature of the product that it’s difficult to monetize. Best you can do is enterprise support contracts, but those aren’t good enough to justify all that VC funding.
I don't think it's necessarily that difficult to monetize, they just ignored a lot of the needs of enterprise users (rootless, not punching holes in the firewall, not fighting with systemd, private registries) for long enough that other projects (e.g. Podman, Quay) filled the void.
BTW just for giggles opened hub.docker.com with Network Tools opened:
368 requests, 9.62MB, Finish: 4.50s.
For a thing which should be a static cached copy with everything bundled. Sure, it's a progress from the previous iterations when it could hang for 30 seconds before actually loading... but this is still ridiculous.
The LXC way of getting an up per container is pretty great - I’m basic bitch with this sort of thing so I assume it’s less of an issue for the experts but really enjoyed the IP per container when I hit it.
Its nice having the flexibility for both system containers and application containers. It does feel though that most people forget that nspawn does both, and that it only can do system containers.
Agreed, I briefly forgot it can do both. Thanks for the reminder.
I went down the rabbit hole of Incus and once I realized it requires the full network aspects to be configured, then I figured using KVM, QEMU, Libvirt was more practical. Then stumbled into nspawn and it was basically ready to go.
Although, I haven’t done enough research to know if Incus provides certain useful capabilities.
I'm a huge fan of Incus. VMs, LXC, and OCI containers all under the same Docker-like CLI interface. I run it on a local server and on a dedicated server in a local DC, and connect to both remotely using Nebula.
It includes project-level resource separation, advanced networking, clustering, snapshots, and more. Plus, Stéphane Graber is super responsive on the forums and issue tracker. I bounced off of Proxmox, and can't recommend Incus more for hypervisor-type scenarios.
Do you have any experience with multi-node operation with Incus? I like the simplicity of Proxmox for moving VMs and LXCs around but I haven't settled on a solution for OCI. K3S is an option but it's a big jump. I'm moving towards Fedora CoreOS VMs at the moment for live migration and security isolation.
I remember going to a demo of Docker in South Carolina before it hit 1.0. Came back and wrote the “Docker is the Heroku Killer” article on my blog and it was the first thing I wrote that made the front page of HN.
They aren't at all the same thing, but at a high level they both reduce the amount of fragile/tedious setup-specific futzing you need to do to get a few basic services up.
I did make a bunch of money with Docker, something I am eternally grateful for.
I did not start a band, but I did build a nice house, help my family etc.
I still program everyday, working on my second startup, https://dagger.io (please forgive the plug). We have a very active discord if you want to come say hi.
I work heavily with CI/CD. From a quick glance Dagger looks very promising and I might experiment with it this weekend.
I guess my follow up question is once you’re financially independent, what continues to motivate you?
Is it fun?
Do you feel like you’re creating tools that the industry needs.
I imagine if I ever get rich I’d work on music and small games.
Video games that you buy once and get to play forever, self host, etc. With a full open source commitment within 5 years or so. I’d probably create a game studio in a lower cost of living country though.
If you do try Dagger, please join our Discord and tell us how it goes! There's a very friendly and active community of CI/CD nerds, who will be happy to help.
We actually use the same Discord server for community and private team communications. So we are literally there all the time.
Prior to the creation of Docker, did you have experience with other containerization technologies like Solaris Zones and FreeBSD Jails?
Did you at some point, either prior to or after making Docker for Linux, consider building something like Docker on top of FreeBSD Jails?
As a user of both Linux and FreeBSD, running Docker containers and Jails on each of them respectively, it would be interesting to see what FreeBSD Jails could be with something like Docker built on top of it while also taking advantage of other features on FreeBSD like fully using ZFS snapshots, ZFS send/recv, ZFS clone, vnet, and so on. Such that spinning up a container would under the hood receive ZFS snapshots from the “Docker registry” equivalent server, and it might clone the snapshot locally before starting up a container instance so that multiple container instances would be using the same locally available ZFS snapshot that was received once from the “Docker registry” equivalent.
We acquired a small team of Cambridge-based computer science & systems engineering wizards, called Unikernel. We then worked together to apply systems engineering to improving and expanding Docker in various ways.
The most successful project that we worked on together was "project Piñata" which later became Docker for Mac. It turns out bundling a Linux VM in a native Mac app, and integrating it into the host filesystem, networking stack etc requires lots of magic..
Now, most of the Unikernel team were very active OCaml developers, and created MirageOS among other ocaml projects. While working on Piñata, they simply reached for the tools they were most familiar with.
When acquiring our company, Solomon took us to a remote farm in France for a week for an offsite, fed us fantastic food and wine, and the first prototype of that Docker for Mac VM was working by the end of the week! It took a few months to hack together the shell. There's a video of Solomon and the pinãta launch party on my blog ;-) https://anil.recoil.org/notes/cacm-docker-cover#getting-invo...
There was a whole other slice of work we did for project Piñata too in the form of datakit (https://github.com/moby/datakit) that was well ahead of its time. I was looking at this archived code again just last week while thinking about agentic git workflows!
I am having a blast working on environmental science these days, and we’re long overdue a beer next in SF!
Am firing up Dagger to give it a try on our geospatial workflows for the first time just now; it’s got the same attention to CLI detail as Docker 1.0 did :-)
What made you decide to use a URI as the namespace for images? I like it, but I think it's been very misunderstood which has caused people to dislike it.
I was inspired by Go's use of URI as a namespace for user packages, and short names for the standard library. I thought it was an elegant way to allow everyone to host anywhere, and dodge a lot of complicated namespacing issues.
One design principle that was very important to me, was portability and avoiding fragmentation. I wanted every image name to have the same meaning in every engine, so that when writing a Dockerfile, you could have confidence that it would actually behave the same on every docker installation. In this we disagreed with other platforms (like Red Hat) who sought to "embrace and extend" Docker by changing the meaning of short names. We always fought this to avoid fragmentation of the platform. This did not make sense from Red Hat's perspective, since they were the platform - who cared about fragmenting Docker? That led to some negativity which is probably what you're remembering.
At the same time early Docker users wanted the ability to host their own images without being forced to use our central registry. Using Go's system solved both problems at once.
Thank you for taking the time to reply. Yes, it was the short names and the way people were trying to run local mirrors that I remember.
I personally never use the short names and have really come to like your solution. I host my own registry and I can glance at a build or compose file and immediately know if I’m depending on any external resources that I shouldn’t be.
I originally disliked it, but it’s been my favorite namespace solution for a long time now.
My only regret is that so many users had to run local docker registries on each node of their cluster, then write `FROM localhost:1234/myimage` in all their Dockerfiles - all to avoid unnecessary re-fetching of remote images. My goal was to make image distribution a native feature of the cluster, so that you wouldn't have to choose between portable Dockerfiles and proper image caching. But Kubernetes ended up dominating the cluster layer, and as far as design goes, we were not invited to that table. So as far as I know, the problem was never solved, at least universally for all Docker users.
Oh gosh, the choice of language is such a context-specific design decision. The first versions of what became Docker were written in Python. The choice of Go was a mix of nerdy excitement ("it's like the best of Python and C!"), and adoption strategy: back then devops tools were segmented by language runtime: ruby, python or Java mostly. Shipping a compiled binary allowed us to be adopted by all tribes.
None of these things can be transposed today, so I guess I can't answer...
For what it's worth, my current project (dagger) is written in Go.
The article's point that Go is a bad fit for some of the low-level interactions with the kernel, is absolutely correct though. We had to do a lot of juggling to get it to work properly. But in the end it was fine. I would say it was the correct choice of language at the time, and it certainly contributed to Docker's early success.
Another explicit goal was to facilitate contribution. We thought Go syntax was "vanilla" enough to attract a broader set of contributors than a more niche, opinionated language like Erlang, Haskell, Clojure etc. I think that also worked out well - Docker was known for attracting many contributors, many of them making their very first open source contributions. We even ran special meetups to help people become contributors, that was fun.
To give a sense of time: Docker launched in 2013, but it was one of many iterations of the same project, which I had been working on continuously since 2008. Over those 5 years, virtually nobody cared or understood what the hell I was working on. I lived in France and had no connection to the tech industry.
So, not only did I not anticipate this kind of success, I did not even have a reference frame for what success might even look like. All I knew was, I had an itch that I had to scratch, and I had nothing better to do :)
I remember in 2013 the first time someone showed me Docker [im in Mexico]. My first thought is that we already had Vagrant for setting up and running virtual machines through console.
In those years in the systems group I was working in at a fortune 500, we were very interested in this type of technology. Mostly using Solaris Zones and cgroups and trying to do this ourselves with limited results.
When Docker came out it solved so many issues we were having. Thank you for scratching that itch.
I believe Docker was always called Docker, but it grew directly out of the previous generation of the technology used by dotCloud (the company Solomon founded), which was using LXC with AUFS.
I think the breakthrough with Docker was bundling the AUFS-based image layer format with a centralized container image sharing registry. This was revolutionary at the time. But another huge element of success was Solomon and others evangelizing the technology relentlessly at meetups and conferences. I distinctly remember attending a demo in 2012 where the crowd was very excited to see the Docker workflow that we now take for granted, in large part because Solomon was so good at presenting it. At the time, I was working at a cloud PaaS company where we developed a similar technology on top of LXC and BTRFS (but no image registry). Docker really made me think hard about what we were doing, although I didn't end up making any moves.
> the previous generation of the technology used by dotCloud [...], which was using LXC with AUFS
One small note, the earliest versions (2007-2011) were based on vserver then openvz, which predated lxc. Back then you had to patch the kernel to get container primitives. And since virtualization also required patches (eg. Xen), we had to carefully combine both sets of patches, which was poorly supported because nobody else was crazy enough to stack 2 kinds of virtualization!
Were you around for the big Moby open sourcing? What went down from your perspective? If I recall, it was a fairly dramatic move that was not driven by the interests of Docker corp.
Yes, I was around for that. It was one of my skunkworks projects, as I had lost formal authority over engineering and product by then, and mostly used my "founder aura" to convince small pockets of people within the company to build cool things with me.
However Moby was a poor choice for a skunkworks project, it involved mostly communications and project management, which requires consistency and scale. Overall it was poorly executed because I failed to build enough support internally ("Solomon's at it again, let him do his thing"). Here I am trying to explain it on Twitter after the keynote, using a picture of a hand-drawn diagram, because nobody in marketing or product was assigned to help. https://x.com/solomonstre/status/855918630915133440
I think the goal of Moby was fundamentally correct: we were trying to replicate the Red Hat model, with well-scoped upstream components (linux -> runc, containerd..) integrated into a commercial product (Docker -> RHEL), with a faster-moving community project in between (Fedora -> Moby).
This was a way for us to clarify that Docker was a product, with a community of users, and Moby was an open source project, with a community of contributors. This clarity was very much needed, because different communities were projecting very different expectations on the brand "Docker", and these expectations conflicted with each other.
In a way, Moby aspired to be what CNCF ended up becoming - the "big tent" where the broader community meets to hammer out the general shape of the container stack. It seems obvious now that a single company can't be the sole authority over such a vast scope. But in the first years of Docker, we really did have that level of influence, because we literally invented the market, and brought the community together in meetups and an ever-growing Dockercon. It's only when Google and Red Hat pushed Kubernetes and CNCF, that our influence over the stack started to wane. Maybe it was for the better, I don't know if we would have been up to the task as designers and engineers. But at least there would have been a consistent design, with a Linus Torvalds figure willing and able to say "no" to stupid and redundant features. For a while I played that role - but then I was replaced by SIGs and committees.
I’d love to hear more about some limitations/tradeoffs of docker that you made at the time that you regret in hindsight. Was there anything that you wish didn’t get baked into the “default norms” for containers, or areas that would benefit from more attention and redesigns?
I’ve been following dagger and it’s been really interesting to see the evolution of the design, and new set of abstractions. There’s also nix which has seen popularity, with their approach of building layered oci containers.
There are many. But there is an alternative universe where I remained CEO instead of transitioning to CTO, and continued pursuing my product vision obsessively instead of compromising on it because of lack of self-confidence and a desire to be loved. I think I would have avoided a lot of the mistakes that the company made - but on the other hand, I would have made a fresh set of mistakes instead, so who knows what the outcome would have been!
Thanks. I needed to hear this. So many things in life can turn out differently if we can overcome our “lack of self-confidence and a desire to be loved”. It’s never too late!
Thanks for your innovation and helping developers! If Docker hasn't been your focus in recent years, what have you mainly been working on and what's been your favorite?
I like to tell people the story of how I got a pre-release peek and frankly didn't understand what I was looking at. You emailed me in reply to a comment I made around that time[0].
Funny how little has changed except the names, I guess.
I feel grateful for having seen from the inside one of those rare "hypergrowth" moments, it's a terrifying and exhausting experience but I would do it all over again without hesitation.
I also feel lucky that people are still using my work, that it endures without me, and that it generates (mostly) positive reactions. There was a time where Docker had a lot of haters, which was hard on me. But with time I've noticed that the negativity gets mostly washed away, and what remains is positive interactions like the ones in this whole thread. So now Docker is this cool experience I share with lots of people around the world, and when we meet we get to talk about this shared interest. I love it.
Python and Go are both memory-managed languages. That is perhaps the hardest part of programming (outside of getting the code to do what is intended).
Go and Rust are both compiled, but Rust requires the developer to figure out all of the memory management before compilation, which slows down development to a crawl.
Almost no company will rationally choose Rust over Go if they know how the development speed tradeoff looks like.
Because Go is a badly designed language that only took off thanks to Docker, and as the article points out, a few C workarounds were anyway required to make it possible in first place.
I would have chosen C++, but given that isn't the new darling any longer as in the 1990's, that begs the question of the new kid in town that everyone is now rewriting everything into.
Yeah but the post criticized Go for a project like Docker, so it makes sense to ask what else could have been used. At the time there weren’t many system languages fit for the purpose.
The obsession isn't with Rust, the obsession is with Go not being a good programming language and the only reason it took off was thanks to Docker.
So they would probably never rewrite from Python into C or C++, because not cool, too hard kind of motivation, otherwise they would not have picked Go.
So that begs the question, if the obession as you call it, was already a thing back then, would Rust have been the lucky candidate?
I generally agree with your comments, but do you really think that a language created by such luminaries as Ken Thompson and Rob Pike and endorsed by Google really needed Docker to "take off"?
It's not hard to understand. It's just not rational to anyone who actually calculates how performant it is (very fast, I agree) with how long it takes to write.
Once I realized how difficult it is to write complex code that actually satisfies the borrow checker, I was perfectly happy with writing somewhat slower code in Go or Python.
A while back you had this tweet which often gets cited, or at least _i_ personally think about it a lot.
> If WASM+WASI existed in 2008, we wouldn't have needed to created Docker. That's how important it is. Webassembly on the server is the future of computing. A standardized system interface was the missing link. Let's hope WASI is up to the task!
Do you think e.g. students learning docker syntax at a uni is beneficial in 2026 or you think LLM-assisted programming means that Docker will become an abstraction under an abstraction and its setup will be mostly automated by LLMs? I find myself very rarely debugging or handwriting dockerfiles nowadays!
If you're going to fully delegate something so trivial as Dockerfiles to slopmachines, why go to university at all? You don't need to learn anything if you delegate everything to the slopmachine.
This is a strange question because you could learn Dockerfile syntax in a single night. There are only 18 directives, and the average Dockerfile uses like 6 of them. 90%+ of Dockerfiles is just "run this shell command" and "copy this file into the docker image". The only time it flirts with complexity is multi-stage Dockerfiles, but 90%+ of Dockerfiles are a single stage.
> The only time it flirts with complexity is multi-stage Dockerfiles, but 90%+ of Dockerfiles are a single stage.
And multi stage builds are just a directed graph short hand for having separate intermediate dockerfiles. There’s nothing special about them either.
If I had to pick one thing that is tricky for someone new to docker to understand, it’s how the filesystem state is cached using the previous state, the env, and the raw command text.
I think the key thing for a university student is to get the big ideas down: what is a container exactly, how does it work, how does it differ from a VM, what assumptions does it make about what you're running (and how do you run something that doesn't fit that pattern), why do images exist in layers and how are layers implemented, what are the performance benefits/tradeoffs, etc. etc.
When I was learning Docker I can't tell you how many people would describe it to me by analogy to a standardized shipping container, and start going off about shipping container dimensions and supersized cargo ships and trucks and railcars... and finish by smiling beatifically, like they had just explained the whole thing in a masterstroke. Let's... uh... not do that. :-D
In a problem set, the student might learn a bit about dockerfiles or docker-compose.yml... but only, I think, so they can try building and running a container of their own!
I'm gonna be real with you. No one needs to know docker syntax. Most engineers just copy and paste chunks of Dockerfiles around anyways.
Docker syntax is one of those things that just immediately falls out of my head when I close the editor. I don't care about it at all.
Even better: there are many, many tools that build OCI images without using the Docker syntax. I haven't hand-written a Dockerfile in at least six years.
LLMs have not loved the needle here in my sphere, it was already bouncing off of the zero on the gauge.
edit: this isn't to say the syntax is good or bad (though there are several questionable parts of it) but extremely simple, and concise enough that you can just look up literally any command with great ease. It's surely a large part of its success, for better or for worse.
If you were implementing an operating system (free from compatibility concerns), what would you do differently to better support lightweight containers?
Oh wow, hi! I think my biggest “gripe” with docker is that everyone grabs it as a hammer and views many problems as a nail. Am I way off base with that? What was your original vision for how docker might be used in the wild?
What made Docker unique IMO is not the container technology itself. I used chroot 20 years ago and it worked just fine. Yes, no network namespaces; no process namespaces, but I don't even think these are that important.
What made Docker unique were there things:
1. Dockerfile concept. Building images layer by layer. Caching layers. Everything in a single file. It made possible to iterate on image quickly. It made possible to implement CI builds efficiently.
2. Docker network registry concept. Putting universal images onto a server and share them with the world.
3. Momentum. A lot of software that was traditionally complicated to deploy, suddenly appeared in docker servers. I remember how fascinated I was, when I could deploy wordpress website with tiny docker-compose. And how contained it was. I could just delete every trace of it from my host. I always felt uneasy installing various things onto my host and docker just solved that uneasiness.
Interesting, I would probably say more-or-less the exact opposite.
The main draw for me was that it made containers absolutely trivial. Sure, you could spend ages figuring out how to DIY some isolation, but in practice an awful lot of people stopped at having Apache fork its workers with a non-root uid. You want isolation and a repeatable setup? Spin up another VM and have Ansible set it up! Docker provided a batteries-included lightweight way to isolate and manage multiple applications at once.
Dockerfile? I would've been happy with a Bash script, or handing it a tarball. Caching and fast iteration? Compute is basically free. A single file? Well, there's already a whole bunch of app source code files sitting next to it, so what's a few more in a `setup/` directory? The registry? A shared base could've also been achieved with, say, a `deb` file providing the right dependencies, files, and post-install scripts.
I do agree on the momentum, though. Having every piece of software reduced from a multi-page setup guide to a single command and not risk screwing over your entire machine is absolutely magical.
The momentum, I think, matters a lot more than the other two. Makefiles already allowed you to build layered things with caching. Artifact repositories have also long been a thing.
But the argument I used for advocating Docker in environments where people were not fans of it was that the Linux container had become the de facto standard unit of deployment. There is so much tooling written under that assumption, and if you want to use any of that tooling, you need to package your deployments as Linux containers, even if you want nothing to do with Docker otherwise. That's why it had so much momentum.
Then a fourth point I think you're missing is that the concepts used by Docker are relatively simple. Before I understood what Linux containers were, I used to think things like, "Why would I care about Docker when I can isolate my processes with Linux users, cgroups, namespaces, etc." What I didn't know at that time is that "Linux users, cgroups, namespaces, etc." is exactly the thing Docker was giving a friendly UI to.
> Docker’s container runtime, runc, is written in Go.
I worked at Docker from 2014-2019. The most important part is actually written in C.
[I wrote a whole comment here about to explain how the hack works because I was too fixated on how technically incorrect "written in Go" is. Then I went back to read the rest of the article only to see that the paragraph immediately after this kind of explains it in its own way! Let that be a lesson to always finish reading the article before posting about it.]
I prefer Podman most of the time, but Docker is/was amazing and a great DX put on top of the so called "5 different linux kernel features in trenchcoat". That's the selling point, it's super lightweight and takes full advantage of the OS features that were already there (or being developed) but were awkward to use and required tons of specialized knowledge.
The first two are just kind of the nature of containers on linux, and applies to any container technology.
Podman does allow you to run it as a non-root user (with some limitations), that was in fact one of the biggest motivations for making podman. This wasn't possible at all in the early days of docker, it wasn't until linux introduced user namespaces, where the root user inside the namespace mapped to a non-root user id outside the namespace that non-root conatiners were possible.
And podman, unfortunately IMO, is also written in go.
When Docker came out, I was a heavy user of OpenVZ.
My bash build scripts looked like Dockerfiles.
Most of the container code in Linux came from those russian guys.
Took me 6 months to find out the difference between stateless and stateful containers, after Jerome Petazzoni published a blog post opposing the usage of an SSH server inside containers.
Lots of positive stuff in this thread which I won't bother repeating. Instead, I'll add a few ambivalent points:
- Today, a lot of functionality still in Docker has been upstreamed into the (Linux) kernel, so containers can be much simpler (and rootless). Docker is still carrying around some of that cruft, which is what makes it slower and heavier than Podman.
- As a devex technology, Docker/containers have been massively successful, but not as infrastructure. Containers never became secure enough to run multitenant workloads without some other form of isolation, so they ended up being redundant with KVM (or gVisor, or a similar second layer) in production a lot of the time.
That second contradiction led to lots of inefficiency. For example, on Google App Engine, for years when you ran a deploy it would create a VM just to run a single container.
Always nice to see someone demystifying a complex thing by learning the building blocks it's made from.
Knowing this, you might find rootless Podman's fork-exec method of starting containers to be more sensible. I certainly do. No root, no daemon, no socket.
If you want to learn more about the Linux Kernel namespaces which work together to make a container, this LWN series is really good: https://lwn.net/Articles/531114/
And not a single mention of the actual weird decision, a client / server model which complicates permission management by at least two orders of magnitude.
I’d argue in 2013, the client / server model is what made docker a success. It allowed for a seamless, and mostly identical, experience for docker on Mac, Windows and Linux for one. It met developers where they were, daily-ing a Mac or a Windows and deploying on Linux headless servers. It had benefit on its native Linux system too that took all the alternative solutions another 8 or 9 years for the entire Linux ecosystem to develop enough solutions around containers to make a daemon unnecessary while still offering docker DX.
Oh yes docker-machine was quite good. You can create a machine on a Mac by using VMware, VirtualBox, or using the cloud such as EC2. Then your docker CLI can work with all of these machines seamlessly without knowing any details about the machine.
That's actually a design decision made to work around the limitation of having to run as root. Since the container had to be started as a privileged user, if you want non-privileged users to be able to start contianers you either need a setuid binary or a client/server interface like docker has. There are pros and cons to each of those options.
Now it's possible to create containers without elevated privileges (assuming user namespaces are allowed), but that wasn't possible until long after docker had established the client/server model.
I spent my whole day dealing with Docker permissions today. Whats interesting and dangerous is how docker just overrides ufw rules like they do dont exist.
A lot of people start using Docker as a black box and have the mental model of a VM, and when they start digging into the internals all the "weirdness" starts to exhibit itself.
Most of them don't seem that weird to me. Except the use of go. It definitely seems strange to pick a language that can't natively call clone, one of the most important syscalls for using namespaces and implementing containers.
To solve the networking setup, typically some sort of "slirp" is needed. Slirp on one hand takes (fake or real) packets from somewhere and then on the other side pass it to a normal Linux process that does CONNECT / BIND / SEND / RECV.
It turns out you can do all the packet processing magic (connection tracking, syn handling, etc) in userspace totally fine, and then indeed hand it over this to userspace syscalls.
Being upset with classic C-based slirp and rust smoltcp (not enough features) I've dug up Fuchsia userspace tcp/ip stack - netstack - and wrapped it in slirp a-like envelope.
So this beast is basically that - on one hand you pass it packets (potentially from inside container) on the other hand it does connect/bind/etc to host OS.
This approach, in one form or another, is used by tailscale, podman and many other container runtimes.
Bocker has a drastically smaller inherent software attack surface than mainline Docker, yet its container isolation is practically non-existent, making it way easier to exploit and break out of.
I am a huge Docker fan! I would reiterate what several commenters have said already that it had a huge impact on my career. My question is relatively pedestrian compared to what others have asked already. I absolutely loved the "adjective-scientist/nerd" container names Docker generates. In fact, that is one of my most favorite things about Docker. How did that come about and who chose those combinations?
I'm glad you enjoyed that! We wanted a way to make container names reasonably unique and reasonably memorable by a human, so this was a fun way to do it. I picked the original combinations, then over time it became a ritual for contributors to propose fun new values.
There's a fun little easter egg: the combination of "Steve Wozniak" and "Boring" is explicitly forbidden, because Steve Wozniak is not boring :) https://github.com/moby/moby/pull/4902
I started hating Docker after seeing all the cargo culting on Reddit. Whenever someone asks about deploying an app on Linux, there's always at least one comment saying, Just use Docker. We finally ended up building our own deployment automation tool for our entire stack that doesn't use Docker at all.
That comment invariably comes up because for most people it’s the objectively correct and most well supported option. If you’re asking redditors how to deploy something, you’re precisely the person docker is for.
Even when I'm not using Docker to deploy something, I'm _probably_ using oci images at the least, but I'm generally running my local dev stack with Docker compose. Not always, but I use it in a lot of projects.
That part about the threading was a huge source of issues back in the early days. Goroutines could hop threads and it's the threads namespace/cgroup configuration that gets inherited when forking..
One of the gnarliest issues I recall was where network namespaces could occasionally not be cleaned up correctly. So "randomly" (but more likely under certain loads) the networking of a reused thread would get jacked and Docker could no longer pull images.
I think you're confusing userland threading (also known as green threads) with a regular thread pool implementation (that uses regular OS threads).
Because most web servers use OS threads. Nginx, Apache, litespeed, they all spin up `core count` threads but they can all work single threaded just fine as well.
No, I mean that Tokio, Go, etc., were built with general-purpose web servers in mind, and they decided that lightweight threads were necessary for that purpose because OS threads had too much overhead to be used per-request.
NGINX uses a state machine per task, which as far as I know is kind of what Rust's async functions end up desugaring to.
Though it would be better if it compared Dockers ways to say Podman, which does things a little different I think.
For others, like me:
Footgun:
A footgun is a programming or system design term for a feature, tool, or API that makes it very easy for a user or developer to shoot themselves in the foot—meaning it causes self-inflicted damage or severe errors while working exactly as intended.
Most importantly, Range doesn't need any daemon, any container runtime, any registry or any image conversion. One binary only. On a Mac it boots its own tiny Linux VM per session, and on Linux it uses KVM.
> And if you know where to look, you can even find that process in /proc.
This is very important for debugging! Not just that process, but the processes running in that Docker can be found in /proc.
Why this is useful is that you can run gdb outside of that docker and point it to a process in the docker, including setting the file system root to the docker one, so that it gdb reads the correct objects from the correct paths, to get the right symbols and addresses.
If you're old enough to remember some techniques that people used before docker (like chroot(2)-ing daemons for security), the "containers aren't real" observations are not really new info. When containers came on the scene it felt like an outgrowth or evolution of that technique, which was extremely crude and simple.
Docker question. Who prefers docker for local dev vs. run the thing directly. I usually prefer run the thing for debugging and hot reloading. Sure: build the containers and test them in CI. Dependencies like PG sure run in containers as they change less often and you save the install.
some people do all of their software development in a container. It really only works well if you are comfortable using TUIs instead of desktop apps though.
This only scratches the tip of the iceberg. People's minds are blown when they realize Linux has no concept of a container.
It's not a vm: It's all namespaces, baby! Unshare one or more of your mnt, pid, net, ipc, uts, user, cgroup, or time namespaces and you have a "container".
Oh man, what a fun thing to do! I may have to replace what used to be my fascination with programming languages and compilers with virtualization or containerization now.
This is a better way to understand it: we can think of a container as an isolated process. When deploying containers, having a clear understanding of the boundaries between containers and the host machine in terms of permissions, resources, and data is extremely helpful for troubleshooting issues.
Today, I would definitely choose rust over go. And I lament that so much container software uses go rather than rust. But back when docker started rust was a lot less mature and probably wouldn't have been a good choice.
Performance is basically irrelevant because you're not really interacting with the running application. You're running as root, so security-critical. You're talking to the internet for things like fetching images. You need rather complicated business logic to deal with all the possible options, protocols, and file formats.
Sure, you could do it in C, but dealing with dependencies is a major PITA and you're permanently one tiny mistake away from getting pwned, while not needing any of C's strengths. Go is safe, easy to write, and ships as a single batteries-included binary: what's not to like?
What's not to like? For example the fact that Go's threads and OS threads are two different things. For example the fact that I can't just use fork in Go. And many other things.
Go is good for writing web servers. For system software? Not so much. 90% of Unix is written with C. It's not that bad.
That isn't really relevant for the tool starting the container. The biggest problem with go here is that it can't natively perform the operations necessary to set up a container, due to the nature of the runtime, so it has to resort to complicated workarounds like a shim in c, and starting spawning from /proc/self/exe instead of forking/cloning.
> Unlike metal servers, VMs are just operating systems being gaslit by a hypervisor. Your actual CPU is still executing those instructions! And if you know where to look, yyou can even read the other VMs’ memory in L1D
> Go is kind of a terrible language for a container runtime
I love Go and use it for everything I can but will absolutely concede this point. It's clearly possible to write the most successful container runtimes on earth in Go, but you absolutely have to fight the language/runtime sometimes. Since containers are based on processes, handing off a container spec to a minimal rust runtime seems like what we should all be doing by now. crun has just proven good enough that AFAIK no one has RIIR.
For as "terrible of a language" Go is alleged to be for a container runtime, its not like crun or youki are exactly faster in many meaningful ways unless you count churning containers running true and exiting.
And implying the system kernel still has to run the container kinda ignores kata containers and gvisor runtimes.
It's about using go to develop container runtime being terrible and not that "if it was C it would be faster". OS runs the container, docker just does the setup and some managmenent.
Honestly it took me so long to pick up containerisation because of how it's marketed. "Orchestrate a zillion fake computers, ultralight VMs, livestock not pets" big whoop.
"It's an idempotent language for OS level dependencies so you can be sure exactly what is running on your server"
Docker hasn't required root for 7 years, I wish people would stop repeating that. It uses a root daemon by default because it makes things easier. Also, Containers are only as fake as a "Desktop" in an OS, which isn't real either. It's a user, network & process namespace with a control bus and a bunch of other weird things. A Desktop is a Container with a GUI.
Other weird facts about OSes: a filesystem is a database, the Linux kernel is a monolithic program, "hidden" dot-files were a Unix bug that got carried forward, deleting a file doesn't remove the file from your machine, virtual memory is a lie, your computer can't run multiple programs at once (it's just running one at a time real fast), in 12 years most systems will crash due to one number getting too big, you technically never need to reboot because you can launch a different version of Linux while already running Linux, you can freeze a program to a file and then later resume running that same program in the same state (but nobody uses it), files aren't actually stored in directories, there's a maximum number of files (and size of file) on most filesystems (which you can use to lock up a machine, despite file quotas existing and nobody using them), you can lock up most systems by forking too many times / using too much RAM / accessing disk too much, there's a default number of open files and/or TCP ports which is artificially low and you can use to again lock up a machine, pipes are just two file descriptors connected to one memory buffer, an IP address is just one long number split up into bytes, etc.
The rootless Docker experience used to be pretty bad and it was less friction to use something that was rootless by default (like Podman). Has this changed?
it changed and its better now.
rootless dockerd wants newuidmap/newgidmap + subuid range before it start, and cgroups only work with cgroup v2 + systemd.
also, on a host that already has those, its fine on one without systemd, and podman is still less friction, mostly because there is no daemon to babysit
One of my deeply unpopular opinions is that Docker only exists because Linux as a bad operating system that is incapable of deploying Docker. I have not once worked on a project where “we’ll solve this with Docker!” didn’t result in never ending Docker fixes and workarounds and hacks.
This is not a slight on Docker itself. It’s a slight on the conditions that induced Docker to exist.
One of the great selling points of one of my projects is “this is easy to run without needing Docker”.
>>Images are just folders that Docker puts somewhere and then calls chroot (or pivot_root) to make that folder your container process’s new root directory.
And folders are graphical representations of directories--sometimes, not always--and not directories themselves. When will the newbies get this?
The high level abstraction is more important than the low level implementation unless the low level implementation is leaking through, or there is a significant advantage conveyed by understanding the low level implementation.
I know a lot about how http works, and it comes in handy, but I can understand how juniors get away with not knowing it as fully as I do.
Hi, I'm the creator of Docker and came across this post. It's always fun to see other engineers think about how you designed a system. I haven't been involved in developing Docker in many years, but it's a fun trip down memory lane.
I'm happy to answer any questions you may have.
I'm a big fan of docker and mainly just want to say thanks for bringing it about.
I remember my first time trying to use a container was about 2015 on a beach cafe's wifi in Bocas del Toro, Panama. You can imagine how it went...
After a brief hiatus though I was back to it and since then it's been a core part of many a local dev stack and even production stack (compose) I've set up. I recall the magic moment for me being when I first realised you could mount src to use all your local native dev tools while completely virtualising everything else, including and especially whatever combo of runtimes and dependencies you need.
It was just a lightbulb moment, like it entirely solves 'works on my machine', setting new people up on a project, it just seems like how software engineering ought to work and once you have experienced it it's difficult to go back.
I'm curious, as its creator are you still using it, or as someone who can clearly look ahead to experimental ways of doing things are you on to the next thing whatever that may be? If so, what is it? :-)
Thank you for sharing that!
Yes, I still use Docker. Although my way of using it has changed over time, like many others I assume.
My new project, Dagger, although it doesn't directly compete with Docker, does implement its own container runtime and orchestrator (for CI pipelines), and we recently added compatibility with Dockerfiles and Docker Compose files... So increasingly I find myself using Dagger instead of Docker, in the context of my CI pipelines.
On the other hand, I use docker compose on almost every personal project - like a minecraft server for my kids, home automation, AI hacks, etc. These days I just let my AI agent set it up for me, though :)
I'll check out Dagger, sounds interesting.
Yeah I've really been enjoying using AI to write docker/compose config and related ad hoc scripts. It was all pretty understandable anyway, but now you can just set up whatever you can think of in literally minutes, even if you just want a particular configuration as a one off. It's a great era for devops!
> does implement its own container runtime
I take it vito talked you into adopting Garden? </injoke>
I’m curious if your background gave you experience with other operating systems, such as TandemOS?
Thank you for inventing Docker, it truly changed the life of many developers.
I was there when dotCloud announced Docker in 2013 and it was truly pragmatic. I had previously burned my hands on LXC, Juju Charms and Vagrant to containerize the apps, but it was all a house of cards. I remember helping a friend ship his whole Vagrant Box to the interview panel for a job!
It’s incredible to think about how much boilerplate Docker removed right at it's launch. Thank you.
I was in the audience when you gave the 5-minute docker lightning talk at PyCon US Santa Clara in 2013. I remember chatting with a friend/colleague about it (https://nvie.com/about/) and we both agreed it was the most interesting lightning talk that day.
What’s funny, though, is that since it was “just” a lightning talk, it didn’t even make it into the roundup of notes I sent to my team after PyCon. But a mere year later, Docker was how my team (and seemingly everyone else) ran projects locally.
Any interesting memories from that day in 2013? From your perspective, what happened in the year that followed that caused docker to be adopted so widely? Were there other key moments or turning points?
> Any interesting memories from that day in 2013?
That day is forever engraved in my memory... Off the top of my head:
I signed up for a lightning talk expecting an audience of 10-20 people, for a confidential demo of our unfinished project that we had so far only showed in 1-1 demos. Imagine my surprise when I learned Pycon does lightning talks on the main stage!
The previous speaker didn't show, so I started early. In the recording you can hear me joke that perhaps I should get his time allocation too - the MC was not amused.
I went over time and had to speedtalk through the end.. The MC once again was not amused and practically shoved me off stage :-D
We had a small sponsor booth at the even, where our small marketing team was prepared to pitch our product (DotCloud) but was thoroughly confused when a horde of people showed up after the talk, to ask about "Docker" - which as far as they knew was the founder's side project.
I drove down to Santa Clara on my newly acquired motorcycle - my very first trip on the freeway. I remember telling myself, "today is the day to take all the risks!"
How much have you ridden since that day? Any new bikes?
The Video in Question: https://www.youtube.com/watch?v=wW9CAH9nSLs
Thanks for making so many lives easier!
Docker is probably the highest-profile software written in Go. Are you happy with that choice? Would you do it in Rust if it was available at the time, or Zug?
Hi, question from a sysadmin / network engineer. Why localhost with port forwarding? Why not native mechanism like routing and mdns? I’ve gave Docker and other container managers which do networking the same way a try a few times and I always found the network unnecessarily complicated. Eventually I came across OrbStack which uses the aforementioned mechanisms. Every container gets an IP address in the local bridge and responds to mdsns queries. This plays especially well in “compose” scenario when services on containers can all use original port numbers. And containers can be accessed with hostnames, so there won’t be conflict between port numbers especially if I run multiple composed stacks.
I definitely enjoy learning other people's opinions on things I didn't realize I had an opinion on until I learned there was something to opine about.
To me the arbitrarily complex docker networking makes perfect sense. Especially in a compose setup. I want my containers to be able to talk to each other, but I don't want them on my local network or even accessible (sometimes) from the local machine. The various internal networks offer isolation of containers from the rest of the network stack. If I have a rails app in a docker compose configuration with PG, Redis, Elasticsearch, etc, in a development environment, most of the time I don't want the service/accessory containers anywhere near anything other than the app's container. And if I want several compose stacks to work together, I just put them on the same named network. I apologize if I misunderstood your point.
Thank you for docker.
Went from the good old days of vagrant (xen/kvm) to lxc/lxd to docker, and docker truly clicked all the way from development to production.
But honestly, thanks to Docker hub even more. The CLI is only one part of the puzzle, the worldwide registry is an immense undertaking.
I'm curious what made think of it in the first place? What was your thought process that made you think that these Linux primitives give the opportunity to make it a platform for distributing and deploying packaged applications and eliminating a lot of the issue around "it works my machine" argument?
You may have already answered this somewhere, so feel free to tell me what to search for or if you have a link to hand, that works too :)
Docker was basically a commercialisation of lxc and now we got nspawn going full circle.
This is incorrect.
Docker in its earlier incarnations was based on openvz, which predates lxc. The lxc project was initially a kernel development project: a collection of kernel patches to add container support. The userland tool was just a companion. When it became apparent that these patches would "win" the race to be upstreamed even though openvz was much more mature, we switched our tooling to lxc. The patches matured over time, but the userland tool (what we now know as "lxc") was finicky.
The original open-source release of Docker wrapped lxc for low-level container operations. But it was just too unreliable and we swapped it out for an in-house Go implementation called "libcontainer".
As for "commercializing": Docker was famously bad at commercializing anything, we open sourced so much (maybe too much) and still people complained that we were not open enough.
“Docker was famously bad at commercializing anything, we open sourced so much (maybe too much) and still people complained that we were not open enough”
Developers are a tough market to please, especially when your business revolves around open source software.
Unless you have an option to do private cloud hosting (like all these databases and other platforms do), it’s incredibly difficult to monetize.
In the case of docker, I think it’s in the nature of the product that it’s difficult to monetize. Best you can do is enterprise support contracts, but those aren’t good enough to justify all that VC funding.
I don't think it's necessarily that difficult to monetize, they just ignored a lot of the needs of enterprise users (rootless, not punching holes in the firewall, not fighting with systemd, private registries) for long enough that other projects (e.g. Podman, Quay) filled the void.
> Docker was famously bad at commercializing anything
That's because there was never a proper sysadmin on your team to slap some sense in your decisions.
You've got too opinionated[0], too tied on yourself[1] monster which required too much resources what you can handle alone.
And your market is not the developers but the companies where whose developers have a daytime jobs: https://news.ycombinator.com/item?id=34377674
BTW just for giggles opened hub.docker.com with Network Tools opened:
368 requests, 9.62MB, Finish: 4.50s.
For a thing which should be a static cached copy with everything bundled. Sure, it's a progress from the previous iterations when it could hang for 30 seconds before actually loading... but this is still ridiculous.
[0] https://news.ycombinator.com/item?id=46739153
[1] Eg: the syntax to point the image to a specific registry is something... bad. So bad I just abandoned running my own registry, it wasn't worth it.
PS there is still (AFAIK) no support for a transaction-like image building in Compose. So instead of something like this:
people need to write these one-liner abominations: EDIT: to add on [1]: https://news.ycombinator.com/item?id=35312087In the end, how were you able to monetize Docker?
The LXC way of getting an up per container is pretty great - I’m basic bitch with this sort of thing so I assume it’s less of an issue for the experts but really enjoyed the IP per container when I hit it.
I always liked lxc for containers. I never understood Docker. Not that I really put a lot of time into trying. These days I don't use either one.
I’m a huge fan of nspawn. Always wanted to use VMs instead of Docker and nspawn feels like a natural choice.
Its nice having the flexibility for both system containers and application containers. It does feel though that most people forget that nspawn does both, and that it only can do system containers.
Agreed, I briefly forgot it can do both. Thanks for the reminder.
I went down the rabbit hole of Incus and once I realized it requires the full network aspects to be configured, then I figured using KVM, QEMU, Libvirt was more practical. Then stumbled into nspawn and it was basically ready to go.
Although, I haven’t done enough research to know if Incus provides certain useful capabilities.
I'm a huge fan of Incus. VMs, LXC, and OCI containers all under the same Docker-like CLI interface. I run it on a local server and on a dedicated server in a local DC, and connect to both remotely using Nebula.
It includes project-level resource separation, advanced networking, clustering, snapshots, and more. Plus, Stéphane Graber is super responsive on the forums and issue tracker. I bounced off of Proxmox, and can't recommend Incus more for hypervisor-type scenarios.
Do you have any experience with multi-node operation with Incus? I like the simplicity of Proxmox for moving VMs and LXCs around but I haven't settled on a solution for OCI. K3S is an option but it's a big jump. I'm moving towards Fedora CoreOS VMs at the moment for live migration and security isolation.
Given all the popularity and use cases you know today. Is there any fundamental design choice you would have done differently?
So you are responsible for my career and the reason i can put food on my table. Thank you.
People are underestimating now big of a tech shift it was back in the day.
No that’s all you. He just made it easier.
Well said :)
I remember going to a demo of Docker in South Carolina before it hit 1.0. Came back and wrote the “Docker is the Heroku Killer” article on my blog and it was the first thing I wrote that made the front page of HN.
but heroku and docker serve different purpose no ???
They aren't at all the same thing, but at a high level they both reduce the amount of fragile/tedious setup-specific futzing you need to do to get a few basic services up.
Docker ended up as a primary way of delivering, packaging applications etc., for the web and beyond.
What was your imagined use case when you first came up with it?
Did you think Docker as it is used in the industry right now is where you imagined it would be used?
Any use cases for it that the industry missed but you think would have made things even better?
Thank you.
For one, did you get rich beyond all comprehension and did you start a band with your wealth?
Second, do you still program ?
I was lucky enough to get in this industry when Docker was new. Sure beats trying to figure out why X happens on machine A and doesn’t on machine B.
I did make a bunch of money with Docker, something I am eternally grateful for.
I did not start a band, but I did build a nice house, help my family etc.
I still program everyday, working on my second startup, https://dagger.io (please forgive the plug). We have a very active discord if you want to come say hi.
I work heavily with CI/CD. From a quick glance Dagger looks very promising and I might experiment with it this weekend.
I guess my follow up question is once you’re financially independent, what continues to motivate you?
Is it fun?
Do you feel like you’re creating tools that the industry needs.
I imagine if I ever get rich I’d work on music and small games.
Video games that you buy once and get to play forever, self host, etc. With a full open source commitment within 5 years or so. I’d probably create a game studio in a lower cost of living country though.
If you do try Dagger, please join our Discord and tell us how it goes! There's a very friendly and active community of CI/CD nerds, who will be happy to help.
We actually use the same Discord server for community and private team communications. So we are literally there all the time.
How’s dagger’s k8s and/or rust story?
We are working on making Rust builds much faster, and easier to distribute on a cluster.
When self-hosting Dagger, you can run it on Kubernetes. Beyond that, we don't use Kubernetes at all ourselves.
Commenting to be able to find your CI/CD project when back at my desk.
You can favorite messages, then check them out from your profile page just like you check on your previously posted messages.
Hit on a message's permalink (which is the "X hours ago" link), and in there you'll find the action to Favorite it.
Thank you!
Thanks for making Docker :)
Prior to the creation of Docker, did you have experience with other containerization technologies like Solaris Zones and FreeBSD Jails?
Did you at some point, either prior to or after making Docker for Linux, consider building something like Docker on top of FreeBSD Jails?
As a user of both Linux and FreeBSD, running Docker containers and Jails on each of them respectively, it would be interesting to see what FreeBSD Jails could be with something like Docker built on top of it while also taking advantage of other features on FreeBSD like fully using ZFS snapshots, ZFS send/recv, ZFS clone, vnet, and so on. Such that spinning up a container would under the hood receive ZFS snapshots from the “Docker registry” equivalent server, and it might clone the snapshot locally before starting up a container instance so that multiple container instances would be using the same locally available ZFS snapshot that was received once from the “Docker registry” equivalent.
How did MirageOS end up powering parts of Docker stack?
We acquired a small team of Cambridge-based computer science & systems engineering wizards, called Unikernel. We then worked together to apply systems engineering to improving and expanding Docker in various ways.
The most successful project that we worked on together was "project Piñata" which later became Docker for Mac. It turns out bundling a Linux VM in a native Mac app, and integrating it into the host filesystem, networking stack etc requires lots of magic..
Now, most of the Unikernel team were very active OCaml developers, and created MirageOS among other ocaml projects. While working on Piñata, they simply reached for the tools they were most familiar with.
(Unikernel cofounder here)
When acquiring our company, Solomon took us to a remote farm in France for a week for an offsite, fed us fantastic food and wine, and the first prototype of that Docker for Mac VM was working by the end of the week! It took a few months to hack together the shell. There's a video of Solomon and the pinãta launch party on my blog ;-) https://anil.recoil.org/notes/cacm-docker-cover#getting-invo...
There was a whole other slice of work we did for project Piñata too in the form of datakit (https://github.com/moby/datakit) that was well ahead of its time. I was looking at this archived code again just last week while thinking about agentic git workflows!
Hi Anil! Welcome to the 2026 Docker reunion thread :)
How did I not know about this video until today!
Hope you're doing well.
I am having a blast working on environmental science these days, and we’re long overdue a beer next in SF!
Am firing up Dagger to give it a try on our geospatial workflows for the first time just now; it’s got the same attention to CLI detail as Docker 1.0 did :-)
What made you decide to use a URI as the namespace for images? I like it, but I think it's been very misunderstood which has caused people to dislike it.
I was inspired by Go's use of URI as a namespace for user packages, and short names for the standard library. I thought it was an elegant way to allow everyone to host anywhere, and dodge a lot of complicated namespacing issues.
One design principle that was very important to me, was portability and avoiding fragmentation. I wanted every image name to have the same meaning in every engine, so that when writing a Dockerfile, you could have confidence that it would actually behave the same on every docker installation. In this we disagreed with other platforms (like Red Hat) who sought to "embrace and extend" Docker by changing the meaning of short names. We always fought this to avoid fragmentation of the platform. This did not make sense from Red Hat's perspective, since they were the platform - who cared about fragmenting Docker? That led to some negativity which is probably what you're remembering.
At the same time early Docker users wanted the ability to host their own images without being forced to use our central registry. Using Go's system solved both problems at once.
Thank you for taking the time to reply. Yes, it was the short names and the way people were trying to run local mirrors that I remember.
I personally never use the short names and have really come to like your solution. I host my own registry and I can glance at a build or compose file and immediately know if I’m depending on any external resources that I shouldn’t be.
I originally disliked it, but it’s been my favorite namespace solution for a long time now.
Any regrets about the namespacing or things you’d do differently with it now looking back?
My only regret is that so many users had to run local docker registries on each node of their cluster, then write `FROM localhost:1234/myimage` in all their Dockerfiles - all to avoid unnecessary re-fetching of remote images. My goal was to make image distribution a native feature of the cluster, so that you wouldn't have to choose between portable Dockerfiles and proper image caching. But Kubernetes ended up dominating the cluster layer, and as far as design goes, we were not invited to that table. So as far as I know, the problem was never solved, at least universally for all Docker users.
Did it though? The amount of grief I've had because `foo/bar` gets silently expanded to `docker.io/foo/bar` is extensive.
Would you consider writing Docker/moby in a different language, if you were to create it scratch today?
Oh gosh, the choice of language is such a context-specific design decision. The first versions of what became Docker were written in Python. The choice of Go was a mix of nerdy excitement ("it's like the best of Python and C!"), and adoption strategy: back then devops tools were segmented by language runtime: ruby, python or Java mostly. Shipping a compiled binary allowed us to be adopted by all tribes.
None of these things can be transposed today, so I guess I can't answer...
For what it's worth, my current project (dagger) is written in Go.
I interviewed Solomon about the Go history (plus some others) for this talk https://www.infoq.com/presentations/languages-cloud-native/
Actually there are some longer unedited versions I might release.
That's a good answer that effectively rebuts the article's claim that Go was a bad choice imo.
The article's point that Go is a bad fit for some of the low-level interactions with the kernel, is absolutely correct though. We had to do a lot of juggling to get it to work properly. But in the end it was fine. I would say it was the correct choice of language at the time, and it certainly contributed to Docker's early success.
Another explicit goal was to facilitate contribution. We thought Go syntax was "vanilla" enough to attract a broader set of contributors than a more niche, opinionated language like Erlang, Haskell, Clojure etc. I think that also worked out well - Docker was known for attracting many contributors, many of them making their very first open source contributions. We even ran special meetups to help people become contributors, that was fun.
This is the coolest set of contributions to HN this month, most likely.
Thank you.
I was a dotcloud customer around 2012. You might have responded to one of my many support emails.
Ha ha, thank you for your trusting us back then! Hopefully it was a good experience? Scaling reliably on AWS was not so simple back then...
Haha, well, let’s say the experience motivated me to learn how to run nginx and uwsgi on my own :)
(And as a result that, with ansible, continues to be my preferred way to deploy while the rest of the world passed me by…)
Digital meet cute right here!
General question. When you were designing Docker, did you anticipate the success it would achieve?
To give a sense of time: Docker launched in 2013, but it was one of many iterations of the same project, which I had been working on continuously since 2008. Over those 5 years, virtually nobody cared or understood what the hell I was working on. I lived in France and had no connection to the tech industry.
So, not only did I not anticipate this kind of success, I did not even have a reference frame for what success might even look like. All I knew was, I had an itch that I had to scratch, and I had nothing better to do :)
I remember in 2013 the first time someone showed me Docker [im in Mexico]. My first thought is that we already had Vagrant for setting up and running virtual machines through console.
I didnt get it at that time haha.
I was still hearing that in 2015.
In those years in the systems group I was working in at a fortune 500, we were very interested in this type of technology. Mostly using Solaris Zones and cgroups and trying to do this ourselves with limited results.
When Docker came out it solved so many issues we were having. Thank you for scratching that itch.
Was it always called Docker? Do you think the branding played a larger role in success than you would have expected?
I believe Docker was always called Docker, but it grew directly out of the previous generation of the technology used by dotCloud (the company Solomon founded), which was using LXC with AUFS.
I think the breakthrough with Docker was bundling the AUFS-based image layer format with a centralized container image sharing registry. This was revolutionary at the time. But another huge element of success was Solomon and others evangelizing the technology relentlessly at meetups and conferences. I distinctly remember attending a demo in 2012 where the crowd was very excited to see the Docker workflow that we now take for granted, in large part because Solomon was so good at presenting it. At the time, I was working at a cloud PaaS company where we developed a similar technology on top of LXC and BTRFS (but no image registry). Docker really made me think hard about what we were doing, although I didn't end up making any moves.
https://jpetazzo.github.io/2017/02/24/from-dotcloud-to-docke...
You are correct :)
> the previous generation of the technology used by dotCloud [...], which was using LXC with AUFS
One small note, the earliest versions (2007-2011) were based on vserver then openvz, which predated lxc. Back then you had to patch the kernel to get container primitives. And since virtualization also required patches (eg. Xen), we had to carefully combine both sets of patches, which was poorly supported because nobody else was crazy enough to stack 2 kinds of virtualization!
It used to be dotCloud iirc
Dotcloud was a PaaS:
https://news.ycombinator.com/item?id=2032821
Then it pivoted to docker:
https://news.ycombinator.com/item?id=6634002
I hope one day I can say something like “I made Docker” :,)
Not luck: pay attention to what the community needs, and be in a position to execute when you spot it.
Were you around for the big Moby open sourcing? What went down from your perspective? If I recall, it was a fairly dramatic move that was not driven by the interests of Docker corp.
Yes, I was around for that. It was one of my skunkworks projects, as I had lost formal authority over engineering and product by then, and mostly used my "founder aura" to convince small pockets of people within the company to build cool things with me.
However Moby was a poor choice for a skunkworks project, it involved mostly communications and project management, which requires consistency and scale. Overall it was poorly executed because I failed to build enough support internally ("Solomon's at it again, let him do his thing"). Here I am trying to explain it on Twitter after the keynote, using a picture of a hand-drawn diagram, because nobody in marketing or product was assigned to help. https://x.com/solomonstre/status/855918630915133440
I think the goal of Moby was fundamentally correct: we were trying to replicate the Red Hat model, with well-scoped upstream components (linux -> runc, containerd..) integrated into a commercial product (Docker -> RHEL), with a faster-moving community project in between (Fedora -> Moby).
This was a way for us to clarify that Docker was a product, with a community of users, and Moby was an open source project, with a community of contributors. This clarity was very much needed, because different communities were projecting very different expectations on the brand "Docker", and these expectations conflicted with each other.
In a way, Moby aspired to be what CNCF ended up becoming - the "big tent" where the broader community meets to hammer out the general shape of the container stack. It seems obvious now that a single company can't be the sole authority over such a vast scope. But in the first years of Docker, we really did have that level of influence, because we literally invented the market, and brought the community together in meetups and an ever-growing Dockercon. It's only when Google and Red Hat pushed Kubernetes and CNCF, that our influence over the stack started to wane. Maybe it was for the better, I don't know if we would have been up to the task as designers and engineers. But at least there would have been a consistent design, with a Linus Torvalds figure willing and able to say "no" to stupid and redundant features. For a while I played that role - but then I was replaced by SIGs and committees.
I’d love to hear more about some limitations/tradeoffs of docker that you made at the time that you regret in hindsight. Was there anything that you wish didn’t get baked into the “default norms” for containers, or areas that would benefit from more attention and redesigns?
I’ve been following dagger and it’s been really interesting to see the evolution of the design, and new set of abstractions. There’s also nix which has seen popularity, with their approach of building layered oci containers.
Given how things have changed since your creation and the way things are going, are there any things you would have done differently?
There are many. But there is an alternative universe where I remained CEO instead of transitioning to CTO, and continued pursuing my product vision obsessively instead of compromising on it because of lack of self-confidence and a desire to be loved. I think I would have avoided a lot of the mistakes that the company made - but on the other hand, I would have made a fresh set of mistakes instead, so who knows what the outcome would have been!
Thanks. I needed to hear this. So many things in life can turn out differently if we can overcome our “lack of self-confidence and a desire to be loved”. It’s never too late!
Thanks for your innovation and helping developers! If Docker hasn't been your focus in recent years, what have you mainly been working on and what's been your favorite?
Not related to the topic but. Do you have any book recommendations? Not necessarily programming or system related, looking for great books ~
Thanks for developing such a valuable tool!
basically my whole career has revolved around being good with docker and other related tech, so thank you! started on 0.7 :D
I like to tell people the story of how I got a pre-release peek and frankly didn't understand what I was looking at. You emailed me in reply to a comment I made around that time[0].
Funny how little has changed except the names, I guess.
[0] https://news.ycombinator.com/item?id=5347255
How does it feel to have fundamentally changed how many/most enterprise software runs?
I feel grateful for having seen from the inside one of those rare "hypergrowth" moments, it's a terrifying and exhausting experience but I would do it all over again without hesitation.
I also feel lucky that people are still using my work, that it endures without me, and that it generates (mostly) positive reactions. There was a time where Docker had a lot of haters, which was hard on me. But with time I've noticed that the negativity gets mostly washed away, and what remains is positive interactions like the ones in this whole thread. So now Docker is this cool experience I share with lots of people around the world, and when we meet we get to talk about this shared interest. I love it.
I am firmly convinced that had Rust been more mature at the time, the pivot from Python would have beem into Rust instead of Go.
Am I on the right track?
Why?
Python and Go are both memory-managed languages. That is perhaps the hardest part of programming (outside of getting the code to do what is intended).
Go and Rust are both compiled, but Rust requires the developer to figure out all of the memory management before compilation, which slows down development to a crawl.
Almost no company will rationally choose Rust over Go if they know how the development speed tradeoff looks like.
Because Go is a badly designed language that only took off thanks to Docker, and as the article points out, a few C workarounds were anyway required to make it possible in first place.
I would have chosen C++, but given that isn't the new darling any longer as in the 1990's, that begs the question of the new kid in town that everyone is now rewriting everything into.
This obsession with Rust is very hard to understand
Yeah but the post criticized Go for a project like Docker, so it makes sense to ask what else could have been used. At the time there weren’t many system languages fit for the purpose.
The obsession isn't with Rust, the obsession is with Go not being a good programming language and the only reason it took off was thanks to Docker.
So they would probably never rewrite from Python into C or C++, because not cool, too hard kind of motivation, otherwise they would not have picked Go.
So that begs the question, if the obession as you call it, was already a thing back then, would Rust have been the lucky candidate?
I generally agree with your comments, but do you really think that a language created by such luminaries as Ken Thompson and Rob Pike and endorsed by Google really needed Docker to "take off"?
Yes, see Plan 9, Alef, Inferno and Limbo, such luminaries as Ken Thompson and Rob Pike aren't always right.
As for endorsed by Google, see Dart and Flutter, Android Things, Fuchsia.
Now everything combined, plus Docker and Kubernetes, yes it was definitely a factor.
It's not hard to understand. It's just not rational to anyone who actually calculates how performant it is (very fast, I agree) with how long it takes to write.
Once I realized how difficult it is to write complex code that actually satisfies the borrow checker, I was perfectly happy with writing somewhat slower code in Go or Python.
Anyone that has written enough systems software in various kinds of languages, using Rust isn't that bad, only when coming from scripting languages.
Anyway, I clarified on a sibling comment the reasoning behind my question.
A while back you had this tweet which often gets cited, or at least _i_ personally think about it a lot.
> If WASM+WASI existed in 2008, we wouldn't have needed to created Docker. That's how important it is. Webassembly on the server is the future of computing. A standardized system interface was the missing link. Let's hope WASI is up to the task!
How are you feeling about WASI today?
Do you think e.g. students learning docker syntax at a uni is beneficial in 2026 or you think LLM-assisted programming means that Docker will become an abstraction under an abstraction and its setup will be mostly automated by LLMs? I find myself very rarely debugging or handwriting dockerfiles nowadays!
If you're going to fully delegate something so trivial as Dockerfiles to slopmachines, why go to university at all? You don't need to learn anything if you delegate everything to the slopmachine.
This is a strange question because you could learn Dockerfile syntax in a single night. There are only 18 directives, and the average Dockerfile uses like 6 of them. 90%+ of Dockerfiles is just "run this shell command" and "copy this file into the docker image". The only time it flirts with complexity is multi-stage Dockerfiles, but 90%+ of Dockerfiles are a single stage.
https://docs.docker.com/reference/dockerfile/
> The only time it flirts with complexity is multi-stage Dockerfiles, but 90%+ of Dockerfiles are a single stage.
And multi stage builds are just a directed graph short hand for having separate intermediate dockerfiles. There’s nothing special about them either.
If I had to pick one thing that is tricky for someone new to docker to understand, it’s how the filesystem state is cached using the previous state, the env, and the raw command text.
I think the key thing for a university student is to get the big ideas down: what is a container exactly, how does it work, how does it differ from a VM, what assumptions does it make about what you're running (and how do you run something that doesn't fit that pattern), why do images exist in layers and how are layers implemented, what are the performance benefits/tradeoffs, etc. etc.
When I was learning Docker I can't tell you how many people would describe it to me by analogy to a standardized shipping container, and start going off about shipping container dimensions and supersized cargo ships and trucks and railcars... and finish by smiling beatifically, like they had just explained the whole thing in a masterstroke. Let's... uh... not do that. :-D
In a problem set, the student might learn a bit about dockerfiles or docker-compose.yml... but only, I think, so they can try building and running a container of their own!
I'm gonna be real with you. No one needs to know docker syntax. Most engineers just copy and paste chunks of Dockerfiles around anyways.
Docker syntax is one of those things that just immediately falls out of my head when I close the editor. I don't care about it at all.
Even better: there are many, many tools that build OCI images without using the Docker syntax. I haven't hand-written a Dockerfile in at least six years.
LLMs have not loved the needle here in my sphere, it was already bouncing off of the zero on the gauge.
edit: this isn't to say the syntax is good or bad (though there are several questionable parts of it) but extremely simple, and concise enough that you can just look up literally any command with great ease. It's surely a large part of its success, for better or for worse.
I find the Dockerfile syntax easier to use than any if the tools that generate it indirectly.
If you were implementing an operating system (free from compatibility concerns), what would you do differently to better support lightweight containers?
What learnings did you take from Docker when conceiving Dagger? Or is it more about unlocking BuildKit?
Oh wow, hi! I think my biggest “gripe” with docker is that everyone grabs it as a hammer and views many problems as a nail. Am I way off base with that? What was your original vision for how docker might be used in the wild?
Thanks but I still run LXCs /s
You're a genius well done making containerization actually happen.
What made Docker unique IMO is not the container technology itself. I used chroot 20 years ago and it worked just fine. Yes, no network namespaces; no process namespaces, but I don't even think these are that important.
What made Docker unique were there things:
1. Dockerfile concept. Building images layer by layer. Caching layers. Everything in a single file. It made possible to iterate on image quickly. It made possible to implement CI builds efficiently.
2. Docker network registry concept. Putting universal images onto a server and share them with the world.
3. Momentum. A lot of software that was traditionally complicated to deploy, suddenly appeared in docker servers. I remember how fascinated I was, when I could deploy wordpress website with tiny docker-compose. And how contained it was. I could just delete every trace of it from my host. I always felt uneasy installing various things onto my host and docker just solved that uneasiness.
Interesting, I would probably say more-or-less the exact opposite.
The main draw for me was that it made containers absolutely trivial. Sure, you could spend ages figuring out how to DIY some isolation, but in practice an awful lot of people stopped at having Apache fork its workers with a non-root uid. You want isolation and a repeatable setup? Spin up another VM and have Ansible set it up! Docker provided a batteries-included lightweight way to isolate and manage multiple applications at once.
Dockerfile? I would've been happy with a Bash script, or handing it a tarball. Caching and fast iteration? Compute is basically free. A single file? Well, there's already a whole bunch of app source code files sitting next to it, so what's a few more in a `setup/` directory? The registry? A shared base could've also been achieved with, say, a `deb` file providing the right dependencies, files, and post-install scripts.
I do agree on the momentum, though. Having every piece of software reduced from a multi-page setup guide to a single command and not risk screwing over your entire machine is absolutely magical.
The momentum, I think, matters a lot more than the other two. Makefiles already allowed you to build layered things with caching. Artifact repositories have also long been a thing.
But the argument I used for advocating Docker in environments where people were not fans of it was that the Linux container had become the de facto standard unit of deployment. There is so much tooling written under that assumption, and if you want to use any of that tooling, you need to package your deployments as Linux containers, even if you want nothing to do with Docker otherwise. That's why it had so much momentum.
Then a fourth point I think you're missing is that the concepts used by Docker are relatively simple. Before I understood what Linux containers were, I used to think things like, "Why would I care about Docker when I can isolate my processes with Linux users, cgroups, namespaces, etc." What I didn't know at that time is that "Linux users, cgroups, namespaces, etc." is exactly the thing Docker was giving a friendly UI to.
> Docker’s container runtime, runc, is written in Go.
I worked at Docker from 2014-2019. The most important part is actually written in C.
[I wrote a whole comment here about to explain how the hack works because I was too fixated on how technically incorrect "written in Go" is. Then I went back to read the rest of the article only to see that the paragraph immediately after this kind of explains it in its own way! Let that be a lesson to always finish reading the article before posting about it.]
Hi Josh!
I prefer Podman most of the time, but Docker is/was amazing and a great DX put on top of the so called "5 different linux kernel features in trenchcoat". That's the selling point, it's super lightweight and takes full advantage of the OS features that were already there (or being developed) but were awkward to use and required tons of specialized knowledge.
Does Postman sidestep these weird things about Docker? Or just some of them?
I assume you mean podman?
The first two are just kind of the nature of containers on linux, and applies to any container technology.
Podman does allow you to run it as a non-root user (with some limitations), that was in fact one of the biggest motivations for making podman. This wasn't possible at all in the early days of docker, it wasn't until linux introduced user namespaces, where the root user inside the namespace mapped to a non-root user id outside the namespace that non-root conatiners were possible.
And podman, unfortunately IMO, is also written in go.
I switched to Podman because it’s daemon less (no resources are used unless a container is running) and it doesn’t need to run as root.
Otherwise it’s basically the same and you can continue using your Dockerfiles and Docker Compose files.
Unless you’re doing something really fancy it’s basically a drop-in replacement.
I think you meant "Podman", but yes, Postman almost entirely sidesteps docker.
:D
I think the only thing in the "weird things" list that don't necessarily apply to another container system on linux would be the go-specific issues.
Most of the other container runtimes don't require root for many of the use cases in which Docker does.
When Docker came out, I was a heavy user of OpenVZ.
My bash build scripts looked like Dockerfiles.
Most of the container code in Linux came from those russian guys.
Took me 6 months to find out the difference between stateless and stateful containers, after Jerome Petazzoni published a blog post opposing the usage of an SSH server inside containers.
Lots of positive stuff in this thread which I won't bother repeating. Instead, I'll add a few ambivalent points:
- Today, a lot of functionality still in Docker has been upstreamed into the (Linux) kernel, so containers can be much simpler (and rootless). Docker is still carrying around some of that cruft, which is what makes it slower and heavier than Podman.
- As a devex technology, Docker/containers have been massively successful, but not as infrastructure. Containers never became secure enough to run multitenant workloads without some other form of isolation, so they ended up being redundant with KVM (or gVisor, or a similar second layer) in production a lot of the time.
That second contradiction led to lots of inefficiency. For example, on Google App Engine, for years when you ran a deploy it would create a VM just to run a single container.
I would also point to no secrets primitive.
K8s requires some sort of tool to have any hope of storing secrets in a way that isn't trivallly findable from the rest of the cluster.
But, perhaps I am being too harsh, its not like there are secrets primitives in linux that are easily deployable to namespaces/cgroups.
Always nice to see someone demystifying a complex thing by learning the building blocks it's made from.
Knowing this, you might find rootless Podman's fork-exec method of starting containers to be more sensible. I certainly do. No root, no daemon, no socket.
If you want to learn more about the Linux Kernel namespaces which work together to make a container, this LWN series is really good: https://lwn.net/Articles/531114/
Every statement in the article make me ask: "what's weird about it?"
Everything in that list feel natural when you know how things evolves over many years. Maybe it is just me getting older and less curious.
And not a single mention of the actual weird decision, a client / server model which complicates permission management by at least two orders of magnitude.
I’d argue in 2013, the client / server model is what made docker a success. It allowed for a seamless, and mostly identical, experience for docker on Mac, Windows and Linux for one. It met developers where they were, daily-ing a Mac or a Windows and deploying on Linux headless servers. It had benefit on its native Linux system too that took all the alternative solutions another 8 or 9 years for the entire Linux ecosystem to develop enough solutions around containers to make a daemon unnecessary while still offering docker DX.
Oh yes docker-machine was quite good. You can create a machine on a Mac by using VMware, VirtualBox, or using the cloud such as EC2. Then your docker CLI can work with all of these machines seamlessly without knowing any details about the machine.
That's actually a design decision made to work around the limitation of having to run as root. Since the container had to be started as a privileged user, if you want non-privileged users to be able to start contianers you either need a setuid binary or a client/server interface like docker has. There are pros and cons to each of those options.
Now it's possible to create containers without elevated privileges (assuming user namespaces are allowed), but that wasn't possible until long after docker had established the client/server model.
I spent my whole day dealing with Docker permissions today. Whats interesting and dangerous is how docker just overrides ufw rules like they do dont exist.
lxc never had a daemon but Canonical added one (lxd in 2015) and lxc forked it to incus to be community managed.
rkt (2013) was CoreOS's alternative to Docker and was daemon-less and failed.
Podman came along relatively late (2018) and benefited from a lot of Linux kernel and systemd improvements since Docker was designed.
So while arguably the client/server model is outdated for container management tools, I'm not sure it's weird. This is my workstation for example:
Daemons are maybe the least weird paradigm you can choose for a Unix program.Yeah, the daemon is the real problem with Docker.
This is why I prefer Podman.
A lot of people start using Docker as a black box and have the mental model of a VM, and when they start digging into the internals all the "weirdness" starts to exhibit itself.
Maybe. That's why i start thinking i am getting old and less curious :(
These days they may as well have the mental model of Docker when they start using a VM :)
Most of them don't seem that weird to me. Except the use of go. It definitely seems strange to pick a language that can't natively call clone, one of the most important syscalls for using namespaces and implementing containers.
To solve the networking setup, typically some sort of "slirp" is needed. Slirp on one hand takes (fake or real) packets from somewhere and then on the other side pass it to a normal Linux process that does CONNECT / BIND / SEND / RECV.
It turns out you can do all the packet processing magic (connection tracking, syn handling, etc) in userspace totally fine, and then indeed hand it over this to userspace syscalls.
Being upset with classic C-based slirp and rust smoltcp (not enough features) I've dug up Fuchsia userspace tcp/ip stack - netstack - and wrapped it in slirp a-like envelope.
https://github.com/cloudflare/slirpnetstack
So this beast is basically that - on one hand you pass it packets (potentially from inside container) on the other hand it does connect/bind/etc to host OS.
This approach, in one form or another, is used by tailscale, podman and many other container runtimes.
Poke around in bocker[1]. For technical people, it makes the same points the article makes in a way that's very clear.
[1] "Docker implemented in around 100 lines of bash" https://github.com/p8952/bocker
Bocker has a drastically smaller inherent software attack surface than mainline Docker, yet its container isolation is practically non-existent, making it way easier to exploit and break out of.
I am a huge Docker fan! I would reiterate what several commenters have said already that it had a huge impact on my career. My question is relatively pedestrian compared to what others have asked already. I absolutely loved the "adjective-scientist/nerd" container names Docker generates. In fact, that is one of my most favorite things about Docker. How did that come about and who chose those combinations?
I'm glad you enjoyed that! We wanted a way to make container names reasonably unique and reasonably memorable by a human, so this was a fun way to do it. I picked the original combinations, then over time it became a ritual for contributors to propose fun new values.
There's a fun little easter egg: the combination of "Steve Wozniak" and "Boring" is explicitly forbidden, because Steve Wozniak is not boring :) https://github.com/moby/moby/pull/4902
> I decided to make my own container runtime for fun [...] Unlike VMs, containers are just processes
Try to make your own VM runtime next, you might be surprised!
I started hating Docker after seeing all the cargo culting on Reddit. Whenever someone asks about deploying an app on Linux, there's always at least one comment saying, Just use Docker. We finally ended up building our own deployment automation tool for our entire stack that doesn't use Docker at all.
That comment invariably comes up because for most people it’s the objectively correct and most well supported option. If you’re asking redditors how to deploy something, you’re precisely the person docker is for.
Even when I'm not using Docker to deploy something, I'm _probably_ using oci images at the least, but I'm generally running my local dev stack with Docker compose. Not always, but I use it in a lot of projects.
That part about the threading was a huge source of issues back in the early days. Goroutines could hop threads and it's the threads namespace/cgroup configuration that gets inherited when forking..
One of the gnarliest issues I recall was where network namespaces could occasionally not be cleaned up correctly. So "randomly" (but more likely under certain loads) the networking of a reused thread would get jacked and Docker could no longer pull images.
Plus Go's function for locking a goroutine to an OS thread was broken to the point where you could not safely rely on it for quite a while.
Never, ever, ever, write your own userland threading library.
(rule broken by almost every language in common use, unfortunately)
That’s an absurd statement. Optimizing every language around what a runc-like program would need is absurd.
Where does this advice come from? Most web servers would use a lot more resources if they didn't use userland threading, wouldn't they?
I think you're confusing userland threading (also known as green threads) with a regular thread pool implementation (that uses regular OS threads).
Because most web servers use OS threads. Nginx, Apache, litespeed, they all spin up `core count` threads but they can all work single threaded just fine as well.
No, I mean that Tokio, Go, etc., were built with general-purpose web servers in mind, and they decided that lightweight threads were necessary for that purpose because OS threads had too much overhead to be used per-request.
NGINX uses a state machine per task, which as far as I know is kind of what Rust's async functions end up desugaring to.
Interesting read! Thanks.
Though it would be better if it compared Dockers ways to say Podman, which does things a little different I think.
For others, like me:
Footgun:
A footgun is a programming or system design term for a feature, tool, or API that makes it very easy for a user or developer to shoot themselves in the foot—meaning it causes self-inflicted damage or severe errors while working exactly as intended.
I have recently discovered INCUS, very nice alternative to docker when a simple container is not sufficient https://linuxcontainers.org/incus/
A good talk on the tech behind containers.
Containers From Scratch • Liz Rice • GOTO 2018
https://www.youtube.com/watch?v=8fi7uSYlOdc
I created Range (for fun). No daemon. No root.
A few cool tricks:
`range shell python:3.12` - opens a shell image without pulling the entire image.
`range shell ubuntu --commands` - you are in Ubuntu. Type any command Ubuntu doesn't have and it comes from the image that has it. On the fly.
`range run --network none ubuntu` - network-isolated sandbox
range run ebs://snap-... -- cat /etc/os-release - opens an EBS snapshot with no EC2 instance
Another cool trick:
`range run python:3.12 --mount hf://moonshotai/Kimi-K2-Instruct:/model -- du -sh /model` says 959G instantly :)
Check it out here - https://getrange.sh
Most importantly, Range doesn't need any daemon, any container runtime, any registry or any image conversion. One binary only. On a Mac it boots its own tiny Linux VM per session, and on Linux it uses KVM.
It also works with more than contaienr images - HuggingFace repos (hf://org/model), S3 (s3://bucket/...) or any HTTP server really.
Range always reads only the bytes your program touches. Nothing else. You can prob shell into a 1TB image from a Raspberry Pi :)
> And if you know where to look, you can even find that process in /proc.
This is very important for debugging! Not just that process, but the processes running in that Docker can be found in /proc.
Why this is useful is that you can run gdb outside of that docker and point it to a process in the docker, including setting the file system root to the docker one, so that it gdb reads the correct objects from the correct paths, to get the right symbols and addresses.
interesting read as a relatively less experienced developer.
If you're old enough to remember some techniques that people used before docker (like chroot(2)-ing daemons for security), the "containers aren't real" observations are not really new info. When containers came on the scene it felt like an outgrowth or evolution of that technique, which was extremely crude and simple.
Docker question. Who prefers docker for local dev vs. run the thing directly. I usually prefer run the thing for debugging and hot reloading. Sure: build the containers and test them in CI. Dependencies like PG sure run in containers as they change less often and you save the install.
some people do all of their software development in a container. It really only works well if you are comfortable using TUIs instead of desktop apps though.
This only scratches the tip of the iceberg. People's minds are blown when they realize Linux has no concept of a container.
It's not a vm: It's all namespaces, baby! Unshare one or more of your mnt, pid, net, ipc, uts, user, cgroup, or time namespaces and you have a "container".
Oh man, what a fun thing to do! I may have to replace what used to be my fascination with programming languages and compilers with virtualization or containerization now.
This is a better way to understand it: we can think of a container as an isolated process. When deploying containers, having a clear understanding of the boundaries between containers and the host machine in terms of permissions, resources, and data is extremely helpful for troubleshooting issues.
I can't use Docker locally only because lack of enough space on my SSD.
That's the case for Range - https://getrange.sh/ - runs any OCI image without pulling it. HTTP range requests FTW! :)
Today, I would definitely choose rust over go. And I lament that so much container software uses go rather than rust. But back when docker started rust was a lot less mature and probably wouldn't have been a good choice.
C was and is a good choice for that kind of software. I don't understand the prevalence of Go in these tools.
Easy: far few footguns.
Performance is basically irrelevant because you're not really interacting with the running application. You're running as root, so security-critical. You're talking to the internet for things like fetching images. You need rather complicated business logic to deal with all the possible options, protocols, and file formats.
Sure, you could do it in C, but dealing with dependencies is a major PITA and you're permanently one tiny mistake away from getting pwned, while not needing any of C's strengths. Go is safe, easy to write, and ships as a single batteries-included binary: what's not to like?
What's not to like? For example the fact that Go's threads and OS threads are two different things. For example the fact that I can't just use fork in Go. And many other things.
Go is good for writing web servers. For system software? Not so much. 90% of Unix is written with C. It's not that bad.
Go has grown too, since 1.25 it knows it’s in a container, uses all cores on CPU for go routines without hiccup.
That isn't really relevant for the tool starting the container. The biggest problem with go here is that it can't natively perform the operations necessary to set up a container, due to the nature of the runtime, so it has to resort to complicated workarounds like a shim in c, and starting spawning from /proc/self/exe instead of forking/cloning.
CTRL+F "rust"
Just use Podman
> Unlike metal servers, VMs are just operating systems being gaslit by a hypervisor. Your actual CPU is still executing those instructions! And if you know where to look, yyou can even read the other VMs’ memory in L1D
> Go is kind of a terrible language for a container runtime
I love Go and use it for everything I can but will absolutely concede this point. It's clearly possible to write the most successful container runtimes on earth in Go, but you absolutely have to fight the language/runtime sometimes. Since containers are based on processes, handing off a container spec to a minimal rust runtime seems like what we should all be doing by now. crun has just proven good enough that AFAIK no one has RIIR.
For as "terrible of a language" Go is alleged to be for a container runtime, its not like crun or youki are exactly faster in many meaningful ways unless you count churning containers running true and exiting.
And implying the system kernel still has to run the container kinda ignores kata containers and gvisor runtimes.
It's about using go to develop container runtime being terrible and not that "if it was C it would be faster". OS runs the container, docker just does the setup and some managmenent.
For me it's not about the cpu time, but the memory. If you spin up hundreds of containers the docker overhead becomes very real.
Docker does not need root.
Side note almost nothing about Docker is weird. This article chooses rather boring design decisions and picks on them.
> Next up: Databases are weird
> Did you know that databases are just a file?
oOoOoOoOo
Mostly joking. It's cool to see tech demystified, but I feel like a lot of things we use and take for granted are like this.
Honestly it took me so long to pick up containerisation because of how it's marketed. "Orchestrate a zillion fake computers, ultralight VMs, livestock not pets" big whoop.
"It's an idempotent language for OS level dependencies so you can be sure exactly what is running on your server"
Gimme gimme gimme!
Docker hasn't required root for 7 years, I wish people would stop repeating that. It uses a root daemon by default because it makes things easier. Also, Containers are only as fake as a "Desktop" in an OS, which isn't real either. It's a user, network & process namespace with a control bus and a bunch of other weird things. A Desktop is a Container with a GUI.
Other weird facts about OSes: a filesystem is a database, the Linux kernel is a monolithic program, "hidden" dot-files were a Unix bug that got carried forward, deleting a file doesn't remove the file from your machine, virtual memory is a lie, your computer can't run multiple programs at once (it's just running one at a time real fast), in 12 years most systems will crash due to one number getting too big, you technically never need to reboot because you can launch a different version of Linux while already running Linux, you can freeze a program to a file and then later resume running that same program in the same state (but nobody uses it), files aren't actually stored in directories, there's a maximum number of files (and size of file) on most filesystems (which you can use to lock up a machine, despite file quotas existing and nobody using them), you can lock up most systems by forking too many times / using too much RAM / accessing disk too much, there's a default number of open files and/or TCP ports which is artificially low and you can use to again lock up a machine, pipes are just two file descriptors connected to one memory buffer, an IP address is just one long number split up into bytes, etc.
The rootless Docker experience used to be pretty bad and it was less friction to use something that was rootless by default (like Podman). Has this changed?
it changed and its better now. rootless dockerd wants newuidmap/newgidmap + subuid range before it start, and cgroups only work with cgroup v2 + systemd. also, on a host that already has those, its fine on one without systemd, and podman is still less friction, mostly because there is no daemon to babysit
> your computer can't run multiple programs at once (it's just running one at a time real fast)
Of course it can.
Give them a break, machines with multiple CPU cores have only been around for 40ish years!
One of my deeply unpopular opinions is that Docker only exists because Linux as a bad operating system that is incapable of deploying Docker. I have not once worked on a project where “we’ll solve this with Docker!” didn’t result in never ending Docker fixes and workarounds and hacks.
This is not a slight on Docker itself. It’s a slight on the conditions that induced Docker to exist.
One of the great selling points of one of my projects is “this is easy to run without needing Docker”.
>>Images are just folders that Docker puts somewhere and then calls chroot (or pivot_root) to make that folder your container process’s new root directory.
And folders are graphical representations of directories--sometimes, not always--and not directories themselves. When will the newbies get this?
The high level abstraction is more important than the low level implementation unless the low level implementation is leaking through, or there is a significant advantage conveyed by understanding the low level implementation.
I know a lot about how http works, and it comes in handy, but I can understand how juniors get away with not knowing it as fully as I do.
Directory is a bad metaphor too.