I went through a similar "thought process" to build this: https://www.endbasic.dev/endbox.html last year. I originally wanted to "just" launch my binary right after the kernel started... but in the end settled for a full NetBSD base system to get things like network and WiFi configuration to work with ease. (That said, I still hook very early in the boot sequence to launch my own program and take over the console so that the rest of the system is invisible and initializes in the background.)
> .. There was a brief moment in the early 2000s where the newly introduced SSDs made booting quick ..
Anyone else reading this and getting the feeling someone is reminiscing about a past they didn't live and thus are seeing it through rose-colored glasses because it didn't exist?
In the early 2000s, SSDs had an obscene price tag of ~$1000 per gigabyte, so even an 8GB drive would cost you about $8000. There was nothing for the tech industry to ruin because only 0.01% of people owned a setup with an SSD. And neither Linux, NT or Mach were optimized for them. Hell, to this day Linux' VM system is by default still tuned for swap and slow HDDs (swappiness 60 and all that) because no one has bothered to update the defaults.
I remember this was the time when Google started pushing the Chromebook idea and highlighting how it could boot in "just a few seconds". One of the earliest models I got as a dogfooding device probably needed 20 seconds to boot or so. Nice, compared to the awful boot times of machines with HDDs... but not stellar.
But then, in 2011, my wife bought a MacBook Air with an SSD and I was blown away. That thing booted to a full desktop (and not the joke that ChromeOS was) in... 5, 6 seconds? It was ridiculous.
And we have lost all of those gains. I find it painful to witness how a recent Mac chews through I/O during boot or doing any sort of software update (iStat Menus is great to watch this sort of thing), and how these feel slower to that early experience of 15 years ago :-/
I distinctly remember buying my first SSD laptop in 2012 and being amazed at the boot time. At the time, I didn't realize it was because of the SSD, I just thought, "wow, the devs have made some fantastic optimizations in boot time!"
I wrote a batch script that that opened the entire Creative Suite in parallel and couldn't believe how much faster it was than just launching just Flash would have been on the old drive.
> Somewhere, a LLM is trained on this code as we speak :)
If we collectively have to change the way we do things to better fit our tools rather than changing our tools to fit what we would naturally like to do, then we've failed as an industry.
I came up with this one on my own many many years ago. Exact order and bs=1M and everything. I taught it that :)
Though I usually do the individual partitions before the main disk, to make sure fdisk doesn't complain about existing signatures as I start making partitions again. These days many disks and SD cards support "blkdiscard" to just wipe the whole thing (and temporarily gain some performance from clearing internal info), and "wipefs" became a thing to wipe the exact sequences fdisk and mkfs look for.
The "dd" method makes most sense on BIOS systems where you want to make sure to wipe the MBR boot code, and know that nothing is accidentally being preserved when you install your preferred bootloader.
I do the “Linux from Scratch” challenge every few years and the last time I did it, I got as far as I needed to be able to build antirez’ LOAD81[1] directly from single mode in a few seconds.. it was so satisfying. I lost the image in some later admin task, but I’m going to do it again, and maybe see about adding threads and SQLite support to LOAD81 alongside the other goodies, just for the fun of it ..
Tried out doing this kind of stuff a while back to create a custom user space, and I found that suckless’ init, sinit, was quite educational for how to deal with init’s responsibilities in a super minimal way: https://core.suckless.org/sinit/
Cool article! I'm working on a tangentially related issue requiring microVMs inside isolated infrastructure environments. Latency isn't really my main priority, but I am always tempted by any option to minimize attack surface. I wonder what it would take to replace the host block mount in this configuration with `vsock` for all communications between the host and guest microVM? Then you could avoid any files being mounted on the host at all while still enabling, e.g., one-way egress to a pre-signed S3 URL via a private VPC endpoint. Very cool article!
Amazon Nitro Enclave does pretty much this, the guest has one method of communication, via vsock, and it's up to you to build the pipes on either side.
It's a huge PITA in practice because whatever you want to run inside some enclave usually ends up being a 'normal program' that needs to talk TCP/IP over sockets... so your vsock I/O becomes a weird mix between a TUN proxy or a SOCKS5 local listener inside the VM that tunnels through vsock.
For example, I have the Windows NT 3.50 kernel compiling from scratch with virtio-net drivers, it's fairly straightforward for me to add a bus driver that runs over vsock inside Nitro Enclave that exposes itself (o,o) as a NIC then handle the tunneling logic in a usermode process in the host - but I don't uderstand the point of why you would do that when you already have sufficient attestation methods that don't require you to do vsock isolation.
As I'm currently exploring kernel build things, the alternative to `make tinyconfig` is `make allnoconfig` which is supposedly will not disable expert options and might be a little bit safer starting point.
> I remember seeing a linux firewall/gateway set up to run with just the kernel, without any userspace at all. Completely unhackable.
Do you remember any details that would let me search for it? Because that does sound cool, and even maybe useful; the thought has certainly crossed my mind that a router or VPN box doesn't really get a lot of use out of userspace... Although maybe it's worth keeping for control/configuration/debugging.
> To print some text or run a simple program, I belive DOS without a memory manager would be even faster.
Or just make your code boot directly. It's not hard to make a .efi, or use
https://github.com/jart/cosmopolitan
to make a binary that runs in many places including bare metal.
IIRC, it ran a script as init process that set up the network cards, set up iptables, etc. and then just exited. Kernel would panic (the "init was killed" panic), but the network would still be functional. Automatic reboot on panic was disabled.
To reconfigure, the admin would simply reset it and start the system with "init=/something/else" as kernel parameter that booted to a normal userspace.
Basically, even if your application _can_ run as the kernel, and it's desirable for it to run with kernel-level permissions, do you really want production to be a world without strace and iotop and the like?
That's a halted firewall setup. Normally as part of shutdown you would tear down networking in SysVinit or systemd but you don't actually have to do that. When shutting down you can choose whether to power off or just to halt. It's basically like the old Windows "It is now safe to power off your PC".
Wouldn't this be useful for embedded types of applications where you have a very specific task you want to do and you want to do it now.... like that firewall example?
Do do something actually useful, the program would have to access some data: network, disk, some sensors, etc. Network alone means scanning PCIe for the network card and configuring it, disk access needs controller also on PCIe, then scanning the ports for the drive, reading partition table, mounting the partition, etc.
All that takes a lot more than 1s. The speedup might not even be significant compared to a kernel optimized for that system (all modules built-in, nothing redundant), but full-featured, plus busybox or sysvinit alone.
> I... fail to see the point of running just one process.
It makes sense if you got some legacy piece of hardware that has extremely limited resources, both in terms of RAM and storage. Write your code in Go and you don't even need libc any more.
It's so annoying how he starts out talking about boot speed, does the entire project, and then doesn't even mention how long it takes to boot up! All he ever says is "boots a bit quicker".
You may also want to build and run busybox for your tiny userspace.
Other things you may want to experiment with is gen_init_cpio.c from linux kernel tree. It makes creating initramfs file structure easier from scripts.
And finally if sys/isolinux is also fun to use for minimal boot images.
I went through a similar "thought process" to build this: https://www.endbasic.dev/endbox.html last year. I originally wanted to "just" launch my binary right after the kernel started... but in the end settled for a full NetBSD base system to get things like network and WiFi configuration to work with ease. (That said, I still hook very early in the boot sequence to launch my own program and take over the console so that the rest of the system is invisible and initializes in the background.)
> .. There was a brief moment in the early 2000s where the newly introduced SSDs made booting quick ..
Anyone else reading this and getting the feeling someone is reminiscing about a past they didn't live and thus are seeing it through rose-colored glasses because it didn't exist?
In the early 2000s, SSDs had an obscene price tag of ~$1000 per gigabyte, so even an 8GB drive would cost you about $8000. There was nothing for the tech industry to ruin because only 0.01% of people owned a setup with an SSD. And neither Linux, NT or Mach were optimized for them. Hell, to this day Linux' VM system is by default still tuned for swap and slow HDDs (swappiness 60 and all that) because no one has bothered to update the defaults.
They probably meant 2010s.
I remember this was the time when Google started pushing the Chromebook idea and highlighting how it could boot in "just a few seconds". One of the earliest models I got as a dogfooding device probably needed 20 seconds to boot or so. Nice, compared to the awful boot times of machines with HDDs... but not stellar.
But then, in 2011, my wife bought a MacBook Air with an SSD and I was blown away. That thing booted to a full desktop (and not the joke that ChromeOS was) in... 5, 6 seconds? It was ridiculous.
And we have lost all of those gains. I find it painful to witness how a recent Mac chews through I/O during boot or doing any sort of software update (iStat Menus is great to watch this sort of thing), and how these feel slower to that early experience of 15 years ago :-/
I distinctly remember buying my first SSD laptop in 2012 and being amazed at the boot time. At the time, I didn't realize it was because of the SSD, I just thought, "wow, the devs have made some fantastic optimizations in boot time!"
I wrote a batch script that that opened the entire Creative Suite in parallel and couldn't believe how much faster it was than just launching just Flash would have been on the old drive.
Or they meant 2010s?
> I found “Building a tiny Linux from scratch” which does most of what I do here but in Rust and a year ago
Linux from scratch seems to still be doing fine at: https://www.linuxfromscratch.org. It's going on 27 years now.
> yeah, I know, proper C code needs to be scattered with return value checks and sensible reports of errno. I’ve left these out for clarity.
Somewhere, a LLM is trained on this code as we speak :)
But anyway, it's great that people are still interested in learning this stuff for fun.
> Somewhere, a LLM is trained on this code as we speak :)
If we collectively have to change the way we do things to better fit our tools rather than changing our tools to fit what we would naturally like to do, then we've failed as an industry.
Hmm LLM hype aside, remember the first writing system (Grafitti) for PalmOS?
Change the way you write to obtain much better recognition.
> Somewhere, a LLM is trained on this code as we speak :)
Had a similar thought when I saw
I came up with this one on my own many many years ago. Exact order and bs=1M and everything. I taught it that :)
Though I usually do the individual partitions before the main disk, to make sure fdisk doesn't complain about existing signatures as I start making partitions again. These days many disks and SD cards support "blkdiscard" to just wipe the whole thing (and temporarily gain some performance from clearing internal info), and "wipefs" became a thing to wipe the exact sequences fdisk and mkfs look for.
The "dd" method makes most sense on BIOS systems where you want to make sure to wipe the MBR boot code, and know that nothing is accidentally being preserved when you install your preferred bootloader.
I do the “Linux from Scratch” challenge every few years and the last time I did it, I got as far as I needed to be able to build antirez’ LOAD81[1] directly from single mode in a few seconds.. it was so satisfying. I lost the image in some later admin task, but I’m going to do it again, and maybe see about adding threads and SQLite support to LOAD81 alongside the other goodies, just for the fun of it ..
[1]- https://github.com/antirez/load81
I bought a couple of $60 refurbs online. This is perfect!
> the linux kernel configuration menu, a wonderful text menu system with a thousand options which has been baffling new users for about 30 years now.
So true.
In addition to C in the article and Rust linked to in the article, Go fans can use the similar https://gokrazy.org/ project.
Tried out doing this kind of stuff a while back to create a custom user space, and I found that suckless’ init, sinit, was quite educational for how to deal with init’s responsibilities in a super minimal way: https://core.suckless.org/sinit/
It's be cool to dual boot with a Linux that has a ~1s boot time, drops you into neovim and lets you save text files to a shared partition.
It's possible to do a similar thing with any old smartphone. :)
https://xnux.eu/p-boot/
> cpio is a very weird and ancient program with a command line which makes tar look user-friendly. But let’s not worry about the details for now.
So true. Been using cpio sporadically for many (many) years. And I still have to look it up. But at least I don't have to do `info cpio` anymore.
Cool article! I'm working on a tangentially related issue requiring microVMs inside isolated infrastructure environments. Latency isn't really my main priority, but I am always tempted by any option to minimize attack surface. I wonder what it would take to replace the host block mount in this configuration with `vsock` for all communications between the host and guest microVM? Then you could avoid any files being mounted on the host at all while still enabling, e.g., one-way egress to a pre-signed S3 URL via a private VPC endpoint. Very cool article!
Amazon Nitro Enclave does pretty much this, the guest has one method of communication, via vsock, and it's up to you to build the pipes on either side.
It's a huge PITA in practice because whatever you want to run inside some enclave usually ends up being a 'normal program' that needs to talk TCP/IP over sockets... so your vsock I/O becomes a weird mix between a TUN proxy or a SOCKS5 local listener inside the VM that tunnels through vsock.
For example, I have the Windows NT 3.50 kernel compiling from scratch with virtio-net drivers, it's fairly straightforward for me to add a bus driver that runs over vsock inside Nitro Enclave that exposes itself (o,o) as a NIC then handle the tunneling logic in a usermode process in the host - but I don't uderstand the point of why you would do that when you already have sufficient attestation methods that don't require you to do vsock isolation.
As I'm currently exploring kernel build things, the alternative to `make tinyconfig` is `make allnoconfig` which is supposedly will not disable expert options and might be a little bit safer starting point.
There's an option `make localyesconfig` (or similar)/which uses the kernel modules loaded on your current system.
I... fail to see the point of running just one process.
If it's just a PoC, then:
1) I remember seeing a linux firewall/gateway set up to run with just the kernel, without any userspace at all. Completely unhackable.
2) To print some text or run a simple program, I belive DOS without a memory manager would be even faster.
3) It takes 1s to boot linux, but an ordinary PC takes 10s to get to that linux. Even U-boot on ARM takes some seconds to load a kernel.
BTW, if anyone knows any current platform that can XiP a linux kernel, please share.
> I remember seeing a linux firewall/gateway set up to run with just the kernel, without any userspace at all. Completely unhackable.
Do you remember any details that would let me search for it? Because that does sound cool, and even maybe useful; the thought has certainly crossed my mind that a router or VPN box doesn't really get a lot of use out of userspace... Although maybe it's worth keeping for control/configuration/debugging.
> To print some text or run a simple program, I belive DOS without a memory manager would be even faster.
Or just make your code boot directly. It's not hard to make a .efi, or use https://github.com/jart/cosmopolitan to make a binary that runs in many places including bare metal.
IIRC, it ran a script as init process that set up the network cards, set up iptables, etc. and then just exited. Kernel would panic (the "init was killed" panic), but the network would still be functional. Automatic reboot on panic was disabled.
To reconfigure, the admin would simply reset it and start the system with "init=/something/else" as kernel parameter that booted to a normal userspace.
Oh, clever; I didn't know you could make Linux panic but keep running.
> control/configuration/debugging
This is one of several major arguments made against unikernels in that famous Triton rant from a decade ago:
https://tritondatacenter.com/blog/unikernels-are-unfit-for-p...
Basically, even if your application _can_ run as the kernel, and it's desirable for it to run with kernel-level permissions, do you really want production to be a world without strace and iotop and the like?
That's a halted firewall setup. Normally as part of shutdown you would tear down networking in SysVinit or systemd but you don't actually have to do that. When shutting down you can choose whether to power off or just to halt. It's basically like the old Windows "It is now safe to power off your PC".
That beautiful orange text on a black background gives me nostalgia warm and fuzzies
Wouldn't this be useful for embedded types of applications where you have a very specific task you want to do and you want to do it now.... like that firewall example?
Yeah, but there's a problem.
Do do something actually useful, the program would have to access some data: network, disk, some sensors, etc. Network alone means scanning PCIe for the network card and configuring it, disk access needs controller also on PCIe, then scanning the ports for the drive, reading partition table, mounting the partition, etc.
All that takes a lot more than 1s. The speedup might not even be significant compared to a kernel optimized for that system (all modules built-in, nothing redundant), but full-featured, plus busybox or sysvinit alone.
Embedded devices or other SoC situations, certain limited scope situations where docker is undesirable/unnecessary, etc.
> I... fail to see the point of running just one process.
It makes sense if you got some legacy piece of hardware that has extremely limited resources, both in terms of RAM and storage. Write your code in Go and you don't even need libc any more.
At that point, it would seem that an RTOS would be even more efficient - and if multithreading is not necessary, then just run it on bare metal.
I do like this as an exploration.
It's possible to boot a VM noticeably faster still, though I'm unclear on whether any of that applies to hardware:
https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-l...
buildroot does a nice job of tying a lot of this stuff together
It's so annoying how he starts out talking about boot speed, does the entire project, and then doesn't even mention how long it takes to boot up! All he ever says is "boots a bit quicker".
The photo of his monitor shows the kernel exec'ing his binary after 0.92 seconds.
You may also want to build and run busybox for your tiny userspace.
Other things you may want to experiment with is gen_init_cpio.c from linux kernel tree. It makes creating initramfs file structure easier from scripts.
And finally if sys/isolinux is also fun to use for minimal boot images.
"Butt Naked Linux" is how I read it.
I know it's off topic. I accept my downvotes.
I could end up stealing that for my own tinkering distro that never ends up releasing because it's awful.