Proxmox feels like a more apt comparison, as they both act like a controlplane for KVM virtual-machines and LXC containers across one or multiple hosts.
If you are interested in running kubernetes on top of incus, that is your kubernetes cluster nodes will be made up of KVM or LXC instances - I highly recommend the cluster-api provider incus https://github.com/lxc/cluster-api-provider-incus
This provider is really well done and maintained, including ClusterClass support and array of pre-built machine images for both KVM and LXC. It also supports pivoting the mgmt cluster on to a workload cluster, enabling the mgmt cluster to upgrade itself which is really cool.
You may have a point there that k8s is not meant for single machines but that’s not a hard rule, more like a “why would you want to” you can absolutely run single node Kubernetes.
Also strictly speaking incus is not a container nor vm runtime, it’s an orchestrator of those things.
Doesn't Vagrant spin up full VMs? Incus/LXD/LXC is about system containers. So like Docker, but with a full distro including init system running inside the container. They are much faster to spin up and have the best resource sharing possible.
Vagrant is not the right comparison against Incus for this use case. Vagrant is used to spin up VM or system container instances that are configured for software development and testing. But vagrant doesn't create those VMs or containers by itself. Instead, it depends on virtualization/container providers like VMware, Virtualbox, libvirt or lxc. In fact, you could create a provider plugin for vagrant to use Incus as its container/VM back end. (I couldn't find any, though.)
if your question is genuine, then the simple answer would be full system inside container of LXD/Incus (I prefer to use term VE for Virtual Environment to distinguish from barebone containers of Docker style).
The full system, means, all your standard stuff works in expected way - crons, systemd units, sshd, multiple users, your regular backup solutions and so on. As well, that "system" can be dumped/exported/backuped/snapshotted as a whole - very much similar like you do with your vSphere/Qemu or whatever you use in your datacenters as hypervisor.
Foreseeing the question - yep, you can run Docker inside LXD/Incus VEs. In practical terms, that makes much simple when you need to give some dev team (who are of course not anywhere known to do sane things) access to environment with Docker access (which in 99% cases means that host's root level access is exposed).
Instead of ephemeral containers, you have instances that are like VM (and incus can manage VM via qemu), so pretty much everything you would use a VM for, but if you do not need the kernel separation. It's more similar to FreeBSD jails than to docker.
Use cases are almost the same as Proxmox. You can orchestrate system containers or VMs. Proxmox runs lxc container images, while incus is built on top of lxc and has its own container images.
System vs Application containers: Both share virtualized kernels. Application containers usually run only a single application like web app (eg: OCI containers). System containers are more like VMs with systemd and multiple apps managed by it. Note: This differentiation is often ambiguous.
> Is this for people who want to run their own cloud provider, or that need to manage the infrastructure of org-owned VM's?
Yes, you could build a private cloud with it.
> When would you use this over k8s or serverless container runtimes like Fargate/Cloudrun?
You would use it when you need traditional application setups inside VMs or system containers, instead of containerized microservices.
I actually use Incus containers as host nodes for testing full fledged multinode K8s setups.
I know some webhosting provider that used one VM for every user. Now they moved to using this. Firstly low resource usage. If one uses ZFS or btrfs then one can save storage as in common bits are not duplicated across system containers. Note this is a system container - not traditional container. This one can be rebooted to get the previous state. It is not ephemeral.
System container tech like Incus powers efficient Linux virtualization environments for developers, so that you can have only one VM but many "machines". OrbStack machines on macOS work like this, and the way WSL works is similar (one VM and one Linux kernel, many guests sharing that kernel via system containers).
Just in case - I'm using LXD inside my WSL and it working great. BTRFS backed storage via loopfile, saves $$$.
For others, why it may be useful in regular sysadmin job:
* say doing Ansible scripting against LOCAL network is hell amount of time faster than against 300+ ms remote machines
* note that because you can use VE snapshots, it's very easy to ensure your playbook works fine without guessing what have you modified when testing things - just do rollback to "clean" state and start over
* creating test MariaDB 3 nodes cluster - easy peasy
* multiple distros available - need to debug Haproxy from say Rocky 8 linux? Check!
Firecracker has some elements in common-- namely images that are to run on Firecracker have a non-standard init system so that they can boot more quickly than machines that are dealing with real hardware and a wider variety of use cases. That's also typically used for the guest VMs that host containers for systems like WSL and OrbStack.
But Firecracker is fundamentally different because it has a different purpose: Firecracker is about offering VM-based isolation for systems that have container-like ephemerality in multitenant environments, especially the cloud. So when you use Firecracker, each system has its own kernel running under its own paravirtualized hardware.
With OrbStack and WSL, you have only one kernel for all of your "guests" (which are container guests, rather tha hardware paravirtualized guests). In exchange you're working with something that's simpler in some ways, more efficient, has less resource contention, etc. And it's easier to share resources between containers dynamically than across VMs, so it's very easy to run 10 "machines" but only allocate 4GB of RAM or whatever, and have it shared freely between them with little overhead. They can also share Unix sockets (like the socket for Docker or a Kubernetes runtime) directly as files, since they share a kernel-- no need for some kind of HTTP-based socket forwarding across virtualized network devices.
I imagine this is nice for many use cases, but as you can imagine, it's especially nice for local development. :)
How do you handle updating the machine that Incus itself runs on? I imagine you have to be super careful not to introduce any breakage, because then all the VMs/containers go down.
What about kernel updates that require reboots? I have heard of ksplice/kexec, but I have never seen them used anywhere.
It is a built-in functionality [1] and requires no extra orchestration. In a cluster setup, you would be using virtualized storage (ceph based) and virtualized network (ovn). You can replace a container/VM on one host with another on a different host with the same storage volumes, network and address. This is what k8s does with pod migrations too (edit: except the address).
There are a couple of differences though. The first is the pet vs cattle treatment of containers by Incus and k8s respectively. Incus tries to resurrect dead containers as faithfully as possible. This means that Incus treats container crashes like system crashes, and its recovery involves systemd bootup inside the container (kernel too in case of VMs). This is what accounts for the delay. K8s on the other hand, doesn't care about dead containers/pods at all. It just creates another pod, likely with a different address and expects it to handle the interruption.
Another difference is the orchestration mechanism behind this. K8s, as you may be aware, uses control loops on controller nodes to detect the crash and initiate the recovery. The recovery is mediated by the kubelets on the worker nodes. Incus seems to have the orchestrator on all nodes. They take decisions based on consensus and manage the recovery process themselves.
> and address. This is what k8s does with pod migrations too.
That's not true of Pods; each Pod has its own distinct network identity. You're correct about the network, though, since AFAIK Service and Pod CIDR are fixed for the lifespan of the k8s cluster
You spoke to it further down, but guarded it with "likely" and I can say with certainty that it's not likely, it unconditionally does. That's not to say address re-use isn't possible over a long enough time horizon, but that bookkeeeping is delegated to the CNI
---
Your "dead container" one also has some nuance, in that kubelet will for sure restart a failed container, in place, with the same network identity. When fresh identity comes into play is if the Node fails, or the control loop determines something in the Pod's configuration has changed (env-vars, resources, scheduling constraints, etc) in which case it will be recreated, even if by coincidence on the same Node
> I can say with certainty that it's not likely, it unconditionally does. That's not to say address re-use isn't possible over a long enough time horizon, but that bookkeeeping is delegated to the CNI
You are 100% wrong then. The kube-ovn CNI enables static address assignment and "sticky" IPAM on both pods and kubevirt vms.
Heh, I knew I was going to get in trouble since the CNI could do whatever it likes, but felt safe due to Pods having mostly random identities. But at that second I had forgotten about StatefulSets, which I agree with your linked CNI's opinion would actually be a great candidate for static address assignment
Sorry for the lapse and I'll try to be more careful when using "unconditional" to describe pluggable software
I agree with everything you pointed out. They were what I had in my mind too. However, I avoided those points on purpose for the sake of brevity. It was getting too long winded and convoluted for my liking. Thanks for adding a separate clarification, though.
the features worth mentioning imho are the different storage backends and their features. Using btrfs, lvm or zfs there is some level of support of thin copy provisioning and snapshotting. I believe btrfs/zfs have parity in terms of supported operations. Cheap snapshots and provisioning of both containers and VMs using the same tool is pretty awesome.
I personally use lxd for running my homelab VMs and containers
Yes, it uses QEMU under the hood for VMs and runs LXC containers. But also, since recently, you can run docker images in it. Very handy, especially since it has 1st class remote support, meaning you can install only the incus client and when doing `incus launch` or whatever, it will transparently start the container/vm on your remote host
Nothing about resource (net, io, disk, cpu) isolation, limits, priorities, or guarantees. Not the same as a type 1 hypervisor. These qualities are needed to run things safely and predictably in the real world™, at scale. Also, accounting and multitenancy if it's going to be used as some sort VAR or VPS offering.
Fun fact, Incus is being used as underlying infrastructure for the NorthSec CTF, i.e. in an "as hostile as it can get" environment. If you have close to a hundred teams of hackers on your systems trying to break stuff, I think it speaks for Incus and its capabilities regarding isolation and limits.
In case you are interested, Zabbly has some interesting behind-the-scenes on Youtube (not affiliated).
If being used in a CTF counts, then running latest docker with no extra privilege and non-root user on a reasonably up-to-date kernel meets the definition of secure I think. At least for what I have seen, this kind of infrastructure is pretty common in CTF.
Short answer: No. Long answer: Depends upon what you use lxc for.
Incus is not a replacement for lxc. It's an alternative for LXD (LXD project is still active). Both Incus and LXD are built upon liblxc (the library version of lxc) and provide a higher level user interface than lxc (eg: projects, cloud-init support, etc). However, lxc gives you fine grained control over container options (this is sort of like flatpak and bubblewrap).
So, if you don't need the fine grained control of lxc, Incus may be a more ergonomic solution.
PS: Confusingly enough, LXD's CLI is also named lxc.
Is there some kind of Terraform/Pulumi integration to make it easy to deploy stuff to some VM running Incus for my deployments? Or I'm missing the point of what Incus is for?
I'm a Pulumi user myself, and I haven't seen a Pulumi provider for Incus yet. Once I get further into my Incus experiments, if someone hasn't made an Incus provider yet, I'll probably go through the TF provider conversion process.
Incus is like a cloud management software - especially in cluster mode. It has management API like many cloud services. So, yes, there's a terraform provider for Incus, which can be used to configure and provision instances. Guest setup can be managed using cloud-init. Ansible is also an alternative option for this.
A little bit of context about where Incus came from:
https://lwn.net/Articles/940684/
So it looks like a Proxmox alternative, this [0] goes into some reasons to switch. Main selling point seems to be fully OSS and no enterprise version.
[0]: https://tadeubento.com/2024/replace-proxmox-with-incus-lxd/
It’s more like a Kubernetes alternative
Proxmox feels like a more apt comparison, as they both act like a controlplane for KVM virtual-machines and LXC containers across one or multiple hosts.
If you are interested in running kubernetes on top of incus, that is your kubernetes cluster nodes will be made up of KVM or LXC instances - I highly recommend the cluster-api provider incus https://github.com/lxc/cluster-api-provider-incus
This provider is really well done and maintained, including ClusterClass support and array of pre-built machine images for both KVM and LXC. It also supports pivoting the mgmt cluster on to a workload cluster, enabling the mgmt cluster to upgrade itself which is really cool.
I was surprised to come across this provider by chance as for some reason it's not listed on the CAPI documentation provider list https://cluster-api.sigs.k8s.io/reference/providers
Not really, Kubernetes does a lot of different things that are out of scope for incus or lxd or docker compose for that matter or any hypervisor or …
like what? I'd love to hear some examples of things Kubernetes does that incus doesn't at this point
Service discovery?
I'm sure you could probably manually hook up Incus to something like Consul, but it would be more effort than it's worth.
One is for cluster orchestration the other is a single machine container/vm runtime.
https://linuxcontainers.org/incus/docs/main/explanation/clus...
https://linuxcontainers.org/incus/docs/main/explanation/clus...
You may have a point there that k8s is not meant for single machines but that’s not a hard rule, more like a “why would you want to” you can absolutely run single node Kubernetes.
Also strictly speaking incus is not a container nor vm runtime, it’s an orchestrator of those things.
Incus is great when developing ansible playbooks. The main benefit for me over docker/podman is systemd works out of the box in incus containers.
I have never actually tested it, but my understanding is that systemd also works out of the box inside of podman containers: https://docs.podman.io/en/latest/markdown/podman-run.1.html#...
Not to mention the easy to use web UI.
What makes it better than Vagrant for this use-case?
Doesn't Vagrant spin up full VMs? Incus/LXD/LXC is about system containers. So like Docker, but with a full distro including init system running inside the container. They are much faster to spin up and have the best resource sharing possible.
Vagrant has a provider plugin for lxc. Please check my reply to the parent commenter for more explanation.
Vagrant is not the right comparison against Incus for this use case. Vagrant is used to spin up VM or system container instances that are configured for software development and testing. But vagrant doesn't create those VMs or containers by itself. Instead, it depends on virtualization/container providers like VMware, Virtualbox, libvirt or lxc. In fact, you could create a provider plugin for vagrant to use Incus as its container/VM back end. (I couldn't find any, though.)
https://github.com/hashicorp/vagrant/blob/v2.4.7/LICENSE for one thing
The only tool I found which allows to easily spin up pre-configured VMs without any gui hassle
I went through the online tutorial, but I'm not really seeing how it's different from docker?
if your question is genuine, then the simple answer would be full system inside container of LXD/Incus (I prefer to use term VE for Virtual Environment to distinguish from barebone containers of Docker style).
The full system, means, all your standard stuff works in expected way - crons, systemd units, sshd, multiple users, your regular backup solutions and so on. As well, that "system" can be dumped/exported/backuped/snapshotted as a whole - very much similar like you do with your vSphere/Qemu or whatever you use in your datacenters as hypervisor.
Foreseeing the question - yep, you can run Docker inside LXD/Incus VEs. In practical terms, that makes much simple when you need to give some dev team (who are of course not anywhere known to do sane things) access to environment with Docker access (which in 99% cases means that host's root level access is exposed).
Instead of ephemeral containers, you have instances that are like VM (and incus can manage VM via qemu), so pretty much everything you would use a VM for, but if you do not need the kernel separation. It's more similar to FreeBSD jails than to docker.
It's a difference between system containers and application containers.
LXC containers used in incus run their own init, they act more like a VM.
However incus can also execute actual VMs via libvirt and since recently even OCI containers like docker.
Can someone explain the usecase for this?
Is this for people who want to run their own cloud provider, or that need to manage the infrastructure of org-owned VM's?
When would you use this over k8s or serverless container runtimes like Fargate/Cloudrun?
> Can someone explain the usecase for this?
Use cases are almost the same as Proxmox. You can orchestrate system containers or VMs. Proxmox runs lxc container images, while incus is built on top of lxc and has its own container images.
System vs Application containers: Both share virtualized kernels. Application containers usually run only a single application like web app (eg: OCI containers). System containers are more like VMs with systemd and multiple apps managed by it. Note: This differentiation is often ambiguous.
> Is this for people who want to run their own cloud provider, or that need to manage the infrastructure of org-owned VM's?
Yes, you could build a private cloud with it.
> When would you use this over k8s or serverless container runtimes like Fargate/Cloudrun?
You would use it when you need traditional application setups inside VMs or system containers, instead of containerized microservices.
I actually use Incus containers as host nodes for testing full fledged multinode K8s setups.
I know some webhosting provider that used one VM for every user. Now they moved to using this. Firstly low resource usage. If one uses ZFS or btrfs then one can save storage as in common bits are not duplicated across system containers. Note this is a system container - not traditional container. This one can be rebooted to get the previous state. It is not ephemeral.
System container tech like Incus powers efficient Linux virtualization environments for developers, so that you can have only one VM but many "machines". OrbStack machines on macOS work like this, and the way WSL works is similar (one VM and one Linux kernel, many guests sharing that kernel via system containers).
Just in case - I'm using LXD inside my WSL and it working great. BTRFS backed storage via loopfile, saves $$$.
For others, why it may be useful in regular sysadmin job:
* say doing Ansible scripting against LOCAL network is hell amount of time faster than against 300+ ms remote machines
* creating test MariaDB 3 nodes cluster - easy peasy* multiple distros available - need to debug Haproxy from say Rocky 8 linux? Check!
Thanks -- though I'm not sure I fully grok how this is different than something like Firecracker?
Firecracker has some elements in common-- namely images that are to run on Firecracker have a non-standard init system so that they can boot more quickly than machines that are dealing with real hardware and a wider variety of use cases. That's also typically used for the guest VMs that host containers for systems like WSL and OrbStack.
But Firecracker is fundamentally different because it has a different purpose: Firecracker is about offering VM-based isolation for systems that have container-like ephemerality in multitenant environments, especially the cloud. So when you use Firecracker, each system has its own kernel running under its own paravirtualized hardware.
With OrbStack and WSL, you have only one kernel for all of your "guests" (which are container guests, rather tha hardware paravirtualized guests). In exchange you're working with something that's simpler in some ways, more efficient, has less resource contention, etc. And it's easier to share resources between containers dynamically than across VMs, so it's very easy to run 10 "machines" but only allocate 4GB of RAM or whatever, and have it shared freely between them with little overhead. They can also share Unix sockets (like the socket for Docker or a Kubernetes runtime) directly as files, since they share a kernel-- no need for some kind of HTTP-based socket forwarding across virtualized network devices.
I imagine this is nice for many use cases, but as you can imagine, it's especially nice for local development. :)
There's no particular usecase, though I do know of a company whose entire infrastructure is maintained within incus.
I personally use it mostly for deploying a bunch of system containers and some OCI containers.
But anyone who uses LXC, LXD, docker, libvirt, qemu etc. could potentially be interested in Incus.
Incus is just an LXD fork btw, developed by Stephane Graber.
Who also developed LXD and contributed to LXC. I wouldn’t say it’s just a fork but a continuation of the project without Canonical.
You're right, I should've worded it differently.
>a continuation of the project without Canonical.
This being a big plus considering Canonical just outed itself as a safe space for pedophiles. Not being a pedo-bar is a good thing.
I first learned about this because colima supports it: https://github.com/abiosoft/colima#incus
How do you handle updating the machine that Incus itself runs on? I imagine you have to be super careful not to introduce any breakage, because then all the VMs/containers go down.
What about kernel updates that require reboots? I have heard of ksplice/kexec, but I have never seen them used anywhere.
As with any such system, you need a spare box. Upgrade the spare, move the clients to it, upgrade the original.
But then the clients have downtime while they’re being moved.
I don't know about Incus, but on ProxMox the downtime when moving a VM is around 200 ms.
Isn't that the exact problem that k8s workloads solve by scaling onto new nodes first etc? No downtime required.
Right but incus is not k8s. You can stand up spares and switch traffic, but it’s not built in functionality and requires extra orchestration.
It is a built-in functionality [1] and requires no extra orchestration. In a cluster setup, you would be using virtualized storage (ceph based) and virtualized network (ovn). You can replace a container/VM on one host with another on a different host with the same storage volumes, network and address. This is what k8s does with pod migrations too (edit: except the address).
There are a couple of differences though. The first is the pet vs cattle treatment of containers by Incus and k8s respectively. Incus tries to resurrect dead containers as faithfully as possible. This means that Incus treats container crashes like system crashes, and its recovery involves systemd bootup inside the container (kernel too in case of VMs). This is what accounts for the delay. K8s on the other hand, doesn't care about dead containers/pods at all. It just creates another pod, likely with a different address and expects it to handle the interruption.
Another difference is the orchestration mechanism behind this. K8s, as you may be aware, uses control loops on controller nodes to detect the crash and initiate the recovery. The recovery is mediated by the kubelets on the worker nodes. Incus seems to have the orchestrator on all nodes. They take decisions based on consensus and manage the recovery process themselves.
[1] https://linuxcontainers.org/incus/docs/main/howto/cluster_ma...
> and address. This is what k8s does with pod migrations too.
That's not true of Pods; each Pod has its own distinct network identity. You're correct about the network, though, since AFAIK Service and Pod CIDR are fixed for the lifespan of the k8s cluster
You spoke to it further down, but guarded it with "likely" and I can say with certainty that it's not likely, it unconditionally does. That's not to say address re-use isn't possible over a long enough time horizon, but that bookkeeeping is delegated to the CNI
---
Your "dead container" one also has some nuance, in that kubelet will for sure restart a failed container, in place, with the same network identity. When fresh identity comes into play is if the Node fails, or the control loop determines something in the Pod's configuration has changed (env-vars, resources, scheduling constraints, etc) in which case it will be recreated, even if by coincidence on the same Node
> I can say with certainty that it's not likely, it unconditionally does. That's not to say address re-use isn't possible over a long enough time horizon, but that bookkeeeping is delegated to the CNI
You are 100% wrong then. The kube-ovn CNI enables static address assignment and "sticky" IPAM on both pods and kubevirt vms.
https://kubeovn.github.io/docs/v1.12.x/en/guide/static-ip-ma...
Heh, I knew I was going to get in trouble since the CNI could do whatever it likes, but felt safe due to Pods having mostly random identities. But at that second I had forgotten about StatefulSets, which I agree with your linked CNI's opinion would actually be a great candidate for static address assignment
Sorry for the lapse and I'll try to be more careful when using "unconditional" to describe pluggable software
All good and i'll cheers you on the composability of k8s for sure
I agree with everything you pointed out. They were what I had in my mind too. However, I avoided those points on purpose for the sake of brevity. It was getting too long winded and convoluted for my liking. Thanks for adding a separate clarification, though.
the features worth mentioning imho are the different storage backends and their features. Using btrfs, lvm or zfs there is some level of support of thin copy provisioning and snapshotting. I believe btrfs/zfs have parity in terms of supported operations. Cheap snapshots and provisioning of both containers and VMs using the same tool is pretty awesome.
I personally use lxd for running my homelab VMs and containers
What can this work with? It says „Containers and VMs“ - I guess that’s LXCs and QEMU VMs?
Yes, it uses QEMU under the hood for VMs and runs LXC containers. But also, since recently, you can run docker images in it. Very handy, especially since it has 1st class remote support, meaning you can install only the incus client and when doing `incus launch` or whatever, it will transparently start the container/vm on your remote host
Nothing about resource (net, io, disk, cpu) isolation, limits, priorities, or guarantees. Not the same as a type 1 hypervisor. These qualities are needed to run things safely and predictably in the real world™, at scale. Also, accounting and multitenancy if it's going to be used as some sort VAR or VPS offering.
Fun fact, Incus is being used as underlying infrastructure for the NorthSec CTF, i.e. in an "as hostile as it can get" environment. If you have close to a hundred teams of hackers on your systems trying to break stuff, I think it speaks for Incus and its capabilities regarding isolation and limits.
In case you are interested, Zabbly has some interesting behind-the-scenes on Youtube (not affiliated).
If being used in a CTF counts, then running latest docker with no extra privilege and non-root user on a reasonably up-to-date kernel meets the definition of secure I think. At least for what I have seen, this kind of infrastructure is pretty common in CTF.
I would guess <https://www.youtube.com/watch?v=7A1yrLRNIp0> is a good starting point "Looking at the NorthSec infrastructure" from April, 2024
The YT description also points to https://github.com/zabbly/incus
Took a few seconds of googling to find this: https://linuxcontainers.org/incus/docs/main/reference/instan...
Incus supports Qemu/KVM VMs. And KVM is arguably a Type 1 hypervisor since it's part of the Linux kernel. So I guess it qualifies?
Should lxc user migrate to incus?
Short answer: No. Long answer: Depends upon what you use lxc for.
Incus is not a replacement for lxc. It's an alternative for LXD (LXD project is still active). Both Incus and LXD are built upon liblxc (the library version of lxc) and provide a higher level user interface than lxc (eg: projects, cloud-init support, etc). However, lxc gives you fine grained control over container options (this is sort of like flatpak and bubblewrap).
So, if you don't need the fine grained control of lxc, Incus may be a more ergonomic solution.
PS: Confusingly enough, LXD's CLI is also named lxc.
an lxd user should
LXD is actually pronounced “lex-d” by the community and similarly, LXC is “lex-c”.
Not to be confused with the cirrus7 incus[0], which are fanless PC models based on the ASRock DeskMini series that I'm using right now.
[0] https://www.cirrus7.com/produkte/cirrus7-incus/
or ear bone: https://en.m.wikipedia.org/wiki/Incus
Is there some kind of Terraform/Pulumi integration to make it easy to deploy stuff to some VM running Incus for my deployments? Or I'm missing the point of what Incus is for?
There is a Terraform provider that is actively maintained, in addition to Ansible integration. https://linuxcontainers.org/incus/docs/main/third_party/
I'm a Pulumi user myself, and I haven't seen a Pulumi provider for Incus yet. Once I get further into my Incus experiments, if someone hasn't made an Incus provider yet, I'll probably go through the TF provider conversion process.
Incus is like a cloud management software - especially in cluster mode. It has management API like many cloud services. So, yes, there's a terraform provider for Incus, which can be used to configure and provision instances. Guest setup can be managed using cloud-init. Ansible is also an alternative option for this.
Or even one of each, since cloud-init has an ansible hook: https://cloudinit.readthedocs.io/en/latest/reference/modules... for something like ansible-pull behavior
Yes. Thanks! The entire cloud computing ecosystem is a hot mess of numerous tools with countless interdependencies.
You could use cloud-init