1---
2title: "LXD: Containers for Human Beings"
3subtitle: "Docker's great and all, but I prefer the workflow of interacting with VMs"
4date: 2023-08-11T16:30:00-04:00
5categories:
6 - Technology
7tags:
8 - Sysadmin
9 - Containers
10 - VMs
11 - Docker
12 - LXD
13draft: true
14rss_only: false
15cover: ./cover.png
16---
17
18This is a blog post version of a talk I presented at both Ubuntu Summit 2022 and
19SouthEast LinuxFest 2023. The first was not recorded, but the second was and is
20on [SELF's PeerTube instance.][selfpeertube] I apologise for the terrible audio,
21but there's unfortunately nothing I can do about that. If you're already
22intimately familiar with the core concepts of VMs or containers, I would suggest
23skipping those respective sections. If you're vaguely familiar with either, I
24would recommend reading them because I do go a little bit in-depth.
25
26[selfpeertube]: https://peertube.linuxrocks.online/w/hjiTPHVwGz4hy9n3cUL1mq?start=1m
27
28{{< adm type="warn" >}}
29
30**Note:** Canonical has decided to [pull LXD out][lxd] from under the Linux
31Containers entity and instead continue development under the Canonical brand.
32The majority of the LXD creators and developers have congregated around a fork
33called [Incus.][inc] I'll be keeping a close eye on the project and intend to
34migrate as soon as there's an installable release.
35
36[lxd]: https://linuxcontainers.org/lxd/
37[inc]: https://linuxcontainers.org/incus/
38
39{{< /adm >}}
40
41## The benefits of VMs and containers
42
43- **Isolation:** you don't want to allow an attacker to infiltrate your email
44 server through your web application; the two should be completely separate
45 from each other and VMs/containers provide strong isolation guarantees.
46- **Flexibility:** <abbr title="Virtual Machines">VMs</abbr> and containers only
47 use the resources they've been given. If you tell the VM it has 200 MBs of
48 RAM, it's going to make do with 200 MBs of RAM and the kernel's <abbr
49 title="Out Of Memory">OOM</abbr> killer is going to have a fun time 🤠
50- **Portability:** once set up and configured, VMs and containers can mostly be
51 treated as closed boxes; as long as the surrounding environment of the new
52 host is similar to the previous in terms of communication (proxies, web
53 servers, etc.), they can just be picked up and dropped between various hosts
54 as necessary.
55- **Density:** applications are usually much lighter than the systems they're
56 running on, so it makes sense to run many applications on one system. VMs and
57 containers facilitate that without sacrificing security.
58- **Cleanliness:** VMs and containers are applications in black boxes. When
59 you're done with the box, you can just throw it away and most everything
60 related to the application is gone.
61
62## Virtual machines
63
64As the name suggests, Virtual Machines are all virtual; a hypervisor creates
65virtual disks for storage, virtual <abbr title="Central Processing
66Units">CPUs</abbr>, virtual <abbr title="Network Interface Cards">NICs</abbr>,
67virtual <abbr title="Random Access Memory">RAM</abbr>, etc. On top of the
68virtualised hardware, you have your kernel. This is what facilitates
69communication between the operating system and the (virtual) hardware. Above
70that is the operating system and all your applications.
71
72At this point, the stack is quite large; VMs aren't exactly lightweight, and
73this impacts how densely you can pack the host.
74
75I mentioned a "hypervisor" a minute ago. I've explained what hypervisors in
76general do, but there are actually two different kinds of hypervisor. They're
77creatively named **Type 1** and **Type 2**.
78
79### Type 1 hypervisors
80
81These run directly in the host kernel without an intermediary OS. A good example
82would be [KVM,][kvm] a **VM** hypervisor than runs in the **K**ernel. Type 1
83hypervisors can communicate directly with the host's hardware to allocate RAM,
84issue instructions to the CPU, etc.
85
86[debian]: https://debian.org
87[kvm]: https://www.linux-kvm.org
88[vb]: https://www.virtualbox.org/
89
90```kroki {type=d2,d2theme=flagship-terrastruct,d2sketch=true}
91hk: Host kernel
92hk.h: Type 1 hypervisor
93hk.h.k1: Guest kernel
94hk.h.k2: Guest kernel
95hk.h.k3: Guest kernel
96hk.h.k1.os1: Guest OS
97hk.h.k2.os2: Guest OS
98hk.h.k3.os3: Guest OS
99hk.h.k1.os1.app1: Many apps
100hk.h.k2.os2.app2: Many apps
101hk.h.k3.os3.app3: Many apps
102```
103
104### Type 2 hypervisors
105
106These run in userspace as an application, like [VirtualBox.][vb] Type 2
107hypervisors have to first go through the operating system, adding an additional
108layer to the stack.
109
110```kroki {type=d2,d2theme=flagship-terrastruct,d2sketch=true}
111hk: Host kernel
112hk.os: Host OS
113hk.os.h: Type 2 hypervisor
114hk.os.h.k1: Guest kernel
115hk.os.h.k2: Guest kernel
116hk.os.h.k3: Guest kernel
117hk.os.h.k1.os1: Guest OS
118hk.os.h.k2.os2: Guest OS
119hk.os.h.k3.os3: Guest OS
120hk.os.h.k1.os1.app1: Many apps
121hk.os.h.k2.os2.app2: Many apps
122hk.os.h.k3.os3.app3: Many apps
123```
124
125## Containers
126
127VMs use virtualisation to achieve isolation. Containers use **namespaces** and
128**cgroups**, technologies pioneered in the Linux kernel. By now, though, there
129are [equivalents for Windows] and possibly other platforms.
130
131[equivalents for Windows]: https://learn.microsoft.com/en-us/virtualization/community/team-blog/2017/20170127-introducing-the-host-compute-service-hcs
132
133**[Linux namespaces]** partition kernel resources like process IDs, hostnames,
134user IDs, directory hierarchies, network access, etc. This prevents one
135collection of processes from seeing or gaining access to data regarding another
136collection of processes.
137
138**[Cgroups]** limit, track, and isolate the hardware resource use of a
139collection of processes. If you tell a cgroup that it's only allowed to spawn
140500 child processes and someone executes a fork bomb, the fork bomb will expand
141until it hits that limit. The kernel will prevent it from spawning further
142children and you'll have to resolve the issue the same way you would with VMs:
143delete and re-create it, restore from a good backup, etc. You can also limit CPU
144use, the number of CPU cores it can access, RAM, disk use, and so on.
145
146[Linux namespaces]: https://en.wikipedia.org/wiki/Linux_namespaces
147[Cgroups]: https://en.wikipedia.org/wiki/Cgroups
148
149### Application containers
150
151The most well-known example of application container tech is probably
152[Docker.][docker] The goal here is to run a single application as minimally as
153possible inside each container. In the case of a single, statically-linked Go
154binary, a minimal Docker container might contain nothing more than the binary.
155If it's a Python application, you're more likely to use an [Alpine Linux image]
156and add your Python dependencies on top of that. If a database is required, that
157goes in a separate container. If you've got a web server to handle TLS
158termination and proxy your application, that's a third container. One cohesive
159system might require many Docker containers to function as intended.
160
161[docker]: https://docker.com/
162[Alpine Linux image]: https://hub.docker.com/_/alpine
163
164```kroki {type=d2,d2theme=flagship-terrastruct,d2sketch=true}
165Host kernel.Container runtime.c1: Container
166Host kernel.Container runtime.c2: Container
167Host kernel.Container runtime.c3: Container
168
169Host kernel.Container runtime.c1.One app
170Host kernel.Container runtime.c2.Few apps
171Host kernel.Container runtime.c3.Full OS.Many apps
172```
173
174### System containers
175
176One of the most well-known examples of system container tech is the subject of
177this post: LXD! Rather than containing a single application or a very small set
178of them, system containers are designed to house entire operating systems, like
179[Debian] or [Rocky Linux,][rocky] along with everything required for your
180application. Using our examples from above, a single statically-linked Go binary
181might run in a full Debian container, just like the Python application might.
182The database and webserver might go in _that same_ container.
183
184[Debian]: https://www.debian.org/
185[rocky]: https://rockylinux.org/
186
187You treat each container more like you would a VM, but you get the performance
188benefit of _not_ virtualising everything. Containers are _much_ lighter than any
189virtual machine.
190
191```kroki {type=d2,d2theme=flagship-terrastruct,d2sketch=true}
192hk: Host kernel
193hk.c1: Container
194hk.c2: Container
195hk.c3: Container
196hk.c1.os1: Full OS
197hk.c2.os2: Full OS
198hk.c3.os3: Full OS
199hk.c1.os1.app1: Many apps
200hk.c2.os2.app2: Many apps
201hk.c3.os3.app3: Many apps
202```
203
204## When to use which
205
206{{< adm type="warn" >}}
207**Warning:** this is my personal opinion. Please evaluate each technology and
208determine for yourself whether it's a suitable fit for your environment.
209{{< /adm >}}
210
211As far as I'm aware, VMs are your only option when you want to work with
212esoteric hardware or hardware you don't physically have on-hand. It's also your
213only option when you want to work with foreign operating systems: running Linux
214on Windows, Windows on Linux, or OpenBSD on a Mac all require virtualisation.
215Another reason to stick with VMs is for compliance purposes. Containers are
216still very new and some regulatory bodies require virtualisation because it's a
217decades-old and battle-tested isolation technique.
218
219{{< adm type="note" >}}
220See Drew DeVault's blog post [_In praise of qemu_][qemu] for a great use of VMs
221
222[qemu]: https://drewdevault.com/2022/09/02/2022-09-02-In-praise-of-qemu.html
223{{< /adm >}}
224
225Application containers are particularly popular for [microservices] and
226[reproducible builds,][repb] though I personally think [NixOS] is a better fit
227for the latter. App containers are also your only option if you want to use
228cloud platforms with extreme scaling capabilities like Google Cloud's App Engine
229standard environment or AWS's Fargate.
230
231[microservices]: https://en.wikipedia.org/wiki/Microservices
232[repb]: https://en.wikipedia.org/wiki/Reproducible_builds
233[NixOS]: https://nixos.org/
234
235 - When the app you want to run is _only_ distributed as a Docker container and
236 the maintainers adamantly refuse to support any other deployment method
237 - (Docker does run in LXD 😉)
238- System containers
239 - Anything not listed above 👍
240
241## Crash course to LXD
242
243### Installation
244
245{{< adm type="note" >}}
246
247**Note:** the instructions below say to install LXD using [Snap.][snap] I
248personally dislike Snap, but LXD is a Canonical product and they're doing their
249best to prmote it as much as possible. One of the first things the Incus project
250did was [rip out Snap support,][rsnap] so it will eventually be installable as a
251proper native package.
252
253[snap]: https://en.wikipedia.org/wiki/Snap_(software)
254[rsnap]: https://github.com/lxc/incus/compare/9579f65cd0f215ecd847e8c1cea2ebe96c56be4a...3f64077a80e028bb92b491d42037124e9734d4c7
255
256{{< /adm >}}
257
2581. Install snap following [Canonical's tutorial](https://earl.run/ZvUK)
259 - LXD is natively packaged for Arch and Alpine, but configuration can be a
260 massive headache.
2612. `sudo snap install lxd`
2623. `lxd init`
2634. `lxc image copy images:debian/11 local: --alias deb-11`
2645. `lxc launch deb-11 container-name`
2656. `lxc shell container-name`
266
267### Usage
268
269{install my URL shortener}
270
271[^1]: Docker containers on Windows and macOS actually run in a Linux VM.