1---
2name: monitoring-with-munin
3description: Deploys and manages Munin monitoring across servers. Use when setting up munin-node on a host, writing munin plugins, adding nodes to a master, configuring alerts, or diagnosing system issues using munin data. Also use when the user mentions munin, monitoring, or graphing server metrics.
4license: GPL-3.0-or-later
5metadata:
6 author: Amolith <amolith@secluded.site>
7---
8
9If the user has an existing Munin setup they want you to work with, ask them for specifics: where the master is, how nodes are connected (Tailscale, direct IP, SSH tunnel), and what OS the target hosts run.
10
11## Installing munin-node
12
13### Debian/Ubuntu
14
15```bash
16apt-get install -y munin-node
17munin-node-configure --shell | sh -x # auto-detect and symlink plugins
18systemctl enable --now munin-node
19```
20
21### Arch Linux
22
23```bash
24pacman -S --noconfirm munin-node
25# Net::CIDR is often unavailable on Arch; use regex allow instead of cidr_allow
26munin-node-configure --shell | sh -x
27systemctl enable --now munin-node
28```
29
30## Configuring munin-node
31
32Config lives at `/etc/munin/munin-node.conf`. Key directives:
33
34```ini
35host * # bind to all interfaces
36port 4949
37allow ^127\.0\.0\.1$ # regex against connecting IP
38allow ^::1$
39allow 100\.107\.78\.23 # master's IP (unanchored works too)
40cidr_allow 100.107.78.23/32 # alternative (needs perl Net::CIDR)
41```
42
43The `allow` directive uses Perl regexes matched against the client IP. When the connection arrives as IPv6-mapped IPv4 (`::ffff:A.B.C.D`), the anchored regex `^A\.B\.C\.D$` won't match. Use an **unanchored** regex like `A\.B\.C\.D` to handle both forms, or add an explicit `allow ^::ffff:A\.B\.C\.D$`.
44
45On Arch Linux, `Net::CIDR` is typically unavailable (only `Net::CIDR::Lite` exists in pacman). If `cidr_allow` causes `Can't locate Net/CIDR.pm` errors, remove all `cidr_allow` lines and use `allow` regexes instead.
46
47After changing config: `systemctl restart munin-node`
48
49### Firewall
50
51If UFW is present, restrict port 4949 to the master only:
52
53```bash
54ufw allow from <MASTER_TS_IP> to any port 4949 comment 'munin master'
55ufw deny in 4949 comment 'deny munin from everyone else'
56```
57
58Order matters — allow must come before deny.
59
60## Adding a node to the master
61
62Append to `/etc/munin/munin.conf` on the master:
63
64```ini
65[groupname;hostname]
66 address <node_tailscale_ip>
67 use_node_name yes
68```
69
70Group names organize the web UI — use logical names like `nixnet`, `exe.xyz`, and `personal`.
71
72Seed data immediately: `su - munin --shell=/bin/bash -c '/usr/bin/munin-cron'`
73
74### Verifying connectivity
75
76From the master, test the node protocol:
77
78```bash
79# Basic test (non-multigraph plugins only)
80echo 'quit' | nc -w3 <node_ip> 4949
81
82# Full test including multigraph plugins
83{ sleep 1; echo 'cap multigraph'; sleep 1; echo 'list'; sleep 1; echo 'quit'; } | nc -w5 <node_ip> 4949
84```
85
86A working node responds with `# munin node at <hostname>` followed by the plugin list.
87
88## Installing third-party plugins
89
90Third-party plugins (including our custom ones) go in `/usr/local/munin/lib/plugins/`, **not** the distribution plugin directory (`/usr/share/munin/plugins/` on Debian, `/usr/lib/munin/plugins/` on Arch). This avoids package updates overwriting custom plugins.
91
92```bash
93mkdir -p /usr/local/munin/lib/plugins
94cp my_plugin /usr/local/munin/lib/plugins/
95chmod +x /usr/local/munin/lib/plugins/my_plugin
96```
97
98Create symlinks in `/etc/munin/plugins/` manually:
99
100```bash
101# Simple plugin
102ln -s /usr/local/munin/lib/plugins/my_plugin /etc/munin/plugins/my_plugin
103# Wildcard plugin
104ln -s /usr/local/munin/lib/plugins/my_plugin_ /etc/munin/plugins/my_plugin_instance
105```
106
107Auto-detection with `munin-node-configure` requires `--libdir`:
108
109```bash
110munin-node-configure --libdir /usr/local/munin/lib/plugins --shell
111```
112
113Note: `munin-node-configure` runs `autoconf`/`suggest` as the munin user. Plugins that need root (e.g. smartctl) will hang. For those, run `autoconf` and `suggest` manually as root and create symlinks by hand.
114
115## Writing plugins
116
117A plugin is any executable in `/etc/munin/plugins/` (usually a symlink from the plugin library directory). It must handle two invocations:
118
119```bash
120./plugin config # print graph metadata
121./plugin # print values
122```
123
124### Minimal shell plugin
125
126```sh
127#!/bin/sh
128if [ "${1:-}" = "config" ]; then
129 echo "graph_title My metric"
130 echo "graph_vlabel units"
131 echo "graph_category system"
132 echo "myfield.label Some value"
133 exit 0
134fi
135echo "myfield.value $(cat /some/source)"
136```
137
138### Field names
139
140Must match `^[A-Za-z_][A-Za-z0-9_]*$`. Sanitize dynamic names:
141
142```sh
143field=$(echo "$name" | sed 's/[^A-Za-z0-9_]/_/g; s/^[0-9]/_/')
144```
145
146### Data types
147
148- `GAUGE` (default): absolute value, plotted as-is
149- `COUNTER`/`DERIVE`: ever-increasing counter; munin computes rate per second. Use `DERIVE` with `.min 0` to avoid spikes on counter reset.
150
151### Multigraph plugins
152
153Output multiple graphs from one plugin by emitting `multigraph <name>` lines before each graph's config/values. Multigraph plugins are hidden from `list` output unless the client sends `cap multigraph` first.
154
155### Plugin configuration
156
157Per-plugin settings go in `/etc/munin/plugin-conf.d/<name>`:
158
159```ini
160[plugin_name]
161 user root
162 env.configfile /path/to/config
163 env.statuses available away chat xa
164```
165
166### Testing
167
168```bash
169munin-run <plugin_name> config # test config output
170munin-run <plugin_name> # test value output
171```
172
173Note: Debian's munin-node ships with `ProtectHome=yes` in systemd, which hides `/home/` from the entire process namespace regardless of user; `user root` in plugin-conf.d doesn't help. See [ProtectHome](#protecthome-and-home-access) for workarounds.
174
175After installing or removing plugins: `systemctl restart munin-node`
176
177## ProtectHome and /home/ access
178
179`ProtectHome=yes` mounts `/home/`, `/root`, `/run/user` as empty tmpfs. No user can see through it.
180
181Fix with
182
183```bash
184sudo mkdir -p /etc/systemd/system/munin-node.service.d
185printf '[Service]\nProtectHome=read-only\n' | sudo tee /etc/systemd/system/munin-node.service.d/override.conf
186sudo systemctl daemon-reload && sudo systemctl restart munin-node
187```
188
189Alternatives: `ProtectHome=tmpfs` + `BindReadOnlyPaths=` for selective exposure, or move data outside `/home/`.
190
191**Pitfall**: even with `ProtectHome=read-only`, a 750 home directory blocks the `munin` user from traversing the path. Use `user root` in plugin-conf.d for such cases.
192
193## Alerting
194
195Alerts are configured in `/etc/munin/munin.conf` on the master. A contact is a command that receives alert text on stdin.
196
197```ini
198contact.ntfy.command /usr/local/bin/munin-ntfy-alert
199contact.ntfy.always_send warning critical
200contact.ntfy.text ${var:host} :: ${var:graph_title} :: ${loop<, >:wfields WARNING ${var:label}=${var:value}} ${loop<, >:cfields CRITICAL ${var:label}=${var:value}}
201```
202
203### Thresholds
204
205Override per host or globally. The `memory` plugin uses percentages:
206
207```ini
208[groupname;hostname]
209 memory.warning 80
210 memory.critical 90
211```
212
213Plugin-specific fields use `pluginname.fieldname.warning` syntax.
214
215### Alert variables
216
217| Variable | Description |
218| -------------------------- | ----------------------------------------------- |
219| `${var:host}` | Node hostname |
220| `${var:graph_title}` | Plugin's graph title |
221| `${var:worst}` | Worst status: OK, WARNING, CRITICAL, UNKNOWN |
222| `${var:worstid}` | Numeric: 0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN |
223| `${loop<sep>:wfields ...}` | Iterate warning fields |
224| `${loop<sep>:cfields ...}` | Iterate critical fields |
225| `${var:label}` | Field label (inside loop) |
226| `${var:value}` | Field value (inside loop) |
227
228## Querying data programmatically
229
230RRD files on the master are queryable:
231
232```bash
233rrdtool fetch /var/lib/munin/group/host-plugin-field-g.rrd AVERAGE --start -1h
234```
235
236The munin-node protocol is also directly queryable over TCP:
237
238```bash
239{ echo 'fetch memory'; sleep 1; echo 'quit'; } | nc <node_ip> 4949
240```
241
242## Reference
243
244- **Plugin gallery**: https://gallery.munin-monitoring.org/
245- **Full docs**: https://guide.munin-monitoring.org/en/latest/
246- **Writing plugins**: See [writing-plugins.md](references/writing-plugins.md)