1---
2name: monitoring-with-munin
3description: Deploys and manages Munin monitoring across servers. Use when setting up munin-node on a host, writing munin plugins, adding nodes to a master, configuring alerts, or diagnosing system issues using munin data. Also use when the user mentions munin, monitoring, or graphing server metrics.
4user-invocable: true
5license: LicenseRef-MutuaL-1.2
6metadata:
7 author: Amolith <amolith@secluded.site>
8---
9
10If the user has an existing Munin setup they want you to work with, ask them for specifics: where the master is, how nodes are connected (Tailscale, direct IP, SSH tunnel), and what OS the target hosts run.
11
12## Installing munin-node
13
14### Debian/Ubuntu
15
16```bash
17apt-get install -y munin-node
18munin-node-configure --shell | sh -x # auto-detect and symlink plugins
19systemctl enable --now munin-node
20```
21
22### Arch Linux
23
24```bash
25pacman -S --noconfirm munin-node
26# Net::CIDR is often unavailable on Arch; use regex allow instead of cidr_allow
27munin-node-configure --shell | sh -x
28systemctl enable --now munin-node
29```
30
31## Configuring munin-node
32
33Config lives at `/etc/munin/munin-node.conf`. Key directives:
34
35```ini
36host * # bind to all interfaces
37port 4949
38allow ^127\.0\.0\.1$ # regex against connecting IP
39allow ^::1$
40allow 100\.107\.78\.23 # master's IP (unanchored works too)
41cidr_allow 100.107.78.23/32 # alternative (needs perl Net::CIDR)
42```
43
44The `allow` directive uses Perl regexes matched against the client IP. When the connection arrives as IPv6-mapped IPv4 (`::ffff:A.B.C.D`), the anchored regex `^A\.B\.C\.D$` won't match. Use an **unanchored** regex like `A\.B\.C\.D` to handle both forms, or add an explicit `allow ^::ffff:A\.B\.C\.D$`.
45
46On Arch Linux, `Net::CIDR` is typically unavailable (only `Net::CIDR::Lite` exists in pacman). If `cidr_allow` causes `Can't locate Net/CIDR.pm` errors, remove all `cidr_allow` lines and use `allow` regexes instead.
47
48After changing config: `systemctl restart munin-node`
49
50### Firewall
51
52If UFW is present, restrict port 4949 to the master only:
53
54```bash
55ufw allow from <MASTER_TS_IP> to any port 4949 comment 'munin master'
56ufw deny in 4949 comment 'deny munin from everyone else'
57```
58
59Order matters — allow must come before deny.
60
61## Adding a node to the master
62
63Append to `/etc/munin/munin.conf` on the master:
64
65```ini
66[groupname;hostname]
67 address <node_tailscale_ip>
68 use_node_name yes
69```
70
71Group names organize the web UI — use logical names like `nixnet`, `exe.xyz`, and `personal`.
72
73Seed data immediately: `su - munin --shell=/bin/bash -c '/usr/bin/munin-cron'`
74
75### Verifying connectivity
76
77From the master, test the node protocol:
78
79```bash
80# Basic test (non-multigraph plugins only)
81echo 'quit' | nc -w3 <node_ip> 4949
82
83# Full test including multigraph plugins
84{ sleep 1; echo 'cap multigraph'; sleep 1; echo 'list'; sleep 1; echo 'quit'; } | nc -w5 <node_ip> 4949
85```
86
87A working node responds with `# munin node at <hostname>` followed by the plugin list.
88
89## Installing third-party plugins
90
91Third-party plugins (including our custom ones) go in `/usr/local/munin/lib/plugins/`, **not** the distribution plugin directory (`/usr/share/munin/plugins/` on Debian, `/usr/lib/munin/plugins/` on Arch). This avoids package updates overwriting custom plugins.
92
93```bash
94mkdir -p /usr/local/munin/lib/plugins
95cp my_plugin /usr/local/munin/lib/plugins/
96chmod +x /usr/local/munin/lib/plugins/my_plugin
97```
98
99Create symlinks in `/etc/munin/plugins/` manually:
100
101```bash
102# Simple plugin
103ln -s /usr/local/munin/lib/plugins/my_plugin /etc/munin/plugins/my_plugin
104# Wildcard plugin
105ln -s /usr/local/munin/lib/plugins/my_plugin_ /etc/munin/plugins/my_plugin_instance
106```
107
108Auto-detection with `munin-node-configure` requires `--libdir`:
109
110```bash
111munin-node-configure --libdir /usr/local/munin/lib/plugins --shell
112```
113
114Note: `munin-node-configure` runs `autoconf`/`suggest` as the munin user. Plugins that need root (e.g. smartctl) will hang. For those, run `autoconf` and `suggest` manually as root and create symlinks by hand.
115
116## Writing plugins
117
118A plugin is any executable in `/etc/munin/plugins/` (usually a symlink from the plugin library directory). It must handle two invocations:
119
120```bash
121./plugin config # print graph metadata
122./plugin # print values
123```
124
125### Minimal shell plugin
126
127```sh
128#!/bin/sh
129if [ "${1:-}" = "config" ]; then
130 echo "graph_title My metric"
131 echo "graph_vlabel units"
132 echo "graph_category system"
133 echo "myfield.label Some value"
134 exit 0
135fi
136echo "myfield.value $(cat /some/source)"
137```
138
139### Field names
140
141Must match `^[A-Za-z_][A-Za-z0-9_]*$`. Sanitize dynamic names:
142
143```sh
144field=$(echo "$name" | sed 's/[^A-Za-z0-9_]/_/g; s/^[0-9]/_/')
145```
146
147### Data types
148
149- `GAUGE` (default): absolute value, plotted as-is
150- `COUNTER`/`DERIVE`: ever-increasing counter; munin computes rate per second. Use `DERIVE` with `.min 0` to avoid spikes on counter reset.
151
152### Multigraph plugins
153
154Output multiple graphs from one plugin by emitting `multigraph <name>` lines before each graph's config/values. Multigraph plugins are hidden from `list` output unless the client sends `cap multigraph` first.
155
156### Plugin configuration
157
158Per-plugin settings go in `/etc/munin/plugin-conf.d/<name>`:
159
160```ini
161[plugin_name]
162 user root
163 env.configfile /path/to/config
164 env.statuses available away chat xa
165```
166
167### Testing
168
169```bash
170munin-run <plugin_name> config # test config output
171munin-run <plugin_name> # test value output
172```
173
174Note: Debian's munin-node ships with `ProtectHome=yes` in systemd, which hides `/home/` from the entire process namespace regardless of user; `user root` in plugin-conf.d doesn't help. See [ProtectHome](#protecthome-and-home-access) for workarounds.
175
176After installing or removing plugins: `systemctl restart munin-node`
177
178## ProtectHome and /home/ access
179
180`ProtectHome=yes` mounts `/home/`, `/root`, `/run/user` as empty tmpfs. No user can see through it.
181
182Fix with
183
184```bash
185sudo mkdir -p /etc/systemd/system/munin-node.service.d
186printf '[Service]\nProtectHome=read-only\n' | sudo tee /etc/systemd/system/munin-node.service.d/override.conf
187sudo systemctl daemon-reload && sudo systemctl restart munin-node
188```
189
190Alternatives: `ProtectHome=tmpfs` + `BindReadOnlyPaths=` for selective exposure, or move data outside `/home/`.
191
192**Pitfall**: even with `ProtectHome=read-only`, a 750 home directory blocks the `munin` user from traversing the path. Use `user root` in plugin-conf.d for such cases.
193
194## Alerting
195
196Alerts are configured in `/etc/munin/munin.conf` on the master. A contact is a command that receives alert text on stdin.
197
198```ini
199contact.ntfy.command /usr/local/bin/munin-ntfy-alert
200contact.ntfy.always_send warning critical
201contact.ntfy.text ${var:host} :: ${var:graph_title} :: ${loop<, >:wfields WARNING ${var:label}=${var:value}} ${loop<, >:cfields CRITICAL ${var:label}=${var:value}}
202```
203
204### Thresholds
205
206Override per host or globally. The `memory` plugin uses percentages:
207
208```ini
209[groupname;hostname]
210 memory.warning 80
211 memory.critical 90
212```
213
214Plugin-specific fields use `pluginname.fieldname.warning` syntax.
215
216### Alert variables
217
218| Variable | Description |
219| -------------------------- | ----------------------------------------------- |
220| `${var:host}` | Node hostname |
221| `${var:graph_title}` | Plugin's graph title |
222| `${var:worst}` | Worst status: OK, WARNING, CRITICAL, UNKNOWN |
223| `${var:worstid}` | Numeric: 0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN |
224| `${loop<sep>:wfields ...}` | Iterate warning fields |
225| `${loop<sep>:cfields ...}` | Iterate critical fields |
226| `${var:label}` | Field label (inside loop) |
227| `${var:value}` | Field value (inside loop) |
228
229## Querying data programmatically
230
231RRD files on the master are queryable:
232
233```bash
234rrdtool fetch /var/lib/munin/group/host-plugin-field-g.rrd AVERAGE --start -1h
235```
236
237The munin-node protocol is also directly queryable over TCP:
238
239```bash
240{ echo 'fetch memory'; sleep 1; echo 'quit'; } | nc <node_ip> 4949
241```
242
243## Reference
244
245- **Plugin gallery**: https://gallery.munin-monitoring.org/
246- **Full docs**: https://guide.munin-monitoring.org/en/latest/
247- **Writing plugins**: See [writing-plugins.md](references/writing-plugins.md)