1---
2name: monitoring-with-munin
3description: Deploys and manages Munin monitoring across servers. Use when setting up munin-node on a host, writing munin plugins, adding nodes to a master, configuring alerts, or diagnosing system issues using munin data. Also use when the user mentions munin, monitoring, or graphing server metrics.
4license: GPL-3.0-or-later
5metadata:
6 author: Amolith <amolith@secluded.site>
7---
8
9If the user has an existing Munin setup they want you to work with, ask them for specifics: where the master is, how nodes are connected (Tailscale, direct IP, SSH tunnel), and what OS the target hosts run.
10
11## Installing munin-node
12
13### Debian/Ubuntu
14
15```bash
16apt-get install -y munin-node
17munin-node-configure --shell | sh -x # auto-detect and symlink plugins
18systemctl enable --now munin-node
19```
20
21### Arch Linux
22
23```bash
24pacman -S --noconfirm munin-node
25# Net::CIDR is often unavailable on Arch; use regex allow instead of cidr_allow
26munin-node-configure --shell | sh -x
27systemctl enable --now munin-node
28```
29
30## Configuring munin-node
31
32Config lives at `/etc/munin/munin-node.conf`. Key directives:
33
34```ini
35host * # bind to all interfaces
36port 4949
37allow ^127\.0\.0\.1$ # regex against connecting IP
38allow ^::1$
39allow 100\.107\.78\.23 # master's IP (unanchored works too)
40cidr_allow 100.107.78.23/32 # alternative (needs perl Net::CIDR)
41```
42
43The `allow` directive uses Perl regexes matched against the client IP. When the connection arrives as IPv6-mapped IPv4 (`::ffff:A.B.C.D`), the anchored regex `^A\.B\.C\.D$` won't match. Use an **unanchored** regex like `A\.B\.C\.D` to handle both forms, or add an explicit `allow ^::ffff:A\.B\.C\.D$`.
44
45On Arch Linux, `Net::CIDR` is typically unavailable (only `Net::CIDR::Lite` exists in pacman). If `cidr_allow` causes `Can't locate Net/CIDR.pm` errors, remove all `cidr_allow` lines and use `allow` regexes instead.
46
47After changing config: `systemctl restart munin-node`
48
49### Firewall
50
51If UFW is present, restrict port 4949 to the master only:
52
53```bash
54ufw allow from <MASTER_TS_IP> to any port 4949 comment 'munin master'
55ufw deny in 4949 comment 'deny munin from everyone else'
56```
57
58Order matters — allow must come before deny.
59
60## Adding a node to the master
61
62Append to `/etc/munin/munin.conf` on the master:
63
64```ini
65[groupname;hostname]
66 address <node_tailscale_ip>
67 use_node_name yes
68```
69
70Group names organize the web UI — use logical names like `nixnet`, `exe.xyz`, and `personal`.
71
72Seed data immediately: `su - munin --shell=/bin/bash -c '/usr/bin/munin-cron'`
73
74### Verifying connectivity
75
76From the master, test the node protocol:
77
78```bash
79# Basic test (non-multigraph plugins only)
80echo 'quit' | nc -w3 <node_ip> 4949
81
82# Full test including multigraph plugins
83{ sleep 1; echo 'cap multigraph'; sleep 1; echo 'list'; sleep 1; echo 'quit'; } | nc -w5 <node_ip> 4949
84```
85
86A working node responds with `# munin node at <hostname>` followed by the plugin list.
87
88## Writing plugins
89
90A plugin is any executable in `/etc/munin/plugins/` (usually a symlink to `/usr/share/munin/plugins/` or `/usr/lib/munin/plugins/`). It must handle two invocations:
91
92```bash
93./plugin config # print graph metadata
94./plugin # print values
95```
96
97### Minimal shell plugin
98
99```sh
100#!/bin/sh
101if [ "${1:-}" = "config" ]; then
102 echo "graph_title My metric"
103 echo "graph_vlabel units"
104 echo "graph_category system"
105 echo "myfield.label Some value"
106 exit 0
107fi
108echo "myfield.value $(cat /some/source)"
109```
110
111### Field names
112
113Must match `^[A-Za-z_][A-Za-z0-9_]*$`. Sanitize dynamic names:
114
115```sh
116field=$(echo "$name" | sed 's/[^A-Za-z0-9_]/_/g; s/^[0-9]/_/')
117```
118
119### Data types
120
121- `GAUGE` (default): absolute value, plotted as-is
122- `COUNTER`/`DERIVE`: ever-increasing counter; munin computes rate per second. Use `DERIVE` with `.min 0` to avoid spikes on counter reset.
123
124### Multigraph plugins
125
126Output multiple graphs from one plugin by emitting `multigraph <name>` lines before each graph's config/values. Multigraph plugins are hidden from `list` output unless the client sends `cap multigraph` first.
127
128### Plugin configuration
129
130Per-plugin settings go in `/etc/munin/plugin-conf.d/<name>`:
131
132```ini
133[plugin_name]
134 user root
135 env.configfile /path/to/config
136 env.statuses available away chat xa
137```
138
139### Testing
140
141```bash
142munin-run <plugin_name> config # test config output
143munin-run <plugin_name> # test value output
144```
145
146Note: on systems where munin-node runs with `ProtectHome=yes` (systemd), plugins running as non-root users cannot access `/home/`. Either run as `user root` or place data outside `/home/`.
147
148After installing or removing plugins: `systemctl restart munin-node`
149
150## Alerting
151
152Alerts are configured in `/etc/munin/munin.conf` on the master. A contact is a command that receives alert text on stdin.
153
154```ini
155contact.ntfy.command /usr/local/bin/munin-ntfy-alert
156contact.ntfy.always_send warning critical
157contact.ntfy.text ${var:host} :: ${var:graph_title} :: ${loop<, >:wfields WARNING ${var:label}=${var:value}} ${loop<, >:cfields CRITICAL ${var:label}=${var:value}}
158```
159
160### Thresholds
161
162Override per host or globally. The `memory` plugin uses percentages:
163
164```ini
165[groupname;hostname]
166 memory.warning 80
167 memory.critical 90
168```
169
170Plugin-specific fields use `pluginname.fieldname.warning` syntax.
171
172### Alert variables
173
174| Variable | Description |
175| -------------------------- | ----------------------------------------------- |
176| `${var:host}` | Node hostname |
177| `${var:graph_title}` | Plugin's graph title |
178| `${var:worst}` | Worst status: OK, WARNING, CRITICAL, UNKNOWN |
179| `${var:worstid}` | Numeric: 0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN |
180| `${loop<sep>:wfields ...}` | Iterate warning fields |
181| `${loop<sep>:cfields ...}` | Iterate critical fields |
182| `${var:label}` | Field label (inside loop) |
183| `${var:value}` | Field value (inside loop) |
184
185## Querying data programmatically
186
187RRD files on the master are queryable:
188
189```bash
190rrdtool fetch /var/lib/munin/group/host-plugin-field-g.rrd AVERAGE --start -1h
191```
192
193The munin-node protocol is also directly queryable over TCP:
194
195```bash
196{ echo 'fetch memory'; sleep 1; echo 'quit'; } | nc <node_ip> 4949
197```
198
199## Reference
200
201- **Plugin gallery**: https://gallery.munin-monitoring.org/
202- **Full docs**: https://guide.munin-monitoring.org/en/latest/
203- **Writing plugins**: See [writing-plugins.md](references/writing-plugins.md)