--- name: monitoring-with-munin description: Deploys and manages Munin monitoring across servers. Use when setting up munin-node on a host, writing munin plugins, adding nodes to a master, configuring alerts, or diagnosing system issues using munin data. Also use when the user mentions munin, monitoring, or graphing server metrics. license: GPL-3.0-or-later metadata: author: Amolith --- If the user has an existing Munin setup they want you to work with, ask them for specifics: where the master is, how nodes are connected (Tailscale, direct IP, SSH tunnel), and what OS the target hosts run. ## Installing munin-node ### Debian/Ubuntu ```bash apt-get install -y munin-node munin-node-configure --shell | sh -x # auto-detect and symlink plugins systemctl enable --now munin-node ``` ### Arch Linux ```bash pacman -S --noconfirm munin-node # Net::CIDR is often unavailable on Arch; use regex allow instead of cidr_allow munin-node-configure --shell | sh -x systemctl enable --now munin-node ``` ## Configuring munin-node Config lives at `/etc/munin/munin-node.conf`. Key directives: ```ini host * # bind to all interfaces port 4949 allow ^127\.0\.0\.1$ # regex against connecting IP allow ^::1$ allow 100\.107\.78\.23 # master's IP (unanchored works too) cidr_allow 100.107.78.23/32 # alternative (needs perl Net::CIDR) ``` The `allow` directive uses Perl regexes matched against the client IP. When the connection arrives as IPv6-mapped IPv4 (`::ffff:A.B.C.D`), the anchored regex `^A\.B\.C\.D$` won't match. Use an **unanchored** regex like `A\.B\.C\.D` to handle both forms, or add an explicit `allow ^::ffff:A\.B\.C\.D$`. On Arch Linux, `Net::CIDR` is typically unavailable (only `Net::CIDR::Lite` exists in pacman). If `cidr_allow` causes `Can't locate Net/CIDR.pm` errors, remove all `cidr_allow` lines and use `allow` regexes instead. After changing config: `systemctl restart munin-node` ### Firewall If UFW is present, restrict port 4949 to the master only: ```bash ufw allow from to any port 4949 comment 'munin master' ufw deny in 4949 comment 'deny munin from everyone else' ``` Order matters — allow must come before deny. ## Adding a node to the master Append to `/etc/munin/munin.conf` on the master: ```ini [groupname;hostname] address use_node_name yes ``` Group names organize the web UI — use logical names like `nixnet`, `exe.xyz`, and `personal`. Seed data immediately: `su - munin --shell=/bin/bash -c '/usr/bin/munin-cron'` ### Verifying connectivity From the master, test the node protocol: ```bash # Basic test (non-multigraph plugins only) echo 'quit' | nc -w3 4949 # Full test including multigraph plugins { sleep 1; echo 'cap multigraph'; sleep 1; echo 'list'; sleep 1; echo 'quit'; } | nc -w5 4949 ``` A working node responds with `# munin node at ` followed by the plugin list. ## Installing third-party plugins Third-party plugins (including our custom ones) go in `/usr/local/munin/lib/plugins/`, **not** the distribution plugin directory (`/usr/share/munin/plugins/` on Debian, `/usr/lib/munin/plugins/` on Arch). This avoids package updates overwriting custom plugins. ```bash mkdir -p /usr/local/munin/lib/plugins cp my_plugin /usr/local/munin/lib/plugins/ chmod +x /usr/local/munin/lib/plugins/my_plugin ``` Create symlinks in `/etc/munin/plugins/` manually: ```bash # Simple plugin ln -s /usr/local/munin/lib/plugins/my_plugin /etc/munin/plugins/my_plugin # Wildcard plugin ln -s /usr/local/munin/lib/plugins/my_plugin_ /etc/munin/plugins/my_plugin_instance ``` Auto-detection with `munin-node-configure` requires `--libdir`: ```bash munin-node-configure --libdir /usr/local/munin/lib/plugins --shell ``` Note: `munin-node-configure` runs `autoconf`/`suggest` as the munin user. Plugins that need root (e.g. smartctl) will hang. For those, run `autoconf` and `suggest` manually as root and create symlinks by hand. ## Writing plugins A plugin is any executable in `/etc/munin/plugins/` (usually a symlink from the plugin library directory). It must handle two invocations: ```bash ./plugin config # print graph metadata ./plugin # print values ``` ### Minimal shell plugin ```sh #!/bin/sh if [ "${1:-}" = "config" ]; then echo "graph_title My metric" echo "graph_vlabel units" echo "graph_category system" echo "myfield.label Some value" exit 0 fi echo "myfield.value $(cat /some/source)" ``` ### Field names Must match `^[A-Za-z_][A-Za-z0-9_]*$`. Sanitize dynamic names: ```sh field=$(echo "$name" | sed 's/[^A-Za-z0-9_]/_/g; s/^[0-9]/_/') ``` ### Data types - `GAUGE` (default): absolute value, plotted as-is - `COUNTER`/`DERIVE`: ever-increasing counter; munin computes rate per second. Use `DERIVE` with `.min 0` to avoid spikes on counter reset. ### Multigraph plugins Output multiple graphs from one plugin by emitting `multigraph ` lines before each graph's config/values. Multigraph plugins are hidden from `list` output unless the client sends `cap multigraph` first. ### Plugin configuration Per-plugin settings go in `/etc/munin/plugin-conf.d/`: ```ini [plugin_name] user root env.configfile /path/to/config env.statuses available away chat xa ``` ### Testing ```bash munin-run config # test config output munin-run # test value output ``` Note: Debian's munin-node ships with `ProtectHome=yes` in systemd, which hides `/home/` from the entire process namespace regardless of user; `user root` in plugin-conf.d doesn't help. See [ProtectHome](#protecthome-and-home-access) for workarounds. After installing or removing plugins: `systemctl restart munin-node` ## ProtectHome and /home/ access `ProtectHome=yes` mounts `/home/`, `/root`, `/run/user` as empty tmpfs. No user can see through it. Fix with ```bash sudo mkdir -p /etc/systemd/system/munin-node.service.d printf '[Service]\nProtectHome=read-only\n' | sudo tee /etc/systemd/system/munin-node.service.d/override.conf sudo systemctl daemon-reload && sudo systemctl restart munin-node ``` Alternatives: `ProtectHome=tmpfs` + `BindReadOnlyPaths=` for selective exposure, or move data outside `/home/`. **Pitfall**: even with `ProtectHome=read-only`, a 750 home directory blocks the `munin` user from traversing the path. Use `user root` in plugin-conf.d for such cases. ## Alerting Alerts are configured in `/etc/munin/munin.conf` on the master. A contact is a command that receives alert text on stdin. ```ini contact.ntfy.command /usr/local/bin/munin-ntfy-alert contact.ntfy.always_send warning critical contact.ntfy.text ${var:host} :: ${var:graph_title} :: ${loop<, >:wfields WARNING ${var:label}=${var:value}} ${loop<, >:cfields CRITICAL ${var:label}=${var:value}} ``` ### Thresholds Override per host or globally. The `memory` plugin uses percentages: ```ini [groupname;hostname] memory.warning 80 memory.critical 90 ``` Plugin-specific fields use `pluginname.fieldname.warning` syntax. ### Alert variables | Variable | Description | | -------------------------- | ----------------------------------------------- | | `${var:host}` | Node hostname | | `${var:graph_title}` | Plugin's graph title | | `${var:worst}` | Worst status: OK, WARNING, CRITICAL, UNKNOWN | | `${var:worstid}` | Numeric: 0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN | | `${loop:wfields ...}` | Iterate warning fields | | `${loop:cfields ...}` | Iterate critical fields | | `${var:label}` | Field label (inside loop) | | `${var:value}` | Field value (inside loop) | ## Querying data programmatically RRD files on the master are queryable: ```bash rrdtool fetch /var/lib/munin/group/host-plugin-field-g.rrd AVERAGE --start -1h ``` The munin-node protocol is also directly queryable over TCP: ```bash { echo 'fetch memory'; sleep 1; echo 'quit'; } | nc 4949 ``` ## Reference - **Plugin gallery**: https://gallery.munin-monitoring.org/ - **Full docs**: https://guide.munin-monitoring.org/en/latest/ - **Writing plugins**: See [writing-plugins.md](references/writing-plugins.md)