name: monitoring-with-munin description: Deploys and manages Munin monitoring across servers. Use when setting up munin-node on a host, writing munin plugins, adding nodes to a master, configuring alerts, or diagnosing system issues using munin data. Also use when the user mentions munin, monitoring, or graphing server metrics. license: GPL-3.0-or-later metadata: author: Amolith amolith@secluded.site

If the user has an existing Munin setup they want you to work with, ask them for specifics: where the master is, how nodes are connected (Tailscale, direct IP, SSH tunnel), and what OS the target hosts run.

Installing munin-node

Debian/Ubuntu

apt-get install -y munin-node
munin-node-configure --shell | sh -x   # auto-detect and symlink plugins
systemctl enable --now munin-node

Arch Linux

pacman -S --noconfirm munin-node
# Net::CIDR is often unavailable on Arch; use regex allow instead of cidr_allow
munin-node-configure --shell | sh -x
systemctl enable --now munin-node

Configuring munin-node

Config lives at /etc/munin/munin-node.conf. Key directives:

host *                          # bind to all interfaces
port 4949
allow ^127\.0\.0\.1$            # regex against connecting IP
allow ^::1$
allow 100\.107\.78\.23          # master's IP (unanchored works too)
cidr_allow 100.107.78.23/32    # alternative (needs perl Net::CIDR)

The allow directive uses Perl regexes matched against the client IP. When the connection arrives as IPv6-mapped IPv4 (::ffff:A.B.C.D), the anchored regex ^A\.B\.C\.D$ won't match. Use an unanchored regex like A\.B\.C\.D to handle both forms, or add an explicit allow ^::ffff:A\.B\.C\.D$.

On Arch Linux, Net::CIDR is typically unavailable (only Net::CIDR::Lite exists in pacman). If cidr_allow causes Can't locate Net/CIDR.pm errors, remove all cidr_allow lines and use allow regexes instead.

After changing config: systemctl restart munin-node

Firewall

If UFW is present, restrict port 4949 to the master only:

ufw allow from <MASTER_TS_IP> to any port 4949 comment 'munin master'
ufw deny in 4949 comment 'deny munin from everyone else'

Order matters — allow must come before deny.

Adding a node to the master

Append to /etc/munin/munin.conf on the master:

[groupname;hostname]
    address <node_tailscale_ip>
    use_node_name yes

Group names organize the web UI — use logical names like nixnet, exe.xyz, and personal.

Seed data immediately: su - munin --shell=/bin/bash -c '/usr/bin/munin-cron'

Verifying connectivity

From the master, test the node protocol:

# Basic test (non-multigraph plugins only)
echo 'quit' | nc -w3 <node_ip> 4949

# Full test including multigraph plugins
{ sleep 1; echo 'cap multigraph'; sleep 1; echo 'list'; sleep 1; echo 'quit'; } | nc -w5 <node_ip> 4949

A working node responds with # munin node at <hostname> followed by the plugin list.

Installing third-party plugins

Third-party plugins (including our custom ones) go in /usr/local/munin/lib/plugins/, not the distribution plugin directory (/usr/share/munin/plugins/ on Debian, /usr/lib/munin/plugins/ on Arch). This avoids package updates overwriting custom plugins.

mkdir -p /usr/local/munin/lib/plugins
cp my_plugin /usr/local/munin/lib/plugins/
chmod +x /usr/local/munin/lib/plugins/my_plugin

Create symlinks in /etc/munin/plugins/ manually:

# Simple plugin
ln -s /usr/local/munin/lib/plugins/my_plugin /etc/munin/plugins/my_plugin
# Wildcard plugin
ln -s /usr/local/munin/lib/plugins/my_plugin_ /etc/munin/plugins/my_plugin_instance

Auto-detection with munin-node-configure requires --libdir:

munin-node-configure --libdir /usr/local/munin/lib/plugins --shell

Note: munin-node-configure runs autoconf/suggest as the munin user. Plugins that need root (e.g. smartctl) will hang. For those, run autoconf and suggest manually as root and create symlinks by hand.

Writing plugins

A plugin is any executable in /etc/munin/plugins/ (usually a symlink from the plugin library directory). It must handle two invocations:

./plugin config    # print graph metadata
./plugin           # print values

Minimal shell plugin

#!/bin/sh
if [ "${1:-}" = "config" ]; then
    echo "graph_title My metric"
    echo "graph_vlabel units"
    echo "graph_category system"
    echo "myfield.label Some value"
    exit 0
fi
echo "myfield.value $(cat /some/source)"

Field names

Must match ^[A-Za-z_][A-Za-z0-9_]*$. Sanitize dynamic names:

field=$(echo "$name" | sed 's/[^A-Za-z0-9_]/_/g; s/^[0-9]/_/')

Data types

• GAUGE (default): absolute value, plotted as-is
• COUNTER/DERIVE: ever-increasing counter; munin computes rate per second. Use DERIVE with .min 0 to avoid spikes on counter reset.

Multigraph plugins

Output multiple graphs from one plugin by emitting multigraph <name> lines before each graph's config/values. Multigraph plugins are hidden from list output unless the client sends cap multigraph first.

Plugin configuration

Per-plugin settings go in /etc/munin/plugin-conf.d/<name>:

[plugin_name]
    user root
    env.configfile /path/to/config
    env.statuses available away chat xa

Testing

munin-run <plugin_name> config   # test config output
munin-run <plugin_name>          # test value output

Note: Debian's munin-node ships with ProtectHome=yes in systemd, which hides /home/ from the entire process namespace regardless of user; user root in plugin-conf.d doesn't help. See ProtectHome for workarounds.

After installing or removing plugins: systemctl restart munin-node

ProtectHome and /home/ access

ProtectHome=yes mounts /home/, /root, /run/user as empty tmpfs. No user can see through it.

Fix with

sudo mkdir -p /etc/systemd/system/munin-node.service.d
printf '[Service]\nProtectHome=read-only\n' | sudo tee /etc/systemd/system/munin-node.service.d/override.conf
sudo systemctl daemon-reload && sudo systemctl restart munin-node

Alternatives: ProtectHome=tmpfs + BindReadOnlyPaths= for selective exposure, or move data outside /home/.

Pitfall: even with ProtectHome=read-only, a 750 home directory blocks the munin user from traversing the path. Use user root in plugin-conf.d for such cases.

Alerting

Alerts are configured in /etc/munin/munin.conf on the master. A contact is a command that receives alert text on stdin.

contact.ntfy.command /usr/local/bin/munin-ntfy-alert
contact.ntfy.always_send warning critical
contact.ntfy.text ${var:host} :: ${var:graph_title} :: ${loop<, >:wfields WARNING ${var:label}=${var:value}} ${loop<, >:cfields CRITICAL ${var:label}=${var:value}}

Thresholds

Override per host or globally. The memory plugin uses percentages:

[groupname;hostname]
    memory.warning 80
    memory.critical 90

Plugin-specific fields use pluginname.fieldname.warning syntax.

Alert variables

Querying data programmatically

RRD files on the master are queryable:

rrdtool fetch /var/lib/munin/group/host-plugin-field-g.rrd AVERAGE --start -1h

The munin-node protocol is also directly queryable over TCP:

{ echo 'fetch memory'; sleep 1; echo 'quit'; } | nc <node_ip> 4949

Reference

• Plugin gallery: https://gallery.munin-monitoring.org/
• Full docs: https://guide.munin-monitoring.org/en/latest/
• Writing plugins: See writing-plugins.md