Monitoring Pexip nodes with munin

Follow

Introduction

Munin is an open-source network monitoring tool, and can be used to monitor Pexip nodes (as well as generic servers, applications, network equipment and more).

This guide will explain how to configure munin to monitor Pexip nodes, both via SNMP (for collecting generic system statistics) and via the Pexip management API (for collecting metrics on participant and conference count, as well as media load per location).

This guide assumes a standard install of Ubuntu Server 14.04 64-bit as the starting point, and nginx will be used to host web content for munin. 

All commands are run via SSH, and are shown in italic like this

Configuration file parameters are also shown in italic like this.

Notes on security

In order for munin to monitor Pexip nodes (and network equipment/servers in general), munin needs various types of credentials for these remote hosts (SNMP community strings, API credentials and so forth).

For this reason, the following principles should be followed for the munin host (and for monitoring systems in general):

  • munin should be deployed on an internal network, only allowing admin access (SSH) from internal/trusted networks.
  • The admin/root user should be set up with a strong, complex password, or preferably SSH key authentication for management access
  • Access to the munin web root should only be allowed from trusted subnets/hosts (instructions for this later in this guide)
  • Directory /etc/munin/plugin-conf.d should only be accessible for users root and munin (Owned by root:munin with chmod 750, which is the default configuration)

 

Section 1 - Getting munin up and running

The following section will cover the munin configuration basics and verify that your basic munin install is working.

1. Ensure that all packages are up to date:

sudo apt-get update

sudo apt-get dist-upgrade

2. Install nginx:

sudo apt-get install nginx

3. Install munin and extra packages:

sudo apt-get install munin munin-node munin-plugins-extra

This will prompt for installing various packages which munin depends on.

4. Create directory /var/www/munin and change ownership of the directory to user munin, group munin:

sudo mkdir -p /var/www/munin

sudo chown -R munin:munin /var/www/munin

These permissions are needed in order for munin to write HTML files in /var/www/munin.

5. Create an nginx configuration file for munin:

sudo nano /etc/nginx/sites-available/munin

Add the following configuration into the 'munin' file (replace munin.yourdomain.com with the FQDN of the munin host) - in this case, we will only allow subnet 10.5.3.0/24 access to the munin pages in 'location /', as this is our management network - replace 10.5.3.0/24 with a suitable CIDR for your environment:

server {
    listen 80;

    server_name munin.yourdomain;
    root /var/www/munin;
    autoindex off;

    location / {
        allow 10.5.3.0/24;
        deny all;
    }
}

Save and exit the file.

6. Make the 'munin' server configuration active, remove the default site and reload nginx:

sudo ln -s /etc/nginx/sites-available/munin /etc/nginx/sites-enabled/munin

sudo rm /etc/nginx/sites-enabled/default

sudo service nginx reload

7. Rename the default munin config file and create a new munin.conf with some suitable defaults:

cd /etc/munin

sudo mv munin.conf munin.conf.original

sudo nano munin.conf

Add the following configuration parameters (replace munin.yourdomain.com with the FQDN of the munin host):

dbdir /var/lib/munin
htmldir /var/www/munin
logdir /var/log/munin
rundir /var/run/munin

includedir /etc/munin/munin-conf.d
graph_period second
graph_strategy cron
html_strategy cron

[General;munin.yourdomain.com]
    address 127.0.0.1
    use_node_name yes

Save the file and exit.

8. Restart munin-node to apply the new munin configuration:

sudo service munin-node restart

9. Wait for minutes and verify that http://munin.yourdomain.com now shows some basic graphs for the munin host itself:

 

10. (Optional) Remove non-wanted/non-needed plugins for the munin host itself:

The default monitoring plugins for the munin host itself are located in /etc/munin/plugins (as symlinks to /usr/share/munin/plugins). If some of these plugins are not of interest (meaning if the admin doesn't see any value in monitoring certain metrics), they can be disabled by removing the symlink.

For instance, if plugins entropy, forks, fw_packets and vmstat are not needed, disable these by removing the symlinks and restarting munin-node:

cd /etc/munin/plugins

sudo rm entropy forks fw_packets vmstat

sudo service munin-node restart

 

Section 2 - Setting up SNMP monitoring of Pexip nodes

This section will describe how to enable SNMP monitoring of Pexip nodes in a deployment.

The guide assumes that the following Pexip deployment has been set up:

  • Management node: pex-mgr.yourdomain.com
  • Conf node 1: pex-conf01.yourdomain.com
  • Conf node 2: pex-conf02.yourdomain.com
  • Conf node 3: pex-conf03.yourdomain.com

All 4 nodes have existing DNS A-records in place, pointing to the IP address of each respective node. It is essential that these records are in place and that the munin host can resolve these FQDNs in DNS. 

1. Configure the SNMP community for the management node and each conferencing node:

For this guide, we will use SNMP mode 'SNMPv2c read-only' and community string 'topsecret' for all 4 nodes (Use a more complex community string for your actual deployment!).

  • Configure SNMP for the management node in Platform Configuration > Management node
  • Configure SNMP for the each individual conferencing node in Platform Configuration > Conferencing nodes

 

2. On the munin host, SNMP community strings are stored in file /etc/munin/plugin-conf.d/snmp_communities, and the syntax for each device entry is:

[snmp_FQDN_*]
env.community COMMUNITYSTRING
 

Become root and create config file /etc/munin/plugin-conf.d/snmp_communities containing the SNMP community strings for your Pexip nodes:

sudo -s

cd /etc/munin/plugin-conf.d

sudo nano snmp_communities

Enter the following values in the snmp_communities file:

[snmp_pex-mgr.yourdomain.com_*]
env.community topsecret

[snmp_pex-conf01.yourdomain.com_*]
env.community topsecret

[snmp_pex-conf02.yourdomain.com_*]
env.community topsecret

[snmp_pex-conf03.yourdomain.com_*]
env.community topsecret

Save and exit the file.

3. Create SNMP symlinks in /etc/munin/plugins for the 4 Pexip nodes

For all commands below, replace pex-mgr.yourdomain.com, pex-conf01.yourdomain.com and so forth with the Pexip node FQDNs for your deployment. We will create symbolic links for SNMP plugins cpuload, diskfree, if, load, memory, uptime from /usr/share/munin/plugins.

Note the double underscores after snmp (snmp__) and single underscore after if (if_).

Run each command individually, and ensure that you change into directory /etc/munin/plugins before creating the symlinks:

cd /etc/munin/plugins

sudo ln -s /usr/share/munin/plugins/snmp__cpuload snmp_pex-mgr.yourdomain.com_cpuload
sudo ln -s /usr/share/munin/plugins/snmp__cpuload snmp_pex-conf01.yourdomain.com_cpuload
sudo ln -s /usr/share/munin/plugins/snmp__cpuload snmp_pex-conf02.yourdomain.com_cpuload
sudo ln -s /usr/share/munin/plugins/snmp__cpuload snmp_pex-conf03.yourdomain.com_cpuload
sudo ln -s /usr/share/munin/plugins/snmp__if_ snmp_pex-mgr.yourdomain.com__if_2
sudo ln -s /usr/share/munin/plugins/snmp__if_ snmp_pex-conf01.yourdomain.com__if_2
sudo ln -s /usr/share/munin/plugins/snmp__if_ snmp_pex-conf02.yourdomain.com__if_2
sudo ln -s /usr/share/munin/plugins/snmp__if_ snmp_pex-conf03.yourdomain.com__if_2
sudo ln -s /usr/share/munin/plugins/snmp__load snmp_pex-mgr.yourdomain.com_load
sudo ln -s /usr/share/munin/plugins/snmp__load snmp_pex-conf01.yourdomain.com_load
sudo ln -s /usr/share/munin/plugins/snmp__load snmp_pex-conf02.yourdomain.com_load
sudo ln -s /usr/share/munin/plugins/snmp__load snmp_pex-conf03.yourdomain.com_load
sudo ln -s /usr/share/munin/plugins/snmp__memory snmp_pex-mgr.yourdomain.com_memory
sudo ln -s /usr/share/munin/plugins/snmp__memory snmp_pex-conf01.yourdomain.com_memory
sudo ln -s /usr/share/munin/plugins/snmp__memory snmp_pex-conf02.yourdomain.com_memory
sudo ln -s /usr/share/munin/plugins/snmp__memory snmp_pex-conf03.yourdomain.com_memory
sudo ln -s /usr/share/munin/plugins/snmp__uptime snmp_pex-mgr.yourdomain.com_uptime
sudo ln -s /usr/share/munin/plugins/snmp__uptime snmp_pex-conf01.yourdomain.com_uptime
sudo ln -s /usr/share/munin/plugins/snmp__uptime snmp_pex-conf02.yourdomain.com_uptime
sudo ln -s /usr/share/munin/plugins/snmp__uptime snmp_pex-conf03.yourdomain.com_uptime

4. Update the munin config file /etc/munin/munin.conf to contain the Pexip nodes in the munin hierarchy.

Note that the 'address' entry for the Pexip nodes should contain 127.0.0.1 since we are using SNMP:

Edit the munin config file:

sudo nano /etc/munin/munin.conf

Add the additional config entries for the 4 Pexip nodes, so that munin.conf contains the following (again replace xyz.yourdomain.com with the Pexip node FQDNs):

dbdir /var/lib/munin
htmldir /var/www/munin
logdir /var/log/munin
rundir /var/run/munin

includedir /etc/munin/munin-conf.d
graph_period second
graph_strategy cron
html_strategy cron

[General;munin.yourdomain.com]
    address 127.0.0.1
    use_node_name yes

[Pexip;pex-mgr.yourdomain.com]
    address 127.0.0.1
    use_node_name no

[Pexip;pex-conf01.yourdomain.com]
    address 127.0.0.1
    use_node_name no

[Pexip;pex-conf02.yourdomain.com]
    address 127.0.0.1
    use_node_name no

[Pexip;pex-conf03.yourdomain.com]
    address 127.0.0.1
    use_node_name no

Save and exit the file, and restart munin-node to apply the changes:

sudo service munin-node restart

Wait 10 minutes and refresh the munin main page, you should now see 4 new entries for your Pexip nodes. Click on each node to verify that munin is successfully collecting metrics for the nodes via SNMP.

Note that it may take a bit longer for some of the graphs to output metrics:

 

Section 3 - Set up munin to collect additional usage metrics from Pexip via management API

The Pexip management node offers a REST API for integration with 3rd party systems (http://www.pexip.com/pexip-infinity-management-api). Among many other features, this API allows a 3rd party system (such as munin) to retrieve real-time data about the participant count, conference count and media load per location within a Pexip environment.

This section will cover the configuration steps needed for allowing munin to collect these metrics. This is achieved by adding 4 additional, Pexip specific, plugin files to the munin host.

Note that these Pexip-specific plugins are a work in progress, and that this guide will be updated whenever new versions of these plugins become available.

1. Download and install python-munin (helper library for python-based munin plugins) - Credit for this library goes to http://samuelks.com/python-munin/:

cd /tmp

wget https://github.com/samuel/python-munin/tarball/master -O python-munin.tar.gz

tar zxvf python-munin.tar.gz

(python-munin.tar.gz will extract to a folder named 'samuel-python-munin-833c886 or similar, so this is the directory you need to cd into):

cd samuel-python-munin-833c886/

sudo python ./setup.py install

2. Download the tar.gz archive containing the Pexip munin plugins onto the munin host, decompress the file and copy the extracted files into /usr/share/munin/plugins:

cd /tmp

wget https://support.pexip.com/hc/en-us/article_attachments/202404169/pexip-munin-plugins-v1.0.tar.gz

tar zxvf pexip-munin-plugins-v1.0.tar.gz

sudo cp pexip-munin-plugins-v1.0/* /usr/share/munin/plugins/

3. Make the pexiprest plugins executable:

sudo chmod +x /usr/share/munin/plugins/pexiprest*

4. Create munin plugin configuration file /etc/plugin-conf.d/pexiprest_credentials:

This config file should contain a username and password for the management node, where the given user has API read access  - Ideally, a separate API service user should be created for this (if LDAP authentication is enabled on management node), but the management web admin user will also work.

For our example, we assume a management node username of admin and password of supersecret.

Become root and create config file /etc/munin/plugin-conf.d/pexiprest_credentials:

sudo -s

cd /etc/munin/plugin-conf.d

nano pexiprest_credentials

Enter the following parameters in the config file:

[pexiprest_pex-mgr.yourdomain.com_*]
env.username admin
env.password supersecret

Save and exit the file.

4. Create symbolic links for the 3 Pexip plugins, pointing to management node pex-mgr.yourdomain.com - note double underscores in plugin path and single underscore in symlink:

cd /etc/munin/plugins

sudo ln -s /usr/share/munin/plugins/pexiprest__conferences pexiprest_pex-mgr.yourdomain.com_conferences

sudo ln -s /usr/share/munin/plugins/pexiprest__participants pexiprest_pex-mgr.yourdomain.com_participants

sudo ln -s /usr/share/munin/plugins/pexiprest__medialoadbylocation pexiprest_pex-mgr.yourdomain.com_medialoadbylocation

5. Restart munin-node to apply the changes:

sudo service munin-node restart

6. Wait 10 minutes and refresh the munin page for the management node - You should now see 3 more graphs for conferences, participants and media load by location:

This concludes the guide, as you should now have a working munin environment that is monitoring your entire Pexip deployment.

Additional documentation and useful information:

  • munin home page: http://munin-monitoring.org/
  • enabling munin to send alerts: http://munin-monitoring.org/wiki/HowToContact

Troubleshooting:

All munin log files are located in /var/log/munin:

  • munin-update.log - contains general log entries on all plugin updates
  • munin-node.log - contains log entries specific to the munin-node instance on the munin host
  • munin-html.log - contains log entries specific to the HTML files that are written to /var/www/munin

Feedback and comments are welcome!

Last updated: May 29th 2015 - Andreas W

 

Have more questions? Submit a request

Comments

Powered by Zendesk