Icinga 1.6 As A Monitoring Solution On Ubuntu 12.04 (Part 2: Icinga Web)

Feb 212012

After getting a basic Icinga installation up and running in the last part, I’ll describe how to get Icinga Web, the new Ajax-powered web interface for Icinga to work. If you encounter any problems feel free to post a comment or check out the helpful Icinga Docs and Wiki.

All parts of my Icinga 1.6 As A Monitoring Solution On Ubuntu 12.04 series:

This part is strongly based on the Wiki article (see link above), so you can get additional informations there.

1) Prerequisites

For the next steps you’ll need some additional packages (mostly PHP modules for Apache) and if you want to fetch Icinga using Git, the Git client, so let’s install them:

apt-get install php5 php5-cli php-pear php5-xmlrpc php5-xsl php5-gd php5-ldap php5-mysql
(optional) apt-get install git

2) Installation

Now you’ll need the Icinga Web source. Unfortunately, at the time of this writing, there is no Ubuntu package available.  I cloned the Git repository, since I wanted the upcoming 1.6.2 version of Icinga Web, however in most cases (and unless told otherwise by a developer), you’re probably better off downloading a stable package from the Icinga Website.

Using a package:

  • Go to https://www.icinga.org/download/packages/ and download the latest stable version of Icinga Web to your server (e.g. using wget <url>)
  • Extract the contents using 
    tar vfxz icinga-web-x.x.x.tar.gz
  • Change into the new directory (cs icinga-web-x.x.x)

Using git:

  • git clone git://git.icinga.org/icinga-web.git
  • cd icinga-web

Now it’s time to configure the installation process. If we don’t specify target directories, everything will go to /usr/local/inciga-web/.., however since we installed Icinga already using packages and therefore use the global directory structure (/etc/icinga, etc), I prefer a similar structure for Icinga Web. The following configure line will therefore make sure that Icinga Web uses /etc and /var:

./configure --with-api-cmd-file=/var/lib/icinga/rw/icinga.cmd --with-conf-dir=/etc/icinga-web --with-log-dir=/var/log/icinga-web --with-cache-dir=/var/cache/icinga-web

To compile everything, you’ll need the “make” command. In most cases this will already be installed, however on a fresh system like mine, I had to install it (and a lot of dependencies) first using

apt-get install build-essentials

Now it’s time to compile Icinga Web and add the neccessary configuration to Apache. This will create a file icinga-web.conf in /etc/apache2/conf.d containing the neccessary Alias directives.

make install
make install-apache-config

Icinga Web uses the rewrite module for Apache, so let’s activate it:

a2enmod rewrite
service apache2 restart

3) Configuration

Your Icinga Web installation is now accessible at /icinga-web, however it’ll greet you with a nice critical exception message. But don’t be discouraged, help is coming! :-)
Icinga Web uses it’s own tables to store information (like it’s own user database, last open crons, etc), so we need to create these tables and provide Icinga Web with the database credentials. It is recommended to use a seperate database for Icinga Web (e.g. icinga_web), however it is also possible to keep the Icinga Web tables in the same database as the rest of Icinga. I chose the second method to reduce the number of databases we have and keep it together for easier backup.

In the following instructions, I’ll use a couple of placeholders that you’ll need to replace with your actual values. The placeholders will be

  • [iwuser] for the database user for your Icinga Web database. You can use the same user as IDOUtils, but it’s probably a good idea to use a different one.
  • [iwpass] the password for iwuser
  • [iwdb] The database name for Icinga Web. This is either the icinga database if you want to  use the same as IDOUtils or for example icinga_web if you prefer to use a different database
  • [idouser] This is the database username from Part 1 (usually icinga_idoutils)
  • [idopass] The password for idouser, if you had one generated, you can look it up in /etc/icinga/ido2db.cfg
  • [idodb] The database for IDOUtils, probably just ‘icinga’

If you want to use a different database, now would be the time to create it (either using a GUI tool like phpMyAdmin or MySQL workbench or using the “create database <name>” sql command) and grant the neccesary user rights for your [iwuser] (see the Wiki for a GRANT command, or use the GUI tool).

Icinga Web provides a SQL script to create the neccessary tables. Let’s import the schema:

mysql -u [iwuser] -p -h hostname.of.database.serv.er [iwdb] < /usr/local/icinga-web/etc/schema/mysql.sql

If your MySQL server is running locally, you can skip the hostname parameter. The command will ask you for the password of [iwuser] and afterwards create the structure in [iwdb].

Now that we installed the software, configured Apache, created a database and tables, all that’s left to do is tell Icinga Web how to connect to the database. While this sounds simple enough, it was ironically the most difficult part in my opinion. Icinga Web uses Doctrine, a database abstraction layer which is good because it dramatically increases the number of supported databases. However it also means that the configuration is stored in XML format.

The database configuration is stored in /etc/icinga-web/databases.xml  (there’s also an equally named file in /usr/local/icinga-web/app/config, don’t edit that file!). In this file there are two sections, the first one to configure the icinga-web database connection, the second one for the IDOUtils database connection. All you have to change is the connection strings (in <ae:parameter name=”dsn”>). So basically change:

mysql://icinga_web:icinga_web@localhost:3306/icinga_web to mysql://[iwuser]:[iwpass]@[hostname or localhost]:3306/[iwdb]
and
mysql://icinga:icinga@localhost:3306/icinga to mysql://[idouser]:[idopass]@[hostname or localhost]:3306/[idodb]

Replace the placeholders with the actual values from your environment. As a reminder: The IDOUtils user and password can be found in /etc/icinga/ido2db.cfg.
Important: Uncomment both sections. XML comments start with <!– and end with –>. You can find those around each db:database tag. Just remove them so your modified configuration becomes active. This is the most likely cause if you still receive critical exceptions after you followed all steps.

The last step is to run /usr/local/icinga-web/bin/clearcache.sh to remove any configuration cache and ensure Icinga Web will load the new configuration. If you open /icinga-web now, you should be greeted with a login prompt. The default username is “root” and default password is “password”.

If you still get a critical exception, make sure you really  removed the comment characters in databases.xml.

4) Using Icinga Web

Icinga Web should now be fully usable. The first thing to do is obviously to change your password. Afterwards feel free to explore Icinga Web and its cronks. Cronks are like widgets or gadgets or modules or plugins in other software, they show data or subsets of data from IDOUtils in Icinga Web. An example for a cronk would be “Unhandled service problems”.

Tackle Cronk

Icinga Web 1.6 introducted a new (experimental!) cronk named Tackle which shows a list of all your hosts with their service status as a stacked (horizontal) bar. An example can be found in the 1.6 docs. This is one of my favorite features of Icinga Web and you should definitively check it out. Unfortunately it is not stable enough to be released just yet and is therefore deactivated in the upcoming 1.6.2 release. If you want to use it anyway (and are aware that it might not work as expected yet!), you can enable it again by reversing the changeset attached to the bug issue. All you need to do is basically change “hide” and “disabled” back to false in the Tackle configuration in /usr/local/icinga-web/app/modules/Cronks/config/cronks_misc.xml. But remember, that it still is highly experimental – you’ve been warned.

 

This is the end of Part 2. If everything went well you should now have a working Icinga installation along with the new Icinga Web interface. Unfortunately we are still only monitoring localhost (unless you changed the configuration files yourself). The next part should take care of that however. I will show you how to install NagiosQL, a web based configuration tool for Nagios and Icinga which allows you to change or add hosts and services without getting your hands dirty with configuration files ;-)

If you have problems, suggestions or remarks, feel free to leave a comment or sent me an email.

Icinga 1.6 As A Monitoring Solution On Ubuntu 12.04 (Part 1: Installation)

Feb 202012

Since it was time to get rid of Nagios and upgrade to a newer monitoring solution, I decided to check out several different programs and finally decided on Icinga. The main reasons for this decision were the facts that it was actively developed, allowed us to continue using a lot of our existing Nagios plugins and is available as an Ubuntu package which makes updating in the future a lot easier.  In this post I will describe the neccessary steps to get Icinga up and running along with idoutils (and ido2db), to write the data into a database (which will allow us to fetch the data and display it in external systems, e.g using the icinga php api).

All parts of my Icinga 1.6 As A Monitoring Solution On Ubuntu 12.04 series:

Software used:

  • Ubuntu 12.04
  • Icinga 1.6.1 (from Ubuntu repository)
  • Idoutils
  • MySQL 5
  • Apache 2

Note: At the time of this writing Ubuntu 12.04 was still in Alpha, so there is a (slim) chance that some of the steps change with the final release at the end of April.

If you have a different environment or need help with the installation or configuration, the Icinga team provides great documentation with the official Icinga Docs and Wiki. Both are definitely worth checking out.

This tutorial will start with a fresh Ubuntu server installation no additional packages installed. If you already have a live system, some steps might not be neccessary.

1) Installing the neccessary software

Lets start with installing dbconfig-common. This package is used during the installation of the remaining software to ask the user questions about database access:

apt-get install dbconfig-common

Now you have two choices: Either you’ll use a local MySQL server in which case, simply install it using apt-get install mysql-server. If you plan to use a remote server (as it’ll be the case in most production systems), edit /etc/dbconfig-common/config and make sure dbc_remote_questions_default is set to ‘true’. This will make dbconfig ask about the configuration of a remote server.

Now install the Icinga and Iciga-Idoutils packages, this should install a whole lot of dependencies as well:

apt-get install icinga icinga-idoutils

During the installation you’ll need to provide database access parameters (like hostname if you enabled remote questions and the password of your MySQL root user). Be sure to provide valid answers because the information will be written to several files and changing them afterwards is annoying.

Now the first part is done and you should already be able to access your Icinga installation at http://yourhostname.com/icinga with the username ‘icingaadmin’ and the password you entered during installation.

“But this was so easy, why the hell did you write a blog post about it?” – Well, while Icinga is in fact already running and checking your local services, unfortunately idoutils and external commands don’t work… yet.

2) Configuring idoutils and ido2db

While the setup program should’ve already created the correct config files, ido2db wasn’t enabled by default (you could run the init.d script but nothing happened).

To enable ido2db, edit /etc/default/icinga and set IDO2DB to ‘yes’ and run /etc/init.d/ido2db start. This should start the ido2db daemon (check in your process list). If ido2db isn’t running, look in /var/log/syslog for any clues what went wrong. In my case it worked like a charm after I set IDO2DB to yes.

Now we need to tell Icinga to actually use ido2db. Go to /etc/icinga/modules and rename idoutils.cfg-sample to idoutils.cfg. If you’d restart Icinga now, you’d probably get a nasty “idomod.o could not be found” error. This is because in the config file we just renamed the path to idomod.o is set to /usr/sbin. It was however located at /usr/lib/icinga/idomod.o for me. I’d recommend to run ‘updatedb’ and afterwards ‘locate idomod.o’ to figure out where the file is on your installation. Now edit the config file (if neccessary) and adjust the path setting to the correct value. Now it’s time to restart Icinga (/etc/init.d/icinga restart) and the database should start to fill out with data. Check /var/log/syslog to make sure no errors appear. If you have errors you can’t resolve yourself, leave a comment below.

Note: I had the problem, that database connection failed. This was a corner case because the installation script added the icinga-idoutils user with the short hostname of the local machine. Since the database server had a different domain however, the connection failed. I fixed it by changing the host name of the user to ‘%’. This shouldn’t happen for most people however.

3) Enabling external commands

External commands allow the web server (or any other user) to issue commands to Icinga. This is useful to force a service check using the web interface after you fixed an error (and don’t want to wait for the next automatic check) and to disable notifications etc. By default external commands are disabled, to enable them, edit /etc/icinga/icinga.conf and set “check_external_commands” to 1. Now restart Icinga. In a world full of green gras and honey this would suffice. Unfortunately some more steps were required (at least one of them appears to be a bug in the installation script, so chances are, it’s gone by the time you read this):

  1. Add the webserver user to the nagios group. To do this, edit /etc/group and find the nagios entry (probably at the end of the file) and after the color add “www-data”.  Save. This is required so that the web server may write to the command pipe used for external commands. The same step is required for any other user that should be able to use external commands.
  2. The directory /var/lib/icinga/rw which contains the command pipe was in my case only accessible by the nagios user (not the nagios group). I expect this to be a bug, however, you should probably check anyway. If it has only 700 permissions, allow the nagions group access by executing “chmod g+rx /var/lib/icinga/rw”.Edit: As Michael Friedrich and Alexander Wirt (see comments) pointed out, this modification wouldn’t survive an update. Here’s the correct way to do it (from README.Debian): 
    /etc/init.d/icinga stop
dpkg-statoverride --update --add nagios nagios 2710 /var/lib/icinga/rw
dpkg-statoverride --update --add nagios nagios 751 /var/lib/icinga
/etc/init.d/icinga start

    (If you haven’t added www-data to the nagios group, change the second ‘nagios’ in the first statoverride command to your webserver group (e.g. www-data))

Now you should be able to execute commands using the web interface (like forcing a service check). If you still encounter problems, the log file (/var/log/syslog or /var/log/icinga/icinga.log) are usually good pointers to the cause of the problem.

This concludes the first part. You now have a basic Icinga system running and are already actively monitoring the local host, which might not be as exciting as you hoped, but is a start nevertheless. If you are familiar with Nagios configuration, you can now go nuts on the Icinga config files, which basically follow the same schema. The paths to each config file can be found in /etc/icinga.conf.  If you have problems following a certain step or getting the installation so far to work, feel free to leave a comment or send me an email and I’ll try to help as good as I can.

In the coming days I’ll try to write a bit about getting Icinga Web, the new Ajax-powered web interface to work. Another topic I’d like to touch is using NagiosQL as a web based configuration option for Icinga and last but not least add a few remote hosts and service checks. I’ll also provide you with a few plugin recommendation and an example for monitoring printers for their toner and paper levels. Stay tuned.