All parts of my Icinga 1.6 As A Monitoring Solution On Ubuntu 12.04 series:

• Part 1: Basic Installation (including the Classic Web Interface)
• Part 2: Icinga Web (New Ajax Interface) [this post]
• Part 3: NagiosQL (Web-based Configuration Interface for Icinga) [coming soon]
• Part 4: Staying Up-To-Date On Host Status (external Tools)
• Part 5: Monitoring Printers (over SNMP)
• Part 6: Monitoring Windows and Linux Maschines
• Part 7: Using Icinga Data In Your Own Applications (using REST API)

1) Prerequisites

For the next steps you’ll need some additional packages (mostly PHP modules for Apache) and if you want to fetch Icinga using Git, the Git client, so let’s install them:

apt-get install php5 php5-cli php-pear php5-xmlrpc php5-xsl php5-gd php5-ldap php5-mysql
(optional) apt-get install git

2) Installation

Now you’ll need the Icinga Web source. Unfortunately, at the time of this writing, there is no Ubuntu package available.  I cloned the Git repository, since I wanted the upcoming 1.6.2 version of Icinga Web, however in most cases (and unless told otherwise by a developer), you’re probably better off downloading a stable package from the Icinga Website.

Using a package:

• Go to https://www.icinga.org/download/packages/ and download the latest stable version of Icinga Web to your server (e.g. using wget <url>)
• Extract the contents using

  tar vfxz icinga-web-x.x.x.tar.gz

• Change into the new directory (cs icinga-web-x.x.x)

Using git:

• git clone git://git.icinga.org/icinga-web.git

• cd icinga-web

Now it’s time to configure the installation process. If we don’t specify target directories, everything will go to /usr/local/inciga-web/.., however since we installed Icinga already using packages and therefore use the global directory structure (/etc/icinga, etc), I prefer a similar structure for Icinga Web. The following configure line will therefore make sure that Icinga Web uses /etc and /var:

./configure --with-api-cmd-file=/var/lib/icinga/rw/icinga.cmd --with-conf-dir=/etc/icinga-web --with-log-dir=/var/log/icinga-web --with-cache-dir=/var/cache/icinga-web

To compile everything, you’ll need the “make” command. In most cases this will already be installed, however on a fresh system like mine, I had to install it (and a lot of dependencies) first using

apt-get install build-essentials

Now it’s time to compile Icinga Web and add the neccessary configuration to Apache. This will create a file icinga-web.conf in /etc/apache2/conf.d containing the neccessary Alias directives.

make install
make install-apache-config

Icinga Web uses the rewrite module for Apache, so let’s activate it:

a2enmod rewrite
service apache2 restart

3) Configuration

Your Icinga Web installation is now accessible at /icinga-web, however it’ll greet you with a nice critical exception message. But don’t be discouraged, help is coming! :-)
Icinga Web uses it’s own tables to store information (like it’s own user database, last open crons, etc), so we need to create these tables and provide Icinga Web with the database credentials. It is recommended to use a seperate database for Icinga Web (e.g. icinga_web), however it is also possible to keep the Icinga Web tables in the same database as the rest of Icinga. I chose the second method to reduce the number of databases we have and keep it together for easier backup.

In the following instructions, I’ll use a couple of placeholders that you’ll need to replace with your actual values. The placeholders will be

• [iwuser] for the database user for your Icinga Web database. You can use the same user as IDOUtils, but it’s probably a good idea to use a different one.
• [iwpass] the password for iwuser
• [iwdb] The database name for Icinga Web. This is either the icinga database if you want to  use the same as IDOUtils or for example icinga_web if you prefer to use a different database
• [idouser] This is the database username from Part 1 (usually icinga_idoutils)
• [idopass] The password for idouser, if you had one generated, you can look it up in /etc/icinga/ido2db.cfg
• [idodb] The database for IDOUtils, probably just ‘icinga’

If you want to use a different database, now would be the time to create it (either using a GUI tool like phpMyAdmin or MySQL workbench or using the “create database <name>” sql command) and grant the neccesary user rights for your [iwuser] (see the Wiki for a GRANT command, or use the GUI tool).

Icinga Web provides a SQL script to create the neccessary tables. Let’s import the schema:

mysql -u [iwuser] -p -h hostname.of.database.serv.er [iwdb] < /usr/local/icinga-web/etc/schema/mysql.sql

If your MySQL server is running locally, you can skip the hostname parameter. The command will ask you for the password of [iwuser] and afterwards create the structure in [iwdb].

Now that we installed the software, configured Apache, created a database and tables, all that’s left to do is tell Icinga Web how to connect to the database. While this sounds simple enough, it was ironically the most difficult part in my opinion. Icinga Web uses Doctrine, a database abstraction layer which is good because it dramatically increases the number of supported databases. However it also means that the configuration is stored in XML format.

The database configuration is stored in /etc/icinga-web/databases.xml  (there’s also an equally named file in /usr/local/icinga-web/app/config, don’t edit that file!). In this file there are two sections, the first one to configure the icinga-web database connection, the second one for the IDOUtils database connection. All you have to change is the connection strings (in <ae:parameter name=”dsn”>). So basically change:

mysql://icinga_web:icinga_web@localhost:3306/icinga_web to mysql://[iwuser]:[iwpass]@[hostname or localhost]:3306/[iwdb]
and
mysql://icinga:icinga@localhost:3306/icinga to mysql://[idouser]:[idopass]@[hostname or localhost]:3306/[idodb]

Replace the placeholders with the actual values from your environment. As a reminder: The IDOUtils user and password can be found in /etc/icinga/ido2db.cfg.
Important: Uncomment both sections. XML comments start with <!– and end with –>. You can find those around each db:database tag. Just remove them so your modified configuration becomes active. This is the most likely cause if you still receive critical exceptions after you followed all steps.

The last step is to run /usr/local/icinga-web/bin/clearcache.sh to remove any configuration cache and ensure Icinga Web will load the new configuration. If you open /icinga-web now, you should be greeted with a login prompt. The default username is “root” and default password is “password”.

If you still get a critical exception, make sure you really  removed the comment characters in databases.xml.

4) Using Icinga Web

Icinga Web should now be fully usable. The first thing to do is obviously to change your password. Afterwards feel free to explore Icinga Web and its cronks. Cronks are like widgets or gadgets or modules or plugins in other software, they show data or subsets of data from IDOUtils in Icinga Web. An example for a cronk would be “Unhandled service problems”.

Tackle Cronk

Icinga Web 1.6 introducted a new (experimental!) cronk named Tackle which shows a list of all your hosts with their service status as a stacked (horizontal) bar. An example can be found in the 1.6 docs. This is one of my favorite features of Icinga Web and you should definitively check it out. Unfortunately it is not stable enough to be released just yet and is therefore deactivated in the upcoming 1.6.2 release. If you want to use it anyway (and are aware that it might not work as expected yet!), you can enable it again by reversing the changeset attached to the bug issue. All you need to do is basically change “hide” and “disabled” back to false in the Tackle configuration in /usr/local/icinga-web/app/modules/Cronks/config/cronks_misc.xml. But remember, that it still is highly experimental – you’ve been warned.

This is the end of Part 2. If everything went well you should now have a working Icinga installation along with the new Icinga Web interface. Unfortunately we are still only monitoring localhost (unless you changed the configuration files yourself). The next part should take care of that however. I will show you how to install NagiosQL, a web based configuration tool for Nagios and Icinga which allows you to change or add hosts and services without getting your hands dirty with configuration files ;-)

If you have problems, suggestions or remarks, feel free to leave a comment or sent me an email.

All parts of my Icinga 1.6 As A Monitoring Solution On Ubuntu 12.04 series:

• Part 1: Basic Installation (including the Classic Web Interface) [this post]
• Part 2: Icinga Web (New Ajax Interface)
• Part 3: NagiosQL (Web-based Configuration Interface for Icinga) [coming soon]
• Part 4: Staying Up-To-Date On Host Status (external Tools)
• Part 5: Monitoring Printers (over SNMP)
• Part 6: Monitoring Windows and Linux Maschines
• Part 7: Using Icinga Data In Your Own Applications (using REST API)

Software used:

• Ubuntu 12.04
• Icinga 1.6.1 (from Ubuntu repository)
• Idoutils
• MySQL 5
• Apache 2

Note: At the time of this writing Ubuntu 12.04 was still in Alpha, so there is a (slim) chance that some of the steps change with the final release at the end of April.

If you have a different environment or need help with the installation or configuration, the Icinga team provides great documentation with the official Icinga Docs and Wiki. Both are definitely worth checking out.

This tutorial will start with a fresh Ubuntu server installation no additional packages installed. If you already have a live system, some steps might not be neccessary.

1) Installing the neccessary software

Lets start with installing dbconfig-common. This package is used during the installation of the remaining software to ask the user questions about database access:

apt-get install dbconfig-common

Now you have two choices: Either you’ll use a local MySQL server in which case, simply install it using apt-get install mysql-server. If you plan to use a remote server (as it’ll be the case in most production systems), edit /etc/dbconfig-common/config and make sure dbc_remote_questions_default is set to ‘true’. This will make dbconfig ask about the configuration of a remote server.

Now install the Icinga and Iciga-Idoutils packages, this should install a whole lot of dependencies as well:

apt-get install icinga icinga-idoutils

During the installation you’ll need to provide database access parameters (like hostname if you enabled remote questions and the password of your MySQL root user). Be sure to provide valid answers because the information will be written to several files and changing them afterwards is annoying.

Now the first part is done and you should already be able to access your Icinga installation at http://yourhostname.com/icinga with the username ‘icingaadmin’ and the password you entered during installation.

“But this was so easy, why the hell did you write a blog post about it?” – Well, while Icinga is in fact already running and checking your local services, unfortunately idoutils and external commands don’t work… yet.

2) Configuring idoutils and ido2db

While the setup program should’ve already created the correct config files, ido2db wasn’t enabled by default (you could run the init.d script but nothing happened).

To enable ido2db, edit /etc/default/icinga and set IDO2DB to ‘yes’ and run /etc/init.d/ido2db start. This should start the ido2db daemon (check in your process list). If ido2db isn’t running, look in /var/log/syslog for any clues what went wrong. In my case it worked like a charm after I set IDO2DB to yes.

Now we need to tell Icinga to actually use ido2db. Go to /etc/icinga/modules and rename idoutils.cfg-sample to idoutils.cfg. If you’d restart Icinga now, you’d probably get a nasty “idomod.o could not be found” error. This is because in the config file we just renamed the path to idomod.o is set to /usr/sbin. It was however located at /usr/lib/icinga/idomod.o for me. I’d recommend to run ‘updatedb’ and afterwards ‘locate idomod.o’ to figure out where the file is on your installation. Now edit the config file (if neccessary) and adjust the path setting to the correct value. Now it’s time to restart Icinga (/etc/init.d/icinga restart) and the database should start to fill out with data. Check /var/log/syslog to make sure no errors appear. If you have errors you can’t resolve yourself, leave a comment below.

Note: I had the problem, that database connection failed. This was a corner case because the installation script added the icinga-idoutils user with the short hostname of the local machine. Since the database server had a different domain however, the connection failed. I fixed it by changing the host name of the user to ‘%’. This shouldn’t happen for most people however.

3) Enabling external commands

External commands allow the web server (or any other user) to issue commands to Icinga. This is useful to force a service check using the web interface after you fixed an error (and don’t want to wait for the next automatic check) and to disable notifications etc. By default external commands are disabled, to enable them, edit /etc/icinga/icinga.conf and set “check_external_commands” to 1. Now restart Icinga. In a world full of green gras and honey this would suffice. Unfortunately some more steps were required (at least one of them appears to be a bug in the installation script, so chances are, it’s gone by the time you read this):

1. Add the webserver user to the nagios group. To do this, edit /etc/group and find the nagios entry (probably at the end of the file) and after the color add “www-data”.  Save. This is required so that the web server may write to the command pipe used for external commands. The same step is required for any other user that should be able to use external commands.
2. The directory /var/lib/icinga/rw which contains the command pipe was in my case only accessible by the nagios user (not the nagios group). I expect this to be a bug, however, you should probably check anyway. If it has only 700 permissions, allow the nagions group access by executing “chmod g+rx /var/lib/icinga/rw”.Edit: As Michael Friedrich and Alexander Wirt (see comments) pointed out, this modification wouldn’t survive an update. Here’s the correct way to do it (from README.Debian):

   /etc/init.d/icinga stop
   dpkg-statoverride --update --add nagios nagios 2710 /var/lib/icinga/rw
   dpkg-statoverride --update --add nagios nagios 751 /var/lib/icinga
   /etc/init.d/icinga start

   (If you haven’t added www-data to the nagios group, change the second ‘nagios’ in the first statoverride command to your webserver group (e.g. www-data))

Now you should be able to execute commands using the web interface (like forcing a service check). If you still encounter problems, the log file (/var/log/syslog or /var/log/icinga/icinga.log) are usually good pointers to the cause of the problem.

This concludes the first part. You now have a basic Icinga system running and are already actively monitoring the local host, which might not be as exciting as you hoped, but is a start nevertheless. If you are familiar with Nagios configuration, you can now go nuts on the Icinga config files, which basically follow the same schema. The paths to each config file can be found in /etc/icinga.conf.  If you have problems following a certain step or getting the installation so far to work, feel free to leave a comment or send me an email and I’ll try to help as good as I can.

In the coming days I’ll try to write a bit about getting Icinga Web, the new Ajax-powered web interface to work. Another topic I’d like to touch is using NagiosQL as a web based configuration option for Icinga and last but not least add a few remote hosts and service checks. I’ll also provide you with a few plugin recommendation and an example for monitoring printers for their toner and paper levels. Stay tuned.

I’ll provide all necessary instructions and hopefully spare some of you some time.

Requirements:

This tutorial assumes the following environment:

• Windows 2008 Active Directory  (Domain: EXAMPLE.COM, Domain Controller: dc01.example.com)
• Clients: Windows XP or newer
• Linux Webserver (web.example.com) running Apache 2.2, examples are provided for Ubuntu 10.04 LTS

ktpass /princ HTTP/#full_hostname_of_your_linux_server#@#domain# /mapuser #username#@#domain# /pass #random password# /crypto rc4-hmac-nt /ptype KRB5_NT_SRV_HST /out #filename#.keytab

ktpass /princ HTTP/web.example.com@EXAMPLE.COM /mapuser kerbweb@EXAMPLE.COM /pass randompasswordetc /crypto rc4-hmac-nt /ptype KRB5_NT_SRV_HST /out web.keytab

apt-get install krb5-user

[libdefaults]
        default_realm = EXAMPLE.COM
        default_tgs_enctypes = arcfour-hmac-md5 RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
        default_tkt_enctypes = arcfour-hmac-md5 RC4-HMAC DES-CBC-MD5 DES-CBC-CRC

[domain_realm]
        web.example.com = EXAMPLE.COM

[realms]
        EXAMPLE.COM = {
                kdc = dc01.example.com
                admin_server = dc01.example.com
                default_domain = example.com
        }

kinit _username_@EXAMPLE

klist

kinit -k -t /etc/apache2/web.keytab HTTP/web.example.com@EXAMPLE.COM

kvno HTTP/web.example.com@EXAMPLE.COM
klist -e

klist -e -k /home/cippool/servicebackend.keytab

apt-get install libapache2-mod-auth-kerb
a2enmod auth_kerb
/etc/init.d/apache2 restart

AuthName "Kerberos Protected Directory"
AuthType Kerberos
Krb5Keytab /etc/apache2/web.keytab
KrbAuthRealms EXAMPLE.COM
KrbServiceName HTTP
KrbMethodNegotiate on
KrbVerifyKDC on
require valid-user

a2enmod authnz_ldap
/etc/init.d/apache2 restart

AuthLDAPURL "ldap://dc01.example.com/dc=example,dc=com?sAMAccountName"
AuthLDAPBindDN ldapuser@example.com
AuthLDAPBindPassword ldappass
Require ldap-group CN=Webusers,OU=Groups,DC=example,DC=com

First of all I’d like to say that I absolutely love webservices. They are one of the greatest things since the invention of sliced bread. They allow interaction between different parties using a standardized communication protocol (well mostly at least) and don’t require any special client software.

Basically there are three kinds of webservices:
Proprietary Webservices: The least “open” kind of webservices. While they usually use a standard protocol (HTTP), the usage is completely dependent on the service provider. A client needs to know the full API to communicate with such a service.  They are often used by URLs such as this:

http://example.ws/myApplication/addUser.php?firstname=Michael&Lastname=Meier&favColor=blue

In other cases the whole interaction is defined in the payload and HTTP is only used for communication. I’d advise to avoid creating new proprietary webservices nowadays, but you can’t always avoid existing services.

RESTful Webservices: My personal preference for most tasks. The URLs usually represent objects on the service side and the default HTTP methods (GET=Read, PUT=Update, POST=Add/Insert, DELETE=Delete) are used to access those objects (similiar to CRUD in databases, but not exactly the same). Errors are indicated using the appropriate HTTP error code and a message in the body. They can use basic authentification provided by HTTP but further features like transactions usually need to be implemented in the payload. RESTful services don’t save any information on the client state which makes creating such services a lot easier. Calls typically look like this:
GET http://example.ws/myApplication/user/12345 to get the information for user 12345
PUT http://example.ws/myApplication/user/12345 to update its information (data in body)

WS-* (SOAP) Webservices: Often called “Big Webservices”, these offer the most functionality. There are a lot of WS-* extensions that add functionality to the web service protocol like Reliability, Security, Transactions and advanced Authentification. SOAP uses (xml-style) messages that can be transfered using a lot of different protocols (usually HTTP, but FTP, Email, even MMS would also be possible). SOAP services keep track of the client state to allow process chains. They are also described by a WSDL (WebService Description Language) file which contains information how to contact the webservice and which methods are available (including parameters), making it possible to use the service without much inside knowledge.

After reading this one might think that SOAP services are the go-to solution for all your webservice needs. Unfortunately that seems to be the conclusion many executives have come to as well. What they don’t realize is that SOAP generates a huge overhead and processing SOAP requests and responses takes a lot more work than handling the simpler RESTful operations. Getting catalog information from a RESTful service is as easy as calling the responsible service with the category as a parameter like “GET http://example.ws/catalog/programming”. This operation in SOAP would look like this:

POST http://example.ws/catalogSvc
<?xml version="1.0"?>
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
  <soap:Header />
  <soap:Body>
    <m:GetCatalogList xmlns:m="http://example.ws/catalog">
      <m:Category>Programming</m:Category>
    </m:GetCatalogList>
  </soap:Body>
</soap:Envelope>

“Slightly” more complicated and that is without any soapHeader information.

So using SOAP services is bad? Not at all. They are great when they are needed. But it is important to know when they are needed and when they’re just bloat (and often slower). Let’s get back to my bad experience mentioned above. I’ll use the catalog example above. We needed a web service that would provide us with 1) a list of all categories (multiple tiers, so preferably a tree like structure), 2) the list of books for certain category and 3) detail information for each book. The service would only be accessed using PHP from a limited number of (known) users.

From this information you should come to the conclusion that a SOAP service would probably be a bit overkill but a single RESTful service with three interfaces  (/catalog, /catalog/category/x, /book/x) would do perfectly fine especially since we only need read access. If for some reason you really, really, really want to use a SOAP service (a client may already exist which uses SOAP), a single service with three operations (e.g. GetCategoryList, GetBookList, GetBookDetails) would be the smart choice.

Well, I mentioned “the ugly” in the title (and I didn’t mean proprietary services). This is what we received as a web service in the catalog case: Not one, not two, but three separate SOAP services with different URLs for each of the operations. Just don’t do stuff like that. Seriously. Don’t. There is hardly ever a reason to spread information that belongs together over different services. It’s BAD DESIGN.

"/raid0/data/usershare" 172.x.x.x (rw,no_root_squash,sync,anonuid=99,anongid=99)

For test purposes I changed it to async and sent the reload command to the NFSd (/app/bin/rc/rc.nfsd reload). Note: Do not attempt to restart the NFSd instead. For some reason it completely stopped working when doing so and the only way to get it back up was to disable and enable the NFS module using the web interface or restart the machine.

After remounting the shares on the Linux box we finally had a good write performance (about 75megs/sec). The only problem was to make the async setting stick. The Thecus machines seem to completely rebuild the configuration whenever a configuration change was done using the web interface and, what was worse, when the system was rebooted. Obviously this overwrote the async change every time. The script responsible for it also resides on a read-only filesystem so changes to the script weren’t possible (and would have been reverted by a firmware upgrade anyway, I’m afraid).

To solve this issue I had to get a bit creative. There is a META module for the Thecus N5200 which runs all executable files in a certain directory on startup. Unfortunately it can’t be installed on the N8800pro by default. To get it to work, download the archive and the link above, extract it on a box with the make command available (your typical Linux box will do) and change to META/Configure. You’ll need to adjust the install.rdf file (XML) and change the NasType n5200 to n8800 and set NasVersion to the version shown at the bottom right in your web interface (3.05.02.2 in my case). Afterwards go back to directory including the META.mod file and run ‘make’. It should generate a new META.mod which can now be installed on the N8800Pro.

Afterwards, SSH to the N8800Pro and change to  /raid/data/module/META/system/etc/startup. Executable files in this directory will be run at system startup.
Create a file (e.g. nfs_async.sh) with the following content:

#!/bin/bash
sed -i 's/,sync,/,async,/g' /etc/exports
/app/bin/rc/rc.nfsd reload

and make it executable (chmod +x nfs_async.sh). Now the async setting should be set to all your exports on reboot.

This still leaves the problem that whenever the share configuration is changed it will be overwritten and the machine has either to be restarted (to run META module) or the script has to be called manually. Since both ways are not really desirable, the next step will be to make sure that the exports file is adjusted during runtime when neccessary. I’ll update this post when I got a working solution. It will probably be based on META to update the crontab file on startup and a cron script that will periodically check whether the /etc/exports file was changed (or has sync entries) and fix it when neccessary.

Attention: Using the async option might have unexpected consequences. We haven’t been running the configuration long enough to be sure that there are no side-effects. Use this method at your own risk and try it with less important data for a while before using it on productive files.

Update 2011-08-13: Matt Robinson from ISP Dr Internet Solutions was kind enough to provide the following script for all readers. It can remotely adjust the async setting and perform a short performance test. If your “sys” user has a password, you might want to store a public key on the Thecus machines.

# Fix up the nas async nfs export to fix speed for Thecus NASs
#       — run this on any nas after it is rebooted etc
#
#  Usage: fixnas.sh nas_name [ fix | show | test ]
#
#  Put this script in a directory where the nas_host_name is
#    another dir name and mount point in side it, ie:
#
#       …./remote-mounts/fixnas.sh
#      …./remote-mounts/my-nas-01/
#      …./remote-mounts/my-nas-02/
#
#  MR: 12-08-2011   matts@ispdr.net.au  thanx to Michael Meier

# Local Progs
DD=”/bin/dd”
SSH=”/usr/bin/ssh”

# Thecus ssh user
USER=”sys”

# Test Parms
COUNT=”100000″
FILE=”SPEED-TEST”

# Commands To Run On Nas
SHO_CMD=”cat /etc/exports”
FIX_CMD=”sed -i ‘s/,sync,/,async,/g’ /etc/exports; /app/bin/rc/rc.nfsd reload”

# Check Args
[ $# -ne 2 ] && echo -e “nUsage: $0 nas_name [ fix | show | test ] n” && exit 1
NAS=”$1″

# Parse Command And Run On Host
case “$2″ in

fix)            # Run Fix
echo -e “nFixing Async Exports On $NAS”
$SSH “$USER@$NAS” “$FIX_CMD”
echo -e “Done!n”
;;

show)           # Run Show
echo -e “nNFS Exports On $NAS:”
$SSH “$USER@$NAS” “$SHO_CMD”
echo
;;

test)           # Run Test
echo -e “nTesting Write To $NAS:”
$DD “if=/dev/zero” “of=$NAS/$FILE” “bs=8k” “count=$COUNT”
rm -f “$NAS/$FILE”
echo
;;

*)              # Invalid Command
echo -e “nInvalid Commandn”
exit 1
esac

exit 0

Blog posts will be in English to address a wider audience unless it’s a topic only relevant to Germans in which case it’ll be (obviously) in German.

I don’t consider myself to be perfect and might make mistakes so feel free to give (constructive) feedback and corrections. All that’s left to say for now is: Have fun :-)