SystemAdministration/Icinga

Sentinel: Icinga / Server & Services Monitoring

Important directories:

Directory	Description
/etc/icinga/objects/groups	(service definitions for) Host Groups
/etc/icinga/objects/servers	Server Definitions
/etc/icinga/objects/templates	Various templates docs here
/etc/icinga/objects/contacts	Contact & time period config docs here
/etc/nagios-plugins/config/	Plugin configs; be sure to checkout 'remote.cfg'
/usr/lib/nagios/plugins/	Plugins

Testing Icinga's config and reloading:

When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:

# /usr/sbin/icinga -v /etc/icinga/icinga.cfg

If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:

# service icinga reload

Access

Our Icinga installation is named 'Sentinel' which means "a soldier or guard whose job is to stand and keep watch", you can find Sentinel here and login with your Keberos username and password.

Kerberized SSH connections

We use AutoSSH to allow Icinga to connect to other nodes to execute commands and get services information back.

Internally, we can use Kerberos to jump from one host to another our Icinga installation requires HTTP Kerberos auth (via https) but we've also set up AutoSSH which allows our Icinga instance to connect to other nodes passwordless & ssh key-less to fetch information.

more information, show how we've setup the connections, etc

icinga.cfg

The file /etc/icinga/icinga.cfg is heavily commented. I suggest you simply look it over.

objects/groups

Firstly, we've got objects named "host groups" which are simple groups with several servers, for instance all Ubuntu servers belong to the 'ubuntu-servers' group and all servers which have a PostgreSQL service running belong to the 'postgresql-servers' group. One host/server can belong to many groups. All these groups are defined in /etc/icinga/objects/groups/hostgroups_icinga.cfg the members of these groups are also defined here.

Let's take the ubuntu-servers & postgresql-servers (respectively) as examples:

/etc/icinga/objects/groups/ $ cat hostgroups_icinga.cfg [note: only copying 'ubuntu-servers' and 'postgresql-servers' definitions out of file]

ubuntu-servers

define hostgroup {
        hostgroup_name  ubuntu-servers
		alias           Ubuntu Servers
		members         boron.li694-22, helium.li694-22, carbon.li694-22, lithium.li694-22, nitrogen.li694-22, oxygen.li694-22, hydrogen.li694-22
        }

postgresql-servers

define hostgroup {
        hostgroup_name  postgresql-servers
                alias           PostgreSQL servers
                members         carbon.li694-22, boron.li694-22
        }

As you can see, we give each group a name, an alias (friendly name, mostly) & we define it's members. Pretty straight forward right? Let's move on to the fun part.

Now instead of defining a service per server, we simply define a service per hostgroup, meaning; when we add an Ubuntu server we simply add it to the 'ubuntu-servers' host group and it'll be automatically monitored for updates, and such.

The service definitions for hostgroups are in the same directory, let's take a look at the service definitions for the ubuntu-servers group:

/etc/icinga/objects/groups/ $ cat ubuntu.cfg

define service{
	hostgroup_name			ubuntu-servers          ; Host group
        use                             generic-service         ; Name of service template to use
        service_description             APT Status              ; Service check name.
        check_command                   remote_one!check_apt    ; This is the command that's being executed.
        }

Let's say, all Ubuntu servers also run a special daemon (let's called it "speciald" - I haven't got a lot of inspiration right now), we would add the following (make sure the command definition/plugin exists) to 'ubuntu.cfg':

define service{
	hostgroup_name			ubuntu-servers          ; Host group
        use                             generic-service         ; Name of service template to use
        service_description             Speciald Status         ; Service check name.
        check_command                   remote_one!check_speciald ; This is the command that's being executed.
        }

and reload Icinga (after testing the configuration, first) and now, 'speciald' is being monitored on all Ubuntu servers.

Note, we also have 'all.cfg' which powers the "All Servers Group" - in which, you guessed it, checks for every server are defined, e.g. ssh, disk space, etc.

objects/servers

Each server has a file in /etc/icinga/objects/servers/ which looks something like:

define host{
        use                     generic-host            ; Name of host template to use
        host_name               carbon.li694-22         ; The server hostname (ensure it's in our DNS).
        alias                   carbon                  ; A shortname
        address                 carbon.li694-22         ; The server's address.
}

Creating a new one (for instance, 'example.li694-22') should be straight forward, you simply copy carbon.li694-22 to example.li64-22:

/etc/icinga/objects/servers/ $ cp carbon.cfg example.cfg

Once that's done, you can either edit the file manually with the editor of your choice or use sed:

/etc/icinga/objects/servers/ $ sed -i 's/carbon/example/g' example.cfg