SystemAdministration/Icinga

Sentinel: Icinga / Server & Services Monitoring

Important Directories

Directory	Description
/etc/icinga/objects/groups	(service definitions for) Host Groups
/etc/icinga/objects/servers	Server Definitions
/etc/icinga/objects/templates	Various templates docs here
/etc/icinga/objects/contacts	Contact & time period config docs here
/etc/nagios-plugins/config/	Plugin configs; be sure to checkout 'remote.cfg'
/usr/lib/nagios/plugins/	Plugins

Useful links

Icinga documentation

Testing Icinga's config and reloading:

When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:

# /usr/sbin/icinga -v /etc/icinga/icinga.cfg

If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:

# service icinga reload

Access

Our Icinga installation is named 'Sentinel' which means "a soldier or guard whose job is to stand and keep watch", you can find Sentinel here and login with your Keberos username and password.

Kerberized SSH connections

We use AutoSSH to allow Icinga to connect to other nodes to execute commands and get services information back.

Internally, we can use Kerberos to jump from one host to another our Icinga installation requires HTTP Kerberos auth (via https) but we've also set up AutoSSH which allows our Icinga instance to connect to other nodes passwordless & ssh key-less to fetch information.

more information, show how we've setup the connections, etc

icinga.cfg

The file /etc/icinga/icinga.cfg is heavily commented. I suggest you simply look it over.

Servers

Icinga's "host" documentation

Adding a server

Each server has a file in /etc/icinga/objects/servers/ which looks something like:

define host{
        use                     generic-host            ; Name of host template to use
        host_name               carbon.li694-22         ; The server hostname (ensure it's in our DNS).
        alias                   carbon                  ; A shortname
        address                 carbon.li694-22         ; The server's address.
}

Creating a new one (for instance, 'example.li694-22') should be straight forward, you simply copy carbon.li694-22 to example.li64-22:

/etc/icinga/objects/servers/ $ cp carbon.cfg example.cfg

Once that's done, you can either edit the file manually with the editor of your choice or use sed:

/etc/icinga/objects/servers/ $ sed -i 's/carbon/example/g' example.cfg

Modifying a server

Simply edit the server object in /etc/icinga/objects/servers/

Removing a server

If you want to remove a server, remove the config and check /etc/icinga/objects/groups/hostgroups_icinga.cfg and remove any references to the server.

Testing Icinga's config and reloading:

When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:

# /usr/sbin/icinga -v /etc/icinga/icinga.cfg

If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:

# service icinga reload

Host groups

Icinga's documentation

Introduction

In Icinga we can group hosts, for instance "Ubuntu Servers" and "PostgreSQL servers" and so on. All of our servers are members of the 'all' host group, so instead of defining services checks for memory & harddisk usage (to name a few) per server, we simply define that check in the 'all' host group and effectively, all of our servers are now being monitored for memory & harddisk usage (and a lot more, check /etc/icinga/objects/groups/all.cfg).

One server can be a member of many host groups and we've got quite a few. We've got the "ubuntu-servers" group, the "postgresql-servers" group and a few more.

Define a hostgroup

We define host groups in /etc/icinga/objects/groups/hostgroup_icinga.cfg, these look a lot like (comments in line):

; Start with the define statement.
define hostgroup {
        hostgroup_name  ubuntu-servers ; Let's give the group a name.
		alias           Ubuntu Servers ; Let's give it a "more" friendly name.
		members         boron.li694-22, helium.li694-22, carbon.li694-22, lithium.li694-22, nitrogen.li694-22, oxygen.li694-22, hydrogen.li694-22 ; It's members.
        }

define hostgroup {
        hostgroup_name  postgresql-servers ; Let's give the group a name.
                alias           PostgreSQL servers ; Let's give it a "more" friendly name.
                members         carbon.li694-22, boron.li694-22 ; It's members.
        }

As you can no doubt see, the definitions are relatively straight forward. Let's say, we install PostgreSQL on our server by the name of "helium"; then we'd simply add helium.li694-22 as a member of the postgresql-servers group and voila, now helium's PostgreSQL is also monitored. (note: monitoring PostgreSQL in itself involves a few more steps which are omitted to keep the explanation simple).

Adding a new host group

Adding a new host group is simple, you copy one of the "hostgroup" definitions to the end of /etc/icinga/objects/groups/hostgroup_icinga.cfg and change the group's name, alias & you define which servers are a member. So let's say, you want to monitor webservers, you would add the following definition (note, we're already monitoring those, it's just an example):

# A list of our web servers
define hostgroup {
        hostgroup_name  http-servers
                alias           HTTP servers
                members         hydrogen.li694-22, carbon.li694-22, boron.li694-22, beryllium.li694-22
        }

Now, this alone, won't work, currently we've only defined the host group but we haven't defined any services, let's do that in the next chapter.

Modifying a host group

Simply modify the appropriate definition in /etc/icinga/objects/groups/hostgroup_icinga.cfg

Removing a host group

Simply remove the definition from /etc/icinga/objects/groups/hostgroup_icinga.cfg

Monitoring Services

junk

As you can see, we give each group a name, an alias (friendly name, mostly) & we define it's members. Pretty straight forward right? Let's move on to the fun part.

Now instead of defining a service per server, we simply define a service per hostgroup, meaning; when we add an Ubuntu server we simply add it to the 'ubuntu-servers' host group and it'll be automatically monitored for updates, and such.

The service definitions for hostgroups are in the same directory, let's take a look at the service definitions for the ubuntu-servers group:

/etc/icinga/objects/groups/ $ cat ubuntu.cfg

define service{
	hostgroup_name			ubuntu-servers          ; Host group
        use                             generic-service         ; Name of service template to use
        service_description             APT Status              ; Service check name.
        check_command                   remote_one!check_apt    ; This is the command that's being executed.
        }

Let's say, all Ubuntu servers also run a special daemon (let's called it "speciald" - I haven't got a lot of inspiration right now), we would add the following (make sure the command definition/plugin exists) to 'ubuntu.cfg':

define service{
	hostgroup_name			ubuntu-servers          ; Host group
        use                             generic-service         ; Name of service template to use
        service_description             Speciald Status         ; Service check name.
        check_command                   remote_one!check_speciald ; This is the command that's being executed.
        }

and reload Icinga (after testing the configuration, first) and now, 'speciald' is being monitored on all Ubuntu servers.

Note, we also have 'all.cfg' which powers the "All Servers Group" - in which, you guessed it, checks for every server are defined, e.g. ssh, disk space, etc.

Testing Icinga's config and reloading:

When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:

# /usr/sbin/icinga -v /etc/icinga/icinga.cfg

If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:

# service icinga reload