SystemAdministration/Icinga

Sentinel: Icinga / Server & Services Monitoring

introduction

note: base directory will be /etc/icinga/ unless stated otherwise

note2: These examples will use 'carbon.li694-22'

Access

Our Icinga installation is named 'Sentinel' which means "a soldier or guard whose job is to stand and keep watch", you can find Sentinel here and login with your Keberos username and password.

Kerberized SSH connections

We use AutoSSH to allow Icinga to connect to other nodes to execute commands and get services information back.

Internally, we can use Kerberos to jump from one host to another our Icinga installation requires HTTP Kerberos auth (via https) but we've also set up AutoSSH which allows our Icinga instance to connect to other nodes passwordless & ssh key-less to fetch information.

more information, show how we've setup the connections, etc

icinga.cfg

Information

objects

general information about objects

objects/templates

template information

objects/contacts

contacts & notifications

objects/groups

Servers can be placed in various "host groups" which is a simple way to organize servers but also to define services monitors once for an entire group.

We've got several service groups, you can see the all by doing a simple: /etc/icinga/objects/groups $ ls

For instance, all Ubuntu servers are in the 'ubuntu-server' group and we have a /etc/icinga/objects/groups/ubuntu.cfg file in which services checks that only apply to Ubuntu are defined. In this example, we're only checking for updates. (note: technically this check could also apply to Debian, a better name for this group could have been chosen in hindsight).

/etc/icinga/objects/groups/ $ cat ubuntu.cfg

define service{
	hostgroup_name			ubuntu-servers
        use                             generic-service         ; Name of service template to use
        service_description             APT Status
        check_command                   remote_one!check_apt
        }

Now every server in the host group 'ubuntu-servers' will check for updates.

objects/servers

Each server has a file in /etc/icinga/objects/servers/ which looks something like:

define host{
        use                     generic-host            ; Name of host template to use
        host_name               carbon.li694-22         ; The server hostname (ensure it's in our DNS).
        alias                   carbon                  ; A shortname
        address                 carbon.li694-22         ; The server's address.
}

Creating a new one (for instance, 'example.li694-22') should be straight forward, you simply copy carbon.li694-22 to example.li64-22:

/etc/icinga/objects/servers/ $ cp carbon.cfg example.cfg

Once that's done, you can either edit the file manually with the editor of your choice or use sed:

/etc/icinga/objects/servers/ $ sed -i 's/carbon/example/g' example.cfg