SystemAdministration/Icinga
Sentinel: Icinga / Server & Services Monitoring
Important Directories
| Directory | Description |
|---|---|
| /etc/icinga/objects/groups | (service definitions for) Host Groups |
| /etc/icinga/objects/servers | Server Definitions |
| /etc/icinga/objects/templates | Various templates docs here |
| /etc/icinga/objects/contacts | Contact & time period config docs here |
| /etc/nagios-plugins/config/ | Plugin configs; be sure to checkout 'remote.cfg' |
| /usr/lib/nagios/plugins/ | Plugins |
Useful links
Testing Icinga's config and reloading:
When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:
# /usr/sbin/icinga -v /etc/icinga/icinga.cfg
If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:
# service icinga reload
Access
Our Icinga installation is named 'Sentinel' which means "a soldier or guard whose job is to stand and keep watch", you can find Sentinel here and login with your Keberos username and password.
Kerberized SSH connections
We use AutoSSH to allow Icinga to connect to other nodes to execute commands and get services information back.
Internally, we can use Kerberos to jump from one host to another our Icinga installation requires HTTP Kerberos auth (via https) but we've also set up AutoSSH which allows our Icinga instance to connect to other nodes passwordless & ssh key-less to fetch information.
more information, show how we've setup the connections, etc
icinga.cfg
The file /etc/icinga/icinga.cfg is heavily commented. I suggest you simply look it over.
Servers
Adding a server
Each server has a file in /etc/icinga/objects/servers/ which looks something like:
define host{
use generic-host ; Name of host template to use
host_name carbon.li694-22 ; The server hostname (ensure it's in our DNS).
alias carbon ; A shortname
address carbon.li694-22 ; The server's address.
}
Creating a new one (for instance, 'example.li694-22') should be straight forward, you simply copy carbon.li694-22 to example.li64-22:
/etc/icinga/objects/servers/ $ cp carbon.cfg example.cfg
Once that's done, you can either edit the file manually with the editor of your choice or use sed:
/etc/icinga/objects/servers/ $ sed -i 's/carbon/example/g' example.cfg
Modifying a server
Simply edit the server object in /etc/icinga/objects/servers/
Removing a server
If you want to remove a server, remove the config and check /etc/icinga/objects/groups/hostgroups_icinga.cfg and remove any references to the server.
Testing Icinga's config and reloading:
When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:
# /usr/sbin/icinga -v /etc/icinga/icinga.cfg
If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:
# service icinga reload
objects/groups
Firstly, we've got objects named "host groups" which are simple groups with several servers, for instance all Ubuntu servers belong to the 'ubuntu-servers' group and all servers which have a PostgreSQL service running belong to the 'postgresql-servers' group. One host/server can belong to many groups. All these groups are defined in /etc/icinga/objects/groups/hostgroups_icinga.cfg the members of these groups are also defined here.
Let's take the ubuntu-servers & postgresql-servers (respectively) as examples:
/etc/icinga/objects/groups/ $ cat hostgroups_icinga.cfg [note: only copying 'ubuntu-servers' and 'postgresql-servers' definitions out of file]
ubuntu-servers
define hostgroup {
hostgroup_name ubuntu-servers
alias Ubuntu Servers
members boron.li694-22, helium.li694-22, carbon.li694-22, lithium.li694-22, nitrogen.li694-22, oxygen.li694-22, hydrogen.li694-22
}
postgresql-servers
define hostgroup {
hostgroup_name postgresql-servers
alias PostgreSQL servers
members carbon.li694-22, boron.li694-22
}
As you can see, we give each group a name, an alias (friendly name, mostly) & we define it's members. Pretty straight forward right? Let's move on to the fun part.
Now instead of defining a service per server, we simply define a service per hostgroup, meaning; when we add an Ubuntu server we simply add it to the 'ubuntu-servers' host group and it'll be automatically monitored for updates, and such.
The service definitions for hostgroups are in the same directory, let's take a look at the service definitions for the ubuntu-servers group:
/etc/icinga/objects/groups/ $ cat ubuntu.cfg
define service{
hostgroup_name ubuntu-servers ; Host group
use generic-service ; Name of service template to use
service_description APT Status ; Service check name.
check_command remote_one!check_apt ; This is the command that's being executed.
}
Let's say, all Ubuntu servers also run a special daemon (let's called it "speciald" - I haven't got a lot of inspiration right now), we would add the following (make sure the command definition/plugin exists) to 'ubuntu.cfg':
define service{
hostgroup_name ubuntu-servers ; Host group
use generic-service ; Name of service template to use
service_description Speciald Status ; Service check name.
check_command remote_one!check_speciald ; This is the command that's being executed.
}
and reload Icinga (after testing the configuration, first) and now, 'speciald' is being monitored on all Ubuntu servers.
Note, we also have 'all.cfg' which powers the "All Servers Group" - in which, you guessed it, checks for every server are defined, e.g. ssh, disk space, etc.
Testing Icinga's config and reloading:
When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:
# /usr/sbin/icinga -v /etc/icinga/icinga.cfg
If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:
# service icinga reload