Difference between revisions of "SystemAdministration/Icinga"

From SoylentNews
Jump to: navigation, search
(Adding a server)
(objects/groups)
Line 85: Line 85:
 
{{#lst:SystemAdministration/Icinga|testandreload}}
 
{{#lst:SystemAdministration/Icinga|testandreload}}
  
== objects/groups ==
+
== Host groups ==
  
Firstly, we've got objects named "host groups" which are simple groups with several servers, for instance all Ubuntu servers belong to the 'ubuntu-servers' group and all servers which have a PostgreSQL service running belong to the 'postgresql-servers' group. One host/server can belong to many groups. All these groups are defined in '''/etc/icinga/objects/groups/hostgroups_icinga.cfg''' the members of these groups are also defined here.
+
[http://docs.icinga.org/latest/en/objectdefinitions.html#objectdefinitions-hostgroup Icinga's documentation]
  
Let's take the ubuntu-servers & postgresql-servers (respectively) as examples:
+
=== Introduction ===
  
<code>/etc/icinga/objects/groups/ $ '''cat hostgroups_icinga.cfg'''</code> [note: only copying 'ubuntu-servers' and 'postgresql-servers' definitions out of file]</code>
+
In Icinga we can group hosts, for instance "Ubuntu Servers" and "PostgreSQL servers" and so on. All of our servers are members of the 'all' host group, so instead of defining services checks for memory & harddisk usage (to name a few) per server, we simply define that check in the 'all' host group and effectively, all of our servers are now being monitored for memory & harddisk usage (and a lot more, check '''/etc/icinga/objects/groups/all.cfg''').
  
'''ubuntu-servers'''
+
One server can be a member of many host groups and we've got quite a few. We've got the "ubuntu-servers" group, the "postgresql-servers" group and a few more.
 +
 
 +
=== Define a hostgroup ===
 +
 
 +
We define host groups in '''/etc/icinga/objects/groups/hostgroup_icinga.cfg''', these look a lot like (comments in line):
  
 
<pre>
 
<pre>
 +
; Start with the define statement.
 
define hostgroup {
 
define hostgroup {
         hostgroup_name  ubuntu-servers
+
         hostgroup_name  ubuntu-servers ; Let's give the group a name.
alias          Ubuntu Servers
+
alias          Ubuntu Servers ; Let's give it a "more" friendly name.
members        boron.li694-22, helium.li694-22, carbon.li694-22, lithium.li694-22, nitrogen.li694-22, oxygen.li694-22, hydrogen.li694-22
+
members        boron.li694-22, helium.li694-22, carbon.li694-22, lithium.li694-22, nitrogen.li694-22, oxygen.li694-22, hydrogen.li694-22 ; It's members.
 
         }
 
         }
</pre>
 
  
'''postgresql-servers'''
 
 
<pre>
 
 
define hostgroup {
 
define hostgroup {
         hostgroup_name  postgresql-servers
+
         hostgroup_name  postgresql-servers ; Let's give the group a name.
                 alias          PostgreSQL servers
+
                 alias          PostgreSQL servers ; Let's give it a "more" friendly name.
                 members        carbon.li694-22, boron.li694-22
+
                 members        carbon.li694-22, boron.li694-22 ; It's members.
 
         }
 
         }
 
</pre>
 
</pre>
  
As you can see, we give each group a name, an alias (friendly name, mostly) & we define it's members. Pretty straight forward right? Let's move on to the fun part.
+
As you can no doubt see, the definitions are relatively straight forward. Let's say, we install PostgreSQL on our server by the name of "helium"; then we'd simply add helium.li694-22 as a member of the postgresql-servers group and voila, now helium's PostgreSQL is also monitored. (note: monitoring PostgreSQL in itself involves a few more steps which are omitted to keep the explanation simple).
  
Now instead of defining a service per server, we simply define a service per hostgroup, meaning; when we add an Ubuntu server we simply add it to the 'ubuntu-servers' host group and it'll be automatically monitored for updates, and such.
+
=== Removing a host group ===
  
The service definitions for hostgroups are in the same directory, let's take a look at the service definitions for the ubuntu-servers group:
+
Simply remove the definition from '''/etc/icinga/objects/groups/hostgroup_icinga.cfg'''
  
<code>/etc/icinga/objects/groups/ $ '''cat ubuntu.cfg'''</code>
+
=== Adding a new host group ===
<pre>
 
define service{
 
hostgroup_name ubuntu-servers          ; Host group
 
        use                            generic-service        ; Name of service template to use
 
        service_description            APT Status              ; Service check name.
 
        check_command                  remote_one!check_apt    ; This is the command that's being executed.
 
        }
 
</pre>
 
  
Let's say, all Ubuntu servers also run a special daemon (let's called it "speciald" - I haven't got a lot of inspiration right now), we would add the following (make sure the command definition/plugin exists) to 'ubuntu.cfg':
+
Adding a new host group is simple, you copy one of the "hostgroup" definitions to the end of '''/etc/icinga/objects/groups/hostgroup_icinga.cfg''' and change the group's name, alias & you define which servers are a member. So let's say, you want to monitor webservers, you would add the following definition (note, we're already monitoring those, it's just an example):
  
 
<pre>
 
<pre>
define service{
+
# A list of our web servers
hostgroup_name ubuntu-servers         ; Host group
+
define hostgroup {
         use                            generic-service        ; Name of service template to use
+
        hostgroup_name http-servers
        service_description            Speciald Status        ; Service check name.
+
                alias          HTTP servers
        check_command                  remote_one!check_speciald ; This is the command that's being executed.
+
                members         hydrogen.li694-22, carbon.li694-22, boron.li694-22, beryllium.li694-22
 
         }
 
         }
 
</pre>
 
</pre>
  
and reload Icinga (after testing the configuration, first) and now, 'speciald' is being monitored on all Ubuntu servers.
+
Now, this alone, won't work, currently we've only defined the host group but we haven't defined any services, let's do that in the next chapter.
 
 
Note, we also have 'all.cfg' which powers the "All Servers Group" - in which, you guessed it, checks for every server are defined, e.g. ssh, disk space, etc.
 
{{#lst:SystemAdministration/Icinga|testandreload}}
 

Revision as of 23:18, 13 April 2014

Sentinel: Icinga / Server & Services Monitoring

Important Directories

Directory Description
/etc/icinga/objects/groups (service definitions for) Host Groups
/etc/icinga/objects/servers Server Definitions
/etc/icinga/objects/templates Various templates docs here
/etc/icinga/objects/contacts Contact & time period config docs here
/etc/nagios-plugins/config/ Plugin configs; be sure to checkout 'remote.cfg'
/usr/lib/nagios/plugins/ Plugins

Useful links

Icinga documentation

Testing Icinga's config and reloading:

When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:

# /usr/sbin/icinga -v /etc/icinga/icinga.cfg

If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:

# service icinga reload


Access

Our Icinga installation is named 'Sentinel' which means "a soldier or guard whose job is to stand and keep watch", you can find Sentinel here and login with your Keberos username and password.

Kerberized SSH connections

We use AutoSSH to allow Icinga to connect to other nodes to execute commands and get services information back.

Internally, we can use Kerberos to jump from one host to another our Icinga installation requires HTTP Kerberos auth (via https) but we've also set up AutoSSH which allows our Icinga instance to connect to other nodes passwordless & ssh key-less to fetch information.

more information, show how we've setup the connections, etc

icinga.cfg

The file /etc/icinga/icinga.cfg is heavily commented. I suggest you simply look it over.

Servers

Icinga's "host" documentation

Adding a server

Each server has a file in /etc/icinga/objects/servers/ which looks something like:

define host{
        use                     generic-host            ; Name of host template to use
        host_name               carbon.li694-22         ; The server hostname (ensure it's in our DNS).
        alias                   carbon                  ; A shortname
        address                 carbon.li694-22         ; The server's address.
}

Creating a new one (for instance, 'example.li694-22') should be straight forward, you simply copy carbon.li694-22 to example.li64-22:

/etc/icinga/objects/servers/ $ cp carbon.cfg example.cfg

Once that's done, you can either edit the file manually with the editor of your choice or use sed:

/etc/icinga/objects/servers/ $ sed -i 's/carbon/example/g' example.cfg

Modifying a server

Simply edit the server object in /etc/icinga/objects/servers/

Removing a server

If you want to remove a server, remove the config and check /etc/icinga/objects/groups/hostgroups_icinga.cfg and remove any references to the server.

Testing Icinga's config and reloading:

When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:

# /usr/sbin/icinga -v /etc/icinga/icinga.cfg

If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:

# service icinga reload


Host groups

Icinga's documentation

Introduction

In Icinga we can group hosts, for instance "Ubuntu Servers" and "PostgreSQL servers" and so on. All of our servers are members of the 'all' host group, so instead of defining services checks for memory & harddisk usage (to name a few) per server, we simply define that check in the 'all' host group and effectively, all of our servers are now being monitored for memory & harddisk usage (and a lot more, check /etc/icinga/objects/groups/all.cfg).

One server can be a member of many host groups and we've got quite a few. We've got the "ubuntu-servers" group, the "postgresql-servers" group and a few more.

Define a hostgroup

We define host groups in /etc/icinga/objects/groups/hostgroup_icinga.cfg, these look a lot like (comments in line):

; Start with the define statement.
define hostgroup {
        hostgroup_name  ubuntu-servers ; Let's give the group a name.
		alias           Ubuntu Servers ; Let's give it a "more" friendly name.
		members         boron.li694-22, helium.li694-22, carbon.li694-22, lithium.li694-22, nitrogen.li694-22, oxygen.li694-22, hydrogen.li694-22 ; It's members.
        }

define hostgroup {
        hostgroup_name  postgresql-servers ; Let's give the group a name.
                alias           PostgreSQL servers ; Let's give it a "more" friendly name.
                members         carbon.li694-22, boron.li694-22 ; It's members.
        }

As you can no doubt see, the definitions are relatively straight forward. Let's say, we install PostgreSQL on our server by the name of "helium"; then we'd simply add helium.li694-22 as a member of the postgresql-servers group and voila, now helium's PostgreSQL is also monitored. (note: monitoring PostgreSQL in itself involves a few more steps which are omitted to keep the explanation simple).

Removing a host group

Simply remove the definition from /etc/icinga/objects/groups/hostgroup_icinga.cfg

Adding a new host group

Adding a new host group is simple, you copy one of the "hostgroup" definitions to the end of /etc/icinga/objects/groups/hostgroup_icinga.cfg and change the group's name, alias & you define which servers are a member. So let's say, you want to monitor webservers, you would add the following definition (note, we're already monitoring those, it's just an example):

# A list of our web servers
define hostgroup {
        hostgroup_name  http-servers
                alias           HTTP servers
                members         hydrogen.li694-22, carbon.li694-22, boron.li694-22, beryllium.li694-22
        }

Now, this alone, won't work, currently we've only defined the host group but we haven't defined any services, let's do that in the next chapter.