Information Security and Network Awareness

Hurricane Labs

Subscribe to Hurricane Labs: eMailAlertsEmail Alerts
Get Hurricane Labs: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Related Topics: PC Security Journal, Cisco Virtualization Journal, Security Journal, Haiti EarthQuake

Blog Feed Post

Icinga and Agentless Monitoring

There are four ways to get data from a host to Icinga. The first is to use an agent, something like NRPE on *nix systems or NSClient++ on Windows based systems. The second way is to pull the data from “publicly” available ports; this can be done by verifying that your website is up or checking to see if a specific port is open. The third way to get host data into Icinga is to use ssh. The Icinga master would connect to the host over ssh and run any service checks. Beware! This method is very resource heavy and not all that practical for large scale monitoring. Each check creates a separate ssh session which can bog down the system. The final method of transferring data is through SNMP. It is the most efficient in terms of non-publicly available service data. Icinga/Nagios does a great job of monitoring using a NRPE agent. However, sometimes that option is not available. You may not have root level access to install NRPE or the os may not support the agent. On Cisco gear in particular, there is no way to install NRPE. This is where SNMP comes into play. SNMP is great for monitoring because it uses a standard network service rather then relying on an agent. I am going to walk through how to setup a service check through SNMP. This post assumes that you understand SNMP and have it up and working in your environment.

The check we are going to look at will return memory from a Cisco Switch. We will then make sure that there is sufficient free memory, or it will go into a “warning” or “critical” state. The check itself is written in bash because I like bash. You can write the same thing in python or perl but your milage may vary. The basic steps are:

1. Write a script that checks over SNMP
2. Define a command to run that script
3. Put that command into a service template
4. Apply that template to a host

1. The check script

Create a file called check_cisco_memory
A she bang and a few comments to get started:

# !/bin/bash
# check_cisco_memory
# Description : Check the free memory of a Cisco device  
# Version : 2.0
# Author : Dru Streicher
# Licence : GPLv2
# A lot of this code is based off a memory check by Yoann Lamy

We are going to set some global environmental priorities here:

# Commands
CMD_BASENAME="/usr/bin/basename"
CMD_SNMPGET="/usr/bin/snmpget"
CMD_SNMPWALK="/usr/bin/snmpwalk"
CMD_EXPR="/usr/bin/expr"
# Script name
SCRIPTNAME=`$CMD_BASENAME $0`
# Version
VERSION="2.0"

Nagios only returns 4 different status messages. We declare all four here:

# Plugin return codes
STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3

These are the OIDs that we are going to use. This will return the used memory and the free memory:

# OID's
OID_USED_MEMORY="1.3.6.1.4.1.9.9.48.1.1.1.5.1"
OID_FREE_MEMORY="1.3.6.1.4.1.9.9.48.1.1.1.6.1"

Then we set the default state. Always set it to unknown that way if there are any issues your check will come back as unknown:

# Default variables
DESCRIPTION="Unknown"
STATE=$STATE_UNKNOWN

Then some default options and arguments:

# Default options
COMMUNITY="public"
HOSTNAME="127.0.0.1"
TYPE="firmware"
DISK=1
WARNING=0
CRITICAL=0

We then will have to set up the arguments:

# Option processing
print_usage() {
  echo "Usage: ./check_cisco_memory -H 127.0.0.1 -C public -w 10 -c 5"
  echo "  $SCRIPTNAME -H ADDRESS"
  echo "  $SCRIPTNAME -C STRING"
  echo "  $SCRIPTNAME -w INTEGER"
  echo "  $SCRIPTNAME -c INTEGER"
  echo "  $SCRIPTNAME -p INTEGER"
  echo "  $SCRIPTNAME -h"
  echo "  $SCRIPTNAME -V"
}

As well as some help documentation so that in 6 months you will know what everything does:

print_version() {
  echo $SCRIPTNAME version $VERSION
  echo ""
  echo "This nagios plugin comes with ABSOLUTELY NO WARRANTY."
  echo "You may redistribute copies of this plugin under the terms of the GNU General Public License v2."
}
print_help() {
  print_version
  echo ""
  print_usage
  echo ""
  echo "Check the memory of a Cisco device "
  echo ""
  echo "-H ADDRESS"
  echo "   Name or IP address of host (default: 127.0.0.1)"
  echo "-C STRING"
  echo "   Community name for the host's SNMP agent (default: public)"
  echo "-w INTEGER"
  echo "   Warning level"
  echo "-c INTEGER"
  echo "   Critical level"
  echo "-h"
  echo "   Print this help screen"
  echo "-V"
  echo "   Print version and license information"
  echo ""
  echo ""
  echo "This plugin uses the 'snmpget' command and the 'snmpwalk' command included with the NET-SNMP package."
  echo "This plugin support performance data output."
  echo "If the percentage of the warning level and the critical level are set to 0, then the script returns an OK state."
}

Take the input and turn it to variables:

while getopts H:C:t:d:w:c:p:hV OPT
do
  case $OPT in
    H) HOSTNAME="$OPTARG" ;;
    C) COMMUNITY="$OPTARG" ;;
    w) WARNING="$OPTARG" ;;
    c) CRITICAL="$OPTARG" ;;
    p) PORT="$OPTARG" ;;
    h)
      print_help
      exit $STATE_UNKNOWN
      ;;
    V)
      print_version
      exit $STATE_UNKNOWN
      ;;
   esac
done

Now it is time for the heavy lifting. We use the snmpget function and the OID’s to get the values for the used and free memory:

# Plugin processing
  USED=`$CMD_SNMPGET -t 2 -r 2 -v 1 -c $COMMUNITY -Ovq $HOSTNAME "${OID_USED_MEMORY}"`
  FREE=`$CMD_SNMPGET -t 2 -r 2 -v 1 -c $COMMUNITY -Ovq $HOSTNAME "${OID_FREE_MEMORY}"`

Then we use bc to preform some math and get the total memory available:

TOTAL_MEM=$(echo "$USED + $FREE" | bc)

Some more math via bc to get the percentage free:

PERCENT_FREE=$(echo "scale=2; $FREE / $TOTAL_MEM * 100" | bc | sed s/\\.[0-9]\\+//)

A nice if loop to set the state to either ok, warning, critical, or unknown”.

if [ "$PERCENT_FREE" -lt "$WARNING" ]; then
    STATE=$STATE_OK;
    STATE_DESC="OK"
  elif [ "$PERCENT_FREE" -gt "$WARNING" ] && [ "$PERCENT_FREE" -lt "$CRITICAL" ]; then
    STATE=$STATE_WARNING;
    STATE_DESC="WARNING"
  elif [ "$PERCENT_FREE" -gt "$WARNING" ] && [ "$PERCENT_FREE" -gt "$CRITICAL" ]; then
    STATE=$STATE_CRITICAL;
    STATE_DESC="CRITICAL"
  else
   STATE=$STATE_UNKNOWN;
    STATE_DESC="UNKNOWN"
  fi

Along with the status we can return some information with our check, in this case we return the percentage free as text.

DESCRIPTION="Memory $PERCENT_FREE% Free";

Then return the state and the description and exit:

echo "$STATE_DESC : $DESCRIPTION"
exit $STATE

if we manually run our check we will get this output:

./check_cisco_asa_memory -C sw0rdf!sh -H 198.162.1.1 -w 10 -c 5 
OK : Memory 84% Free

Great now we know our check is working and the host has plenty of memory. Now we need to make this into a service check in Icinga.

2. Defining the check command:

The purpose of the checkcommands.cfg is to allow the check scripts we write to be used over and over. The checkcommand.cfg also allows us to pass variables. In our case we want to pass the warning and critical variables. So we need to define a command called check_cisco_memory and pass some variable like this:
Create a file called checkcommands.cfg

## 'check_cisco_memory'
        command_name    check_cisco_memory
        command_line    $USER1$/check_cisco_memory -C $_HOSTSNMP_COMMUNITY$ -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$

By using the $_HOSTSNMP_COMMUNITY we can store the community string somewhere else. The $ARG1$ and $ARG2$ will be variables that we can pass on later by the hostgroup template or on a per host basis.

3. Putting the checkcommand into a hostgroup template:

Say we want all of our ASA firewalls to use this check. We would do something like this…
Create a file called cisco-asa.cfg

###--- HOST GROUP DEFINITION
define hostgroup {
	hostgroup_name		template_cisco-asa
	alias			Cisco ASA Firewall
}

###--- SERVICES

define service {
	use			generic-service
	name			cisco-asa-memory
	hostgroup_name		template_cisco-asa
	service_description	free_memory
	display_name		FREE MEMORY
	check_command		check_cisco_memory!10!5
}

The line:

check_command	check_cisco_memory!10!5

is where all the magic happens. The ! passes the variable to the checkcommand so in our case the !10!5 will take place of the $ARG1$ and $ARG2$ on this line of the checkcommand.cfg

command_line    $USER1$/check_cisco_asa_memory -C $_HOSTSNMP_COMMUNITY$ -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$

4. Applying the hostgroup to the host

Now that we have everything setup, we need to actually put the service check onto the host. Before we can do that, we’ll need to do two things: set the community string and add the service check itself.

Create a host.cfg file

###--- HOST DEFINITION
define host {
        use             generic-host
        host_name       fireball
        hostgroups      +template_cisco-asa
        alias           Cisco ASA 
        address         198.162.1.1
	_snmp_community	sw0rdf!sh
}

Now the snmp community string will be passed to the checkcommand very much like the $ARG1$ was passed.

Reload Icinga and behold! There is your shiny new memory check, all done without an agent.

Access the full code example here.

The post Icinga and Agentless Monitoring appeared first on Hurricane Labs.

Read the original blog entry...

More Stories By Hurricane Labs

Christina O’Neill has been working in the information security field for 3 years. She is a board member for the Northern Ohio InfraGard Members Alliance and a committee member for the Information Security Summit, a conference held once a year for information security and physical security professionals.