Being clever in collecting data
© 2017 Dennis Leeuw dleeuw at made-it dot com
License: GPLv2 or later
There are multiple ways to collect system information, but the common protocol supported almost everywhere is SNMP (Simple Network Management Protocol). Forget that simple, because it isn't. But what ever system you want to monitor, there is almost always an SNMP module to do it. And that is what this document is about, how to extend the standard NET-SNMP implementation with a particular "home-brew" extension. The extension choosen as the example is the S.M.A.R.T. harddisk data read with smartctl and connected to NET-SNMP as developed by https://www.mad-hacking.net/.
I assume you have installed this through the provided package manager. To make sure it works and our harddisk can provide SMART data run:
# smartctl -i /dev/sda smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-78-generic] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST500DM002-1BD142 Serial Number: W3T6WNNC LU WWN Device Id: 5 000c50 073ba7b32 Firmware Version: KC48 User Capacity: 500,107,862,016 bytes [500 GB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Jun 28 13:09:10 2017 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabledthe bold part is done by me, since that is the important information you need to look for. In this case our drive supports SMART and SMART is Enabled.
I assume you installed this through the package manager provided by the distribution you are using.
There is no standard that describes the MIB for colleting SMART data. mad-hacking.net provides its own privarte MIB file with its own OID, this is the way to extend SNMP if there is no standard.
Downloaded the following (or newer) packages from https://downloads.mad-hacking.net/software/:
On most distributions the MIB files needed by NET-SNMP are stored in /usr/share/snmp/mibs/. So to extend our system we are going to add the new MIBs from Mad Hacking to this directory:
mkdir tmp cd tmp tar jxvf hacking-snmp-mib-0.1.tar.bz2 cp usr/share/snmp/mibs/HACKING-SNMP-MIB.txt /usr/share/snmp/mibs/ tar jxvf smartctl-snmp-mib-0.3.tar.bz2 cp smartctl-snmp-mib-0.3/usr/share/snmp/mibs/SMARTCTL-MIB.txt /usr/share/snmp/mibs/To make sure that all is well, use:
snmptranslate -Tp SMARTCTL-MIB::smartCtlTableYou should get the output as shown at the SMARTCTL-MIB details page at the mad-hacking.net website. This page also describes the meaning of the values that are returned by SNMP and when you need to start to get worried about drive failures. So this might be the right time to get acquainted with that.
Finaly run:
# snmptranslate -Td SMARTCTL-MIB::smartCtlTable
SMARTCTL-MIB::smartCtlTable
smartCtlTable OBJECT-TYPE
-- FROM SMARTCTL-MIB
MAX-ACCESS not-accessible
STATUS current
DESCRIPTION "Table of HDD SMART information."
::= { iso(1) org(3) dod(6) internet(1) private(4) enterprises(1) hacking(38696) hackingExperimental(2) smartCtl(1) 2 }
which tells you that this MIB is designed to provide to you the OID: .1.3.6.1.4.1.38696.2.1, we need this later.
The software of mad-hacking.net consists of two parts. One part (update-smartctl-cache) collects the data from the harddisk and stores it for future use, and the other part (snmpd-smartctl-connector) provides NET-SNMP with the collected data. Both tools are part of the snmpd-smartctl-connector-X.Y.X.tar.bz2 package. This package depends on some other packages. Below is a complete installation overview. Everything will be installed in /opt/mad-hacking.net/snmp/ so you will have a single place that can be cleaned when needed.
Download the following (or newer) packages from https://downloads.mad-hacking.net/software/:
# Our destination tree
mkdir -p /opt/mad-hacking.net/snmp/{sbin,etc,lib,var/lib/smartctl}
# Some temp dir for the sources
mkdir tmp
cd tmp
# Some basic functions
tar jxvf hacking-bash-lib-0.2.tar.bz2
cp hacking-bash-lib-0.2/usr/lib/hacking-bash.sh /opt/mad-hacking.net/snmp/lib
# Functions for the connector
tar jxvf snmpd-connector-lib-0.3.tar.bz2
cp usr/lib/snmpd-connector-lib.sh /opt/mad-hacking.net/snmp/lib
sed -i 's#/usr/lib/hacking-bash.sh#/opt/mad-hacking.net/snmp/lib/hacking-bash.sh#' /opt/mad-hacking.net/snmp/lib/snmpd-connector-lib.sh
# The actual connector
tar jxvf snmpd-smartctl-connector-0.10.1.tar.bz2
cp usr/sbin/* /opt/mad-hacking.net/snmp/sbin/
sed -i 's#/usr/lib/snmpd-connector-lib.sh#/opt/mad-hacking.net/snmp/lib/snmpd-connector-lib.sh#' /opt/mad-hacking.net/snmp/sbin/*
sed -i 's#/etc/snmpd-smartctl-connector#/opt/mad-hacking.net/snmp/etc/snmpd-smartctl-connector#' /opt/mad-hacking.net/snmp/sbin/*
cp etc/snmpd-smartctl-connector /opt/mad-hacking.net/snmp/etc/
sed -i 's#/tmp/snmp-cache/smartctl#/opt/mad-hacking.net/snmp/var/lib/smartctl#' /opt/mad-hacking.net/snmp/etc/snmpd-smartctl-connector
To fill the initial database run as root:
/opt/mad-hacking.net/snmp/sbin/update-smartctl-cacheOf course you want to regularly check the SMART data of the drive and thus update the database. This is easiest done by using cron. The update-smartctl-cache has two running modes. Normal operation only updates the health and attribute files, with -i the command starts from scratch and updates the entire cache (which also happens when the dev file is newer then the database files. So for normal updates you can copy or link the update-smartctl-cache to e.g. /etc/cron.hourly, for hourly updates of the drive statistics.
This process should run as root, or atleast as a user that is able to read the device files (on my machine root and everybody that is part of the disk group). Later on the data should be readable by the snmpd process. snmpd on my machine runs under the user snmp, so that user should have read access.
In our initial test of the smartmontools we used /dev/sda as our testing drive. The mad-hacking.net tools configuration file is per default set to watch for sd[a-z] devices. See the /opt/mad-hacking.net/snmp/etc/snmpd-smartctl-connector configuration file if you need to change this.
Run the following command and provide at the following commands exactly as is. The return values of the system are in bold:
/opt/mad-hacking.net/snmp/sbin/snmpd-smartctl-connector ping PONG get .1.3.6.1.4.1.38696.2.1.2.1.2.1 .1.3.6.1.4.1.38696.2.1.2.1.2.1 string /dev/sda get .1.3.6.1.4.1.38696.2.1.2.1.3.1 .1.3.6.1.4.1.38696.2.1.2.1.3.1 string Seagate Barracuda 7200.14 (AF) quit Bye
The PONG should always work. If no device is returned, make sure the configuration is correct in the above mentioned configuration file, and make sure the snmpd-smartctl-connector can access the device file. If oid .1.3.6.1.4.1.38696.2.1.2.1.2.1 returns valid data, but the drive information is not returned make sure you smartcltl database from the previous section is created and readable by the snmpd-smartctl-connector tool.
Add to /etc/snmp/snmpd.conf the following line somewhere in the "Pass-through" MIB extension command section (note the OID we wrote down in an earlier section?):
pass_persist .1.3.6.1.4.1.38696.2.1 /bin/bash /opt/mad-hacking.net/snmp/sbin/snmpd-smartctl-connectorAfter the change restart snmpd.
To test our setup we need to ask snmpd to return to us the SMART data of our system. A simple test like the following should confirm everything is working as we planned:
# snmpwalk -c public -v 1 localhost SMARTCTL-MIB::smartCtlTable SMARTCTL-MIB::smartCtlDeviceIndex.1 = INTEGER: 1 SMARTCTL-MIB::smartCtlDeviceDev.1 = STRING: /dev/sda SMARTCTL-MIB::smartCtlDeviceModelFamily.1 = STRING: Seagate Barracuda 7200.14 (AF) SMARTCTL-MIB::smartCtlDeviceDeviceModel.1 = STRING: ST500DM002-1BD142 SMARTCTL-MIB::smartCtlDeviceSerialNumber.1 = STRING: W3T6WNNC SMARTCTL-MIB::smartCtlDeviceUserCapacity.1 = STRING: 500,107,862,016 bytes [500 GB] SMARTCTL-MIB::smartCtlDeviceATAVersion.1 = STRING: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) SMARTCTL-MIB::smartCtlDeviceHealthOK.1 = INTEGER: true(1) SMARTCTL-MIB::smartCtlDeviceTemperatureCelsius.1 = Gauge32: 37 SMARTCTL-MIB::smartCtlDeviceReallocatedSectorCt.1 = Gauge32: 0 SMARTCTL-MIB::smartCtlDeviceCurrentPendingSector.1 = Gauge32: 0 SMARTCTL-MIB::smartCtlDeviceOfflineUncorrectable.1 = Gauge32: 0 SMARTCTL-MIB::smartCtlDeviceUDMACRCErrorCount.1 = Gauge32: 0 SMARTCTL-MIB::smartCtlDeviceReadErrorRate.1 = Gauge32: 4294967295 SMARTCTL-MIB::smartCtlDeviceSeekErrorRate.1 = Gauge32: 52 SMARTCTL-MIB::smartCtlDeviceHardwareECCRecovered.1 = Gauge32: 4294967295 SMARTCTL-MIB::smartCtlDeviceFirmwareVersion.1 = STRING: KC48