Automatic Network Device Monitoring with PRTG


We’re big fans of PRTG at Lockstep, we’re also big fans of automating pretty much everything we can. Now, PRTG has built-in autodiscovery templates for switches and other network devices, but honestly, who wants to monitor every single port on every single switch in their infrastructure? Also, this usually just pulls RMON and snmp traffic statistics, and doesn’t give you operational status of the switch ports.

Luckily, PRTG offers an API, which means we can make sensors on the fly based on outside influences. In our case, we have a policy to label any important port on a switch. So, if it has a label, then we obviously want to monitor it. It only takes a few things to get started.

  • PowerShell – Pretty much all I use for Windows scripting now-a-days
  • Net-SNMP – A set of executables used to get snmp data into PowerShell
  • PrtgShell – A module I wrote to talk to PRTG from PowerShell
  • The Scripts – Two scripts, one for generating sensors, and one for returning data.
  • PRTG – well duh

The Setup

Download and install the Net-SNMP binaries, I used version 5.7. You just need the “Base Components” from the the install. The installer will automatically add the bin folder to your PATH variable so you can access the commands from wherever. It is important to note that you have to restart your PRTG probe service or Core server (depending on where the sensors will be run from) to update the PATH variable for custom sensors.

Download PrtgShell.psm1 and place it somewhere you can get at it. You can place it in your PSModulePath if you like, but since PRTG scripts don’t run as you, and in fact run on x86 PowerShell, I had some issues with that. Instead I just Import the script with the full path.

Download “SNMP Interface.ps1” and “SNMP Switch Index.ps1”. Place these in your PRTG install folder under “Custom Sensors\EXEXML” subfolder.

I’m going to assume you already have PowerShell and PRTG installed. If not, you can get PowerShell through Windows Update and PRTG here.

Stub Sensor

We’ll need to setup a new sensor for duplicating. You can put it anywhere honestly, but I usually put it under a special fake device I store all my stubs in. Wherever you decide it, you want to choose “EXE/Script Advanced” for the sensor type. Select the ‘SNMP Interface.ps1’ script, you’ll probably also want to choose “Use Windows credentials of parent device”, and “Write EXE result to disk” for troubleshooting. Those last 2 options will depend on your configuration. Take note of the object id on the stub sensor.

The Scripts

Before we go adding custom scripts your PRTG install, let’s go through them and see how they work. First, PrtgShell itself. PrtgShell contains 11 new Cmdlets, I’m only going to cover the ones we’re going to use.

  • Get-PrtgServer – create a global variable with the information to access your PrtgServer
  • Get-PrtgObjectProp – retrieve a property from a given PRTG device
  • Get-PrtgDeviceSensors – retrieve all sensors for a given PRTG device
  • Rename-PrtgSensor – rename a given PRTG sensor
  • Copy-PrtgSensor – copy a given PRTG sensor
  • Resume-PrtgSensor – resume a given PRTG sensor from a paused state
  • Set-PrtgObjectProperty – set a given property of a PRTG object
  • Remove-PrtgObject – delete a given PRTG object
  • Set-PrtgResult – Generate XML formatted correctly to return to PRTG

Wow, turns out I used almost all of the cmdlets in these. Next up, Net-SNMP. I only use two commands from the package for this.

  • snmpwalk – get results for a given mib for all interfaces
  • snmpget – get results for a given mib for a specific interface

Ok, now for the meat, the actual scripts. We have two scripts here. The first, SNMP Switch Index, runs through a network device and generates sensors on the same device based on port labels. The second, SNMP Interface, is the script used by the generated sensors to return data about each port. Here’s some details on how these work.

SNMP Switch Index

Parameters – DeviceIp, SNMPCommunity, SNMP Version, DeviceId, CloneId

First, we provide the information needed to connect to the PRTG API. Then kick off a Timer (which we’ll return later), import prtgshell, and add the System.Web Type. The System.Web is used for url-encoding for API calls


$PrtgServer = "prtgserver"
$PrtgUser = "prtguser"
$PrtgPass = "1111111111"

$Timer = get-date
Import-Module C:\_strap\prtgshell\prtgshell.psm1
Add-Type -AssemblyName System.Web

 

Now we’re going to walk through ifName and ifAlias to get the port names and aliases. $IfAliases is joined as a single string so we can use multi-line regex syntax on it later. The $Ports array is initialized, and two regexes are set to find data we need.


$IfNames = snmpwalk.exe -OqsU -r 3 -v $Version -c $Community $Agent "IF-MIB::ifName"
$IfAliases = snmpwalk.exe -OqsU -r 3 -v $Version -c $Community $Agent "IF-MIB::ifAlias"
$IfAliases = [string]::join("`n",$IfAliases)

$Ports = @()
$IndexRx = [regex] ‘ifName\.(\d+)’
$NameRx = [regex] ‘\.\d+\ (.+)’

 

Now we’re going to loop through $IfNames and gather the Index, Name, and Alias of each port into our array.


foreach ($i in $IfNames) {
$Index = $IndexRx.Match($i).Groups[1].Value

$AliasRx = [regex] “(?msx)ifAlias\.$Index\s([^`$]*? )`$”
$Alias = $AliasRx.Match($ifAliases).Groups[1].Value
$Name = $NameRx.Match($i).Groups[1].Value

$Port = “” | Select Index,Name,Alias
$Port.Index = $Index
$Port.Name = $Name
$Port.Alias = $Alias
$Ports += $Port
}

 

Now we’re going to grab some information. $TotalPorts is just a count of all ports, and $ActivePorts is anything with an alias. After that, we use prtgshell to connect to the server, get the IP of the device we’re after, and retrieve the current sensors on that device. Followed up with another regex we’ll use in a second to read the comments of a sensor, and some counters.


$TotalPorts = $Ports.Count
$ActivePorts = $Ports | where {$_.Alias}

$PrtgConnect = Get-PrtgServer $PrtgServer $PrtgUser $PrtgPass
$DeviceIp = Get-PrtgObjectProp $DeviceId host
$DeviceSensors = Get-PrtgDeviceSensors $DeviceId
$CommentRx = [regex] ‘interface\ =\ (\d+)’
$n = 0
$r = 0
$d = 0

 

Here comes the good part, we start by cycling through each labeled port and gathering some info. Then we check to see if that port already has a sensor, this is based on a special comment we’re about to add to each new sensor (originally I did this by sensor name alone, but ran into some inconsistent behavior). If we find that a sensor exists, we make sure there’s only one. If not, we log the double and move on. If there’s only one (which is what we desire), then we make sure it’s named correctly and update it if not. That way, if you re-label a port, the new label will be applied.

After that, we deal with brand new sensors, ports that haven’t already been added. First, we set the parameters for the custom sensor. Then we create the sensor by cloning our stub. Resume the new sensor since all cloned sensor are paused. Apply our parameters, and comment (interface = ). You can see we use the UrlEncode method here to account the spaces. Then, so long as everything went well, we move on.


foreach ($p in $ActivePorts) {
$PortName = $p.Name
$PortAlias = $p.Alias
$SensorName = "$Portname - $PortAlias"
$Exists = $DeviceSensors | Where { $_.comments -match $p.Index }

if ($Exists) {
“Exists: $SensorName”
if ($Exists.count -gt 1) {
“Double detected”
} elseif ($Exists.Sensor -ne $SensorName) {
$Rename = Rename-PrtgSensor $Exists.objid $SensorName
$r++
}
} else {
“adding $SensorName”
$ExeParams = “‘%host’ ‘%snmpcommunity’ ‘2c’ ‘$($p.Index)'”
$NewSensor = Copy-PrtgSensor $CloneSensor $SensorName $DeviceId
$Resume = Resume-PrtgObject $NewSensor
$SetParam = Set-PrtgObjectProperty $NewSensor exeparams $ExeParams
$Comment = [System.Web.HttpUtility]::UrlEncode(“interface = $($p.Index)”)
$SetNote = Set-PrtgObjectProperty $NewSensor comments $Comment
if (!($?)) { Throw “error” }
$n++
}
}

 

This section removes sensors for ports that are no longer labeled (and hopefully no longer important). We start by looping through the device sensors that we already gathered earlier. Narrow it down by looking for our special comment. Then look up the interface index number in the $ActivePorts array. If we can’t find it there, then we remove the sensor.


foreach ($s in $DeviceSensors) {
$Match = $CommentRx.Match($s.comments)
if ($Match.Success) {
$IfIndex = $Match.Groups[1].Value
$Lookup = $ActivePorts | ? {$_.index -eq $IfIndex}
if (!($Lookup)) {
"Removing $($s.objid): $($s.sensor)"
$Remove = Remove-PrtgObject $s.objid
$d++
}
}
}

 

All that by itself, will accomplish creating our sensors. But it would sure be nice to track all that stuff in PRTG wouldn’t it? First, we’ll close out our timer so we can return an execution time to PRTG. Start the XML with the proper tag. Then use Set-PrtgResult to create the result containers for any statistics we’ve collected along the way. Close that out with the appropriate tag, and then output that to the console for PRTG to collect and analyze.


$Elapsed = [math]::round(((Get-Date) - $Timer).TotalSeconds,2)
$XmlOutput = "`n"
$XmlOutput += Set-PrtgResult "Execution Time" $Elapsed secs -sc
$XmlOutput += Set-PrtgResult "Total Ports" $TotalPorts ports
$XmlOutput += Set-PrtgResult "Active Ports" $ActivePorts.Count ports
$XmlOutput += Set-PrtgResult "New Ports" $n ports
$XmlOutput += Set-PrtgResult "Renamed Ports" $r ports
$XmlOutput += Set-PrtgResult "Deleted Ports" $d ports
$XmlOutput += ""

$XmlOutput

 

If everything worked correctly, we’ll end up with something like this (Note, this after a first run has been complete and just a few changes have occurred).

Switch Index

SNMP Interface

Now this is the script that each sensor we just created runs. It gathers/reports data for each individual port

Parameters – DeviceIp, SNMPCommunity, SNMP Version, InterfaceIndex

We’ll start off simple, creating a timer and importing PrtgShell.


$Timer = Get-Date
Import-Module C:\_strap\prtgshell\prtgshell.psm1

 

Now we’re going to gather all the SNMP data we need. It’s all pretty standard with the exception of Octets. Since it’s a counter, it’s a big number, so we initialize it as a 64-bit integer.


$Admin = snmpget.exe -Ovq -r 3 -v $Version -c $Community $Agent "IF-MIB::ifAdminStatus.$Port"
$Oper = snmpget.exe -Ovq -r 3 -v $Version -c $Community $Agent "IF-MIB::ifOperStatus.$Port"
$IfInOctets = [int64](snmpget.exe -Ovq -r 3 -v $Version -c $Community $Agent "IF-MIB::ifInOctets.$Port")
$IfOutOctets = [int64](snmpget.exe -Ovq -r 3 -v $Version -c $Community $Agent "IF-MIB::ifOutOctets.$Port")
$IfInErrors = snmpget.exe -Ovq -r 3 -v $Version -c $Community $Agent "IF-MIB::ifInErrors.$Port"
$IfOutErrors = snmpget.exe -Ovq -r 3 -v $Version -c $Community $Agent "IF-MIB::ifOutErrors.$Port"
$IfInDiscards = snmpget.exe -Ovq -r 3 -v $Version -c $Community $Agent "IF-MIB::ifInDiscards.$Port"
$IfOutDiscards = snmpget.exe -Ovq -r 3 -v $Version -c $Community $Agent "IF-MIB::ifOutDiscards.$Port"

 

Next, we need to take the results of Admin and Oper and convert them to something we can use in PRTG.


if (($Oper -eq "dormant") -or ($Oper -eq "up")) { $Oper = 0 } `
elseif ($Oper -eq "down") { $Oper = 1 } `
elseif ($Oper -eq "notPresent") { $Oper = 2 } `
else { $Oper = 3 }

if ($Admin -eq “up”) { $Admin = 0 } `
else { $Admin = 1 }

 

Then we close out our timer, generate the XML, and send it on it’s way.


$Elapsed = [math]::round(((Get-Date) - $Timer).TotalMilliSeconds,2)

$XmlOutput = “`n”

$XmlOutput += Set-PrtgResult “Total Traffic” ($IfInOctets + $IfOutOctets) BytesBandwidth -ss KiloBit -mo Difference -sc -dm Auto

$XmlOutput += Set-PrtgResult “Inbound Traffic” $IfInOctets BytesBandwidth -ss KiloBit -mo Difference -sc -dm Auto
$XmlOutput += Set-PrtgResult “Outbound Traffic” $IfOutOctets BytesBandwidth -ss KiloBit -mo Difference -sc -dm Auto

$XmlOutput += Set-PrtgResult “Inbound Errors” $IfInErrors Count -mo Difference
$XmlOutput += Set-PrtgResult “Outbound Errors” $IfOutErrors Count -mo Difference
$XmlOutput += Set-PrtgResult “Inbound Discards” $IfInDiscards Count -mo Difference
$XmlOutput += Set-PrtgResult “Outbound Discards” $IfOutDiscards Count -mo Difference

$XmlOutput += Set-PrtgResult “Admin Status” $Admin admin -me 0 -em “Port is administratively down”
$XmlOutput += Set-PrtgResult “Operational Status” $Oper oper -me 0 -em “Operational: 1 = down, 2 = not present, 3 = other”

$XmlOutput += Set-PrtgResult “Execution Time” $Elapsed msecs
$XmlOutput += “”

$XmlOutput

 

If all goes according to plan, we’ll end up with this for each interface. The sensor will go into an error state if Operational or Admin status goes down. And the traffic stats will be shown in charts.

Interface

Now, to make this all automagic, we need to add a sensor to each device to run the ‘Switch Index.ps1’ script. Just like with our stub sensor, you’ll want to choose “EXE/Script Advanced” for the type. And you’ll probably want to use “Use Windows credentials of parent device” and “Write EXE result to disk”. Under Parameters, we need the following (replacing 9999 with the object id of your stub sensor). The single quotes are to make sure everything makes it’s way to PowerShell correctly.


'%host' '%snmpcommunity' '2c' '%deviceid' '9999'

That’s it. I set the Switch Index to run every 5 minutes on my core/distro switches and every hour or so on my edge switches. Not only does this force you to monitor important ports. But it also encourages you to label your important ports. Best practices all around!

From here, you could of course collect more or less data depending on your needs, just pay attention to your execution times vs your scan intervals. This should also give you a pretty good idea of how to use PrtgShell.