Applicable Products

NetScaler

Objective

This article describes how to create a Perl based Custom Monitor on NetScaler.

Background

The NetScaler appliance has a lot of different monitors inbuilt, but there are use cases these monitors do not cover. For this NetScaler supports monitors of type USER, which brings the possibility to run external Perl scripts to track the health of a custom application or server. This article shows the steps you need to do before successfully running a custom monitor.

For an overview about Custom User Monitors refer to Citrix Documentation - https://docs.citrix.com/ko-kr/netscaler/11-1/load-balancing/load-balancing-custom-monitors/understand-user-monitors.html

Instructions

Prerequisites

Log on on NetScaler via SSH and go into Shell.
A common problem is that the Perl interpreter does not recognize KAS.pm module. To solve this, we create a symbolic link to point on the located KAS.pm.
1. mkdir /usr/local/lib/perl5/site_perl/5.14.2/mach/Netscaler
2. ln -s /netscaler/monitors/perl_mod/Netscaler/KAS.pm /usr/local/lib/perl5/site_perl/5.14.2/mach/Netscaler/KAS.pm
To make changes reboot persistent, we create a file /nsconfig/rc.netscaler (if it does not already exists) and insert commands used previously:
1. touch /nsconfig/rc.netscaler
2. chmod a+x rc.netscaler
3. echo "mkdir /usr/local/lib/perl5/site_perl/5.14.2/mach/Netscaler" >> /nsconfig/rc.netscaler
4. echo "ln -s /netscaler/monitors/perl_mod/Netscaler/KAS.pm /usr/local/lib/perl5/site_perl/5.14.2/mach/Netscaler/KAS.pm" >> /nsconfig/rc.netscaler

Perl Script

This is a simple script example showing the requirements. Using the NetScaler KAS module and strict pragma are mandatory, other modules/libraries are optional.

We also need a sub doing the data processing with a response code of 0 (probe successful) or 1 (probe failed). Finally we need to register the sub to KAS.

#!/usr/bin/perl -w
################################################################
##
################################################################
 
use Netscaler::KAS;
use strict;
 
sub soap_probe {
 ## init variable with argument
 my $searchString = $ARGV[0];
 
 ## send request and collect response here
 my $response = "value";
 
 ## check response
 if (index($response, $searchString) != -1) {
  return(0);
 } else {
  return (1,"String not found");
 }
 
## register prob sub to the KAS module
probe(\&soap_probe);

Add Custom Monitor to NetScaler

Dispatcher IP and port must remain at 127.0.0.1:3013 for internal communication. Optional is the parameter "-scriptargs" which allows us to submit parameters like the backend server IP or any search pattern for the given response. In our Perl script we can select these parameters as typical command line arguments. The delimiter between multiple arguments is ";".

From NetScaler GUI

Add new monitor of type User under Traffic Management > Load Balancing > Monitors.
Set the Interval the script should run and the Response Time-out. This is the amount of time the script waits for a response before it gives up and marks the probe as failed.
Go into Special Parameters tab, upload and select the Perl script and define optional script arguments if required.

From NetScaler CLI

Upload the script with an SCP tool to /nsconfig/monitor/ directory.
Add the probe with the following command:
add lb monitor lbm-custom USER -scriptName custom-probe.pl -dispatcherIP 127.0.0.1 -dispatcherPort 3013 -resptimeout 3

Debugging

To debug the script, you must run it by using the nsumon-debug.pl script, located in /netscaler/monitors/.

To the run this debug script you must enter the following arguments:
nsumon-debug.pl <scriptname> <IP> <port> <timeout> <partitionID> [scriptarguments] [is_secure]

Another possibility is to run the script with the Perl interpreter to check any errors:
root@cns1# perl custom-probe.pl test
1,String not found

This example shows a failed probe because the search string was not found in the response.

===============================

Understanding User Monitors

https://docs.citrix.com/ko-kr/netscaler/11-1/load-balancing/load-balancing-custom-monitors/understand-user-monitors.html

Aug 31, 2016

User monitors extend the scope of custom monitors. You can create user monitors to track the health of customized applications and protocols that the NetScaler appliance does not support. The following diagram illustrates how a user monitor works.

Figure 1. User Monitors

A user monitor requires the following components.

Dispatcher. A process, on the appliance, that listens to monitoring requests. A dispatcher can be on the loopback IP address (127.0.0.1) and port 3013. Dispatchers are also known as internal dispatchers. A dispatcher can also be a web server that supports Common Gateway Interface (CGI). Such dispatchers are also known as external dispatchers. They are used for custom scripts that do not run on the FreeBSD environment, such as .NET scripts.

Note: You can configure the monitor and the dispatcher to use HTTPS instead of HTTP by enabling the “secure” option on the monitor and configure it as an external dispatcher. However, an internal dispatcher understands only HTTP, and cannot use HTTPS.

In a HA setup, the dispatcher runs on both the primary and secondary NetScaler appliances. The dispatcher remains inactive on the secondary appliance.
Script. The script is a program that sends custom probes to the load balanced server and returns the response code to the dispatcher. The script can return any value to the dispatcher, but if a probe succeeds, the script must return a value of zero (0). The dispatcher considers any other value as probe failure.
The NetScaler appliance is bundled with sample scripts for commonly used protocols. The scripts exist in the /nsconfig/monitors directory. If you want to add a new script, add it there. If you want to customize an existing script, create a copy with a new name and modify it.
Important: Starting with release 10.1 build 122.17, the script files for user monitors are in a new location.
If you upgrade an MPX or VPX virtual appliance to release 10.1 build 122.17 or later, the changes are as follows:
- /nsconfig/monitors/ and all the built-in scripts of the previous builds are moved to this directory.
- /nsconfig/monitors/ directory.
- /nsconfig/monitors/ directory.
- /netscaler/monitors/ directory takes priority. That is, the custom script does not run.
If you provision a virtual appliance with release 10.1 build 122.17 or later, the changes are as follows:
- /netscaler/monitors/ directory.
- /nsconfig/monitors/ directory is empty.
- /nsconfig/monitors/ directory.
For the scripts to function correctly, the name of the script file must not exceed 63 characters, and the maximum number of script arguments is 512. To debug the script, you must run it by using the nsumon-debug.pl script from the NetScaler command line. You use the script name (with its arguments), IP address, and the port as the arguments of the nsumon-debug.pl script. Users must use the script name, IP address, port, time-out, and the script arguments for the nsumon-debug.pl script.
Important: Starting with release 10.5 build 57.x, and 11.0 script files for user monitors support IPv6 addresses and include the following changes:
- For the following protocols, new pm files have been included for IPv6 support.
  - Radius
  - NNTP
  - POP3
  - SMTP
- /netscaler/monitors/ has been updated for IPv6 support:
  - nsbmradius.pl
  - nsldap.pl
  - nsnntp.pl
  - nspop3 nssf.pl
  - nssnmp.pl
  - nswi.pl
  - nstftp.pl
  - nssmtp.pl
  - nsrdp.pl
  - nsntlm-lwp.pl
  - nsftp.pl
  - nsappc.pl
  After upgrading to release 10.5 build 57.x, or 11.0, if you want to use your existing custom scripts with IPv6 services, make sure that you update the existing custom scripts with the changes provided in the updated sample scripts in /netscaler/monitors/.
  Note: The sample script nsmysql.pl does not support IPv6 address. If an IPv6 service is bound to a user monitor that uses nsmysql.pl, the probe will fail.
- The following LB monitor types have been updated to support IPv6 addresses:
  - USER
  - SMTP
  - NNTP
  - LDAP
  - SNMP
  - POP3
  - FTP_EXTENDED
  - STOREFRONT
  - APPC
  - CITRIX_WI_EXTENDED
  If you are creating a new custom script that uses one of these LB monitors types, make sure that you include IPv6 support in the custom script. Refer to the associated sample script in/netscaler/monitors/ for the changes that you have to make in the custom script for IPv6 support.

To track the status of the server, the monitor sends an HTTP POST request to the configured dispatcher. This POST request contains the IP address and port of the server, and the script that must be executed. The dispatcher executes the script as a child process, with user-defined parameters (if any). Then, the script sends a probe to the server. The script sends the status of the probe (response code) to the dispatcher. The dispatcher converts the response code to an HTTP response and sends it to the monitor. Based on the HTTP response, the monitor marks the service as up or down.

The appliance logs the error messages to the /var/nslog/nsumond.log file when user monitor probes fail. The following table lists the user monitors and the possible reasons for failure.

User monitor type	Probe failure reasons
SMTP	Monitor fails to establish a connection to the server.
NNTP	Monitor fails to establish a connection to the server.
	Missing or invalid script arguments, which can include an invalid number of arguments or argument format.
	Monitor fails to find the NNTP group.
LDAP	Monitor fails to establish a connection to the server.
	Missing or invalid script arguments, which can include an invalid number of arguments or argument format.
	Monitor fails to bind to the LDAP server.
	Monitor fails to locate an entry for the target entity in the LDAP server.
FTP	The connection to the server times out.
	Missing or invalid script arguments, which can include an invalid number of arguments or argument format.
	Logon fails.
	Monitor fails to find the file on the server.
POP3	Monitor fails to establish a connection to the database.
	Missing or invalid script arguments, which can include an invalid number of arguments or argument format.
	Logon fails.
POP3	Monitor fails to establish a connection to the database.
	Missing or invalid script arguments, which can include an invalid number of arguments or argument format.
	Logon fails.
	Preparation of SQL query fails.
	Execution of SQL query fails.
SNMP	Monitor fails to establish a connection to the database.
	Missing or invalid script arguments, which can include an invalid number of arguments or argument format.
	Logon fails.
	Monitor fails to create the SNMP session.
	Monitor fails to find the object identifier.
	The monitor threshold value setting is greater than or equal to the actual threshold of the monitor.
RDP (Windows Terminal Server)	Missing or invalid script arguments, which can include an invalid number of arguments or argument format.
	Monitor fails to create a socket.
	Mismatch in versions.
	Monitor fails to confirm connection.

You can view the log file from the NetScaler command line by using the following commands, which open a BSD shell, display the log file on the screen, and then close the BSD shell and return you to the NetScaler command prompt:

> shell 
root@ns# cat /var/nslog/nsumond.log 
root@ns# exit 
>

User monitors also have a time-out value and a retry count for probe failures. You can use user monitors with non-user monitors. During high CPU utilization, a non-user monitor enables faster detection of a server failure.

Note: If the user monitor probe times out during high CPU usage, the state of the service remains unchanged.

===============================