Category Archives: scripting-isilon

Configuring a Powershell to Isilon Connection with SSL

PowerShell allows an easy method to access the Isilon ReST API, but in my environment I need to use true SSL validation. If you are using the default self-signed certificate of the Isilon, your connection will likely fail with an error similar to the one below:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

Isilon generates a self signed certificate by default.  Certificate validation for the current PowerShell session can be disabled with the script below, however in my environment I’m not allowed to do that.  I’m including it for completeness in case it is useful for someone else, it was not written by me but uses a BSD 3-Clause license.

function Disable-SSLValidation{
<#
.SYNOPSIS
    Disables SSL certificate validation
.DESCRIPTION
    Disable-SSLValidation disables SSL certificate validation by using reflection to implement the System.Net.ICertificatePolicy class.
    Author: Matthew Graeber (@mattifestation)
    License: BSD 3-Clause
.NOTES
    Reflection is ideal in situations when a script executes in an environment in which you cannot call csc.ese to compile source code. If compiling code is an option, then implementing System.Net.ICertificatePolicy in C# and Add-Type is trivial.
.LINK
    http://www.exploit-monday.com
#>
    Set-StrictMode -Version 2
    # You have already run this function if ([System.Net.ServicePointManager]::CertificatePolicy.ToString() -eq 'IgnoreCerts') { Return }
    $Domain = [AppDomain]::CurrentDomain
    $DynAssembly = New-Object System.Reflection.AssemblyName('IgnoreCerts')
    $AssemblyBuilder = $Domain.DefineDynamicAssembly($DynAssembly, [System.Reflection.Emit.AssemblyBuilderAccess]::Run)
    $ModuleBuilder = $AssemblyBuilder.DefineDynamicModule('IgnoreCerts', $false)
    $TypeBuilder = $ModuleBuilder.DefineType('IgnoreCerts', 'AutoLayout, AnsiClass, Class, Public, BeforeFieldInit', [System.Object], [System.Net.ICertificatePolicy])
  $TypeBuilder.DefineDefaultConstructor('PrivateScope, Public, HideBySig, SpecialName, RTSpecialName') | Out-Null
    $MethodInfo = [System.Net.ICertificatePolicy].GetMethod('CheckValidationResult')
    $MethodBuilder = $TypeBuilder.DefineMethod($MethodInfo.Name, 'PrivateScope, Public, Virtual, HideBySig, VtableLayoutMask', $MethodInfo.CallingConvention, $MethodInfo.ReturnType, ([Type[]] ($MethodInfo.GetParameters() | % {$_.ParameterType})))
    $ILGen = $MethodBuilder.GetILGenerator()
    $ILGen.Emit([Reflection.Emit.Opcodes]::Ldc_I4_1)
    $ILGen.Emit([Reflection.Emit.Opcodes]::Ret)
    $TypeBuilder.CreateType() | Out-Null

    # Disable SSL certificate validation
   [System.Net.ServicePointManager]::CertificatePolicy = New-Object IgnoreCerts
}

While that code may work fine for some, for security reasons you may not want to or be able to disable certificate validation.  Fortunately, you can create your own key pair with puttygen.  This solution was tested to work with OneFS v 7.2.x and PowerShell V3.

Here are the steps for creating your own key pair for PowerShell SSL authentication to Isilon:

Generate the Key

  1. Download Puttygen to generate the keypair for authentication.
    Open Puttygen and click Generate.
  2. It’s important to note that PowerShell requires exporting the key in OpenSSH format, which is done under the Conversions menu, and the option ‘Export OpenSSHKey’.  Save the key without a passphrase.  It can be named something like “SSH.key”.
  3. Next we need to save the public key.  Copy the information in the upper text box labeled “public key for pasting into OpenSSH authorized_keys file”, and paste it into a new text file.  You can then save the file as “authorized_keys” for later use.

Copy the Key

  1. Copy the authorized_keys file to the Isilon cluster to the location of your choosing.
  2. Open an SSH connection to the Isilon cluster and create a folder for the authorized_keys file.
    Example command:  isi_for_array mkdir /root/.ssh
  3. Copy the file to all nodes. Example command: isi_for_array cp /ifs/local/authorized_keys /root/.ssh/
  4. Verify that the file is available on all of the nodes, and it’s also a good idea to verify that the checksum is correct. Example command: isi_for_array md5 /root/.ssh/authorized_keys

Install PowerShell SSH Module

  1. In order to execute commands via SSH using PowerShell you will need to use an SSH module.  Various options exist, however the module from PowerShellAdmin works fine. It works for running commands via SSH on remote hosts such as Linux or Unix computers, VMware ESX(i) hosts or network equipment such as routers and switches that support SSH. It works well with OpenSSH-type servers.

You can visit the PowerShellAdmin page here,  and here is the direct download link for the SessionsPSv3.zip file.

  1. Once you’ve downloaded it, unzip the file to the SSH-Sessions folder, located in C:\Windows\System32\WindowsPowerShell\v1.0\Modules. With that module in place, we are now ready to connect with PowerShell to the Isilon cluster.

Test it

Below is a powershell script you can use to test your connection, it simply runs a df command on the cluster.

#PowerShell Test Script
Import-Module "SSH-Sessions"
$Isilon = "<hostname>"
KeyFile = "C:\scripts\<filename>.key"
New-SshSession -ComputerName $Isilon -Username root -KeyFile $KeyFile
Invoke-SshCommand -verbose -ComputerName $Isilon -Command df  
Remove-SshSession -ComputerName $Isilon

 

 

 

Advertisements

Scripting a VNX/Celerra to Isilon Data Migration with EMCOPY and Perl

datamigration

Below is a collection of perl scripts that make data migration from VNX/Celerra file systems to an Isilon system much easier.  I’ve already outlined the process of using isi_vol_copy_vnx in a prior post, however using EMCOPY may be more appropriate in a specific use case, or simply more familiar to administrators for support and management of the tool.  Note that while I have tested these scripts in my environment, they may need some modification for your use.  I recommend running them in a test environment prior to using them in production.

EMCOPY can be downloaded directly from DellEMC with the link below.  You will need to be a registered user in order to download it.

https://download.emc.com/downloads/DL14101_EMCOPY_File_migration_tool_4.17.exe

What is EMCOPY?

For those that haven’t used it before, EMCOPY is an application that allows you to copy a file, directory, and subdirectories between NTFS partitions while maintaining security information, an improvement over the similar robocopy tool that many veteran system administrators are familiar with. It allows you to back up the file and directory security ACLs, owner information, and audit information from a source directory to a destination directory.

Notes about using EMCOPY:

1) In my testing, EMCopy has shown up to a 25% performance improvement when copying CIFS data compared to Robocopy while using the same number of threads. I recommend using EMCopy over Robocopy as it has other feature improvements as well, for instance sidmapfile, which allows migrating local user data to Active Directory users. It’s available in version 4.17 or later.  Robocopy is also not an EMC supported tool, while EMCOPY is.

2) Unlike isi_vol_copy_vnx, EMCOPY is a windows application and must be run from a windows host.  I highly recommend a dedicated server for any migration tasks.  The isi_vol_copy_vnx utility runs directly on the Isilon OneFS CLI which eliminates any intermediary copy hosts, theoretically providing a much faster solution.

3) There are multiple methods to compare data sizes between the source and destination. I would recommend maintaining a log of each EMCopy session as that log indicates how much data was copied and if there were any errors.

4) If you are migrating over a WAN connection, I recommend first restoring from tape and then using an incremental data sync with EMCOPY.

Getting Started

I’ve divided this post up into a four step process.  Each step includes the relevant script and a description of the process.

  • Export File System information (export_fs.pl  Script)

Export file system information from the Celerra & generate the Isilon commands to re-create them.

  • Export SMB information (export_smb.pl Script)

Export SMB share information from the Celerra & generate the Isilon commands to re-create them.

  • Export NFS information (export_nfs.pl Script)

Export NFS information from the Celerra & generate the Isilon commands to re-create them.

  • Create the EMCOPY migration script (EMCOPY_create.pl Script)

Perform the data migration with EMCOPY using the output from this script.

Exporting information from the Celerra to run on the Isilon

These Perl scripts are designed to be run directly on the Control Station and will subsequently create shell scripts that will run on the Isilon to assist with the migration.  You will need to manually copy the output files from the VNX/Celerra to the Isilon. The first three steps I’ve outlined do not move the data or permissions, they simply run a nas_fs query on the Celerra to generate the Isilon script files that actually make the directories, create quotas, and create the NFS and SMB shares. They are “scripts that generate scripts”. 🙂

Before you run the scripts, make sure you edit them to correctly specify the appropriate Data Mover.  Once complete, You’ll end up with three .sh files created for you to move to your Isilon cluster.  They should be run in the same order as they were created.

Note that EMC occasionally changes the syntax of certain commands when they update OneFS.  Below is a sample of the isilon specific commands that are generated by the first three scripts.  I’d recommend verifying that the syntax is still correct with your version of OneFS, and then modify the scripts if necessary with the new syntax.  I just ran a quick test with OneFS 8.0.0.2, and the base commands and switches appear to be compatible.

isi quota create –directory –path=”/ifs/data1″ –enforcement –hard-threshold=”1032575M” –container=1
isi smb share create –name=”Data01″ –path=”/ifs/Data01/data”
isi nfs exports create –path=”/Data01/data”  –roclient=”Data” –rwclient=”Data” –rootclient=”Data”

 

Step 1 – Export File system information

This script will generate a list of the file system names from the Celerra and place the appropriate Isilon commands that create the directories and quotes into a file named “create_filesystems_xx.sh”.

#!/usr/bin/perl

# Export_fs.pl – Export File system information
# Export file system information from the Celerra & generate the Isilon commands to re-create them.

use strict;
my $nas_fs="nas_fs -query:inuse=y:type=uxfs:isroot=false -fields:ServersNumeric,Id,Name,SizeValues -format:'%s,%s,%s,%sQQQQQQ'";
my @data;

open (OUTPUT, ">> create_filesystems_$$.sh") || die "cannot open output: $!\n\n";
open (CMD, "$nas_fs |") || die "cannot open $nas_fs: $!\n\n";

while ()

{
   chomp;
   @data = split("QQQQQQ", $_);
}

close(CMD);
foreach (@data)

{
   my ($dm, $id, $dir,$size,$free,$used_per, $inodes) = split(",", $_);
   print OUTPUT "mkdir /ifs/$dir\n";
   print OUTPUT "chmod 755 /ifs/$dir\n";
   print OUTPUT "isi quota create --directory --path=\"/ifs/$dir\" --enforcement --hard-threshold=\"${size}M\" --container=1\n";
}

The Output of the script looks like this (this is an excerpt from the create_filesystems_xx.sh file):

isi quota create --directory --path="/ifs/data1" --enforcement --hard-threshold="1032575M" --container=1
mkdir /ifs/data1
chmod 755 /ifs/data1
isi quota create --directory --path="/ifs/data2" --enforcement --hard-threshold="20104M" --container=1
mkdir /ifs/data2
chmod 755 /ifs/data2
isi quota create --directory --path="/ifs/data3" --enforcement --hard-threshold="100774M" --container=1
mkdir /ifs/data3
chmod 755 /ifs/data3

The output script can now be copied to and run from the Isilon.

Step 2 – Export SMB Information

This script will generate a list of the smb share names from the Celerra and place the appropriate Isilon commands into a file named “create_smb_exports_xx.sh”.

#!/usr/bin/perl

# Export_smb.pl – Export SMB/CIFS information
# Export SMB share information from the Celerra & generate the Isilon commands to re-create them.

use strict;

my $datamover = "server_8";
my $prot = "cifs";:wq!
my $nfs_cli = "server_export $datamover -list -P $prot -v |grep share";

open (OUTPUT, ">> create_smb_exports_$$.sh") || die "cannot open output: $!\n\n";
open (CMD, "$nfs_cli |") || die "cant open $nfs_cli: $!\n\n";

while ()
{
   chomp;
   my (@vars) = split(" ", $_);
   my $path = $vars[2];
   my $name = $vars[1];

   $path =~ s/^"/\"\/ifs/;
   print  OUTPUT "isi smb share create --name=$name --path=$path\n";
}

close(CMD);

The Output of the script looks like this (this is an excerpt from the create_smb_exports_xx.sh file):

isi smb share create --name="Data01" --path="/ifs/Data01/data"
isi smb share create --name="Data02" --path="/ifs/Data02/data"
isi smb share create --name="Data03" --path="/ifs/Data03/data"
isi smb share create --name="Data04" --path="/ifs/Data04/data"
isi smb share create --name="Data05" --path="/ifs/Data05/data"

 The output script can now be copied to and run from the Isilon.

Step 3 – Export NFS Information

This script will generate a list of the NFS export names from the Celerra and place the appropriate Isilon commands into a file named “create_nfs_exports_xx.sh”.

#!/usr/bin/perl

# Export_nfs.pl – Export NFS information
# Export NFS information from the Celerra & generate the Isilon commands to re-create them.

use strict;

my $datamover = "server_8";
my $prot = "nfs";
my $nfs_cli = "server_export $datamover -list -P $prot -v |grep export";

open (OUTPUT, ">> create_nfs_exports_$$.sh") || die "cannot open output: $!\n\n";
open (CMD, "$nfs_cli |") || die "cant open $nfs_cli: $!\n\n";

while ()
{
   chomp;
   my (@vars) = split(" ", $_);
   my $test = @vars;
   my $i=2;
   my ($ro, $rw, $root, $access, $name);
   my $path=$vars[1];

   for ($i; $i < $test; $i++)
   {
      my ($type, $value) = split("=", $vars[$i]);

      if ($type eq "ro") {
         my @tmp = split(":", $value);
         foreach(@tmp) { $ro .= " --roclient=\"$_\""; }
      }
      if ($type eq "rw") {
         my @tmp = split(":", $value);
         foreach(@tmp) { $rw .= " --rwclient=\"$_\""; }
      }

      if ($type eq "root") {
         my @tmp = split(":", $value);
         foreach(@tmp) { $root .= " --rootclient=\"$_\""; }
      }

      if ($type eq "access") {
         my @tmp = split(":", $value);
         foreach(@tmp) { $ro .= " --roclient=\"$_\""; }
      }

      if ($type eq "name") { $name=$value; }
   }
   print OUTPUT "isi nfs exports create --path=$path $ro $rw $root\n";
}

close(CMD);

The Output of the script looks like this (this is an excerpt from the create_nfs_exports_xx.sh file):

isi nfs exports create --path="/Data01/data" --roclient="Data" --roclient="BACKUP" --rwclient="Data" --rwclient="BACKUP" --rootclient="Data" --rootclient="BACKUP"
isi nfs exports create --path="/Data02/data" --roclient="Data" --roclient="BACKUP" --rwclient="Data" --rwclient="BACKUP" --rootclient="Data" --rootclient="BACKUP"
isi nfs exports create --path="/Data03/data" --roclient="Backup" --roclient="Data" --rwclient="Backup" --rwclient="Data" --rootclient="Backup" --rootclient="Data"
isi nfs exports create --path="/Data04/data" --roclient="Backup" --roclient="ProdGroup" --rwclient="Backup" --rwclient="ProdGroup" --rootclient="Backup" --rootclient="ProdGroup"
isi nfs exports create --path="/" --roclient="127.0.0.1" --roclient="127.0.0.1" --roclient="127.0.0.1" -rootclient="127.0.0.1"

The output script can now be copied to and run from the Isilon.

Step 4 – Generate the EMCOPY commands

Now that the scripts have been generated and run on the Isilon, the next step is the actual data migration using EMCOPY.  This script will generate the commands for a migration script, which should be run from a windows server that has access to both the source and destination locations. It should be run after the previous three scripts have successfully completed.

This script will output the commands directly to the screen, it can then be cut and pasted from the screen directly into a windows batch script on your migration server.

#!/usr/bin/perl

# EMCOPY_create.pl – Create the EMCOPY migration script
# Perform the data migration with EMCOPY using the output from this script.

use strict;

my $datamover = "server_4";
my $source = "\\\\celerra_path\\";
my $dest = "\\\\isilon_path\\";
my $prot = "cifs";
my $nfs_cli = "server_export $datamover -list -P $prot -v |grep share";

open (OUTPUT, ">> create_smb_exports_$$.sh") || die "cant open output: $!\n\n";
open (CMD, "$nfs_cli |") || die "cant open $nfs_cli: $!\n\n";

while ()
{
   chomp;
   my (@vars) = split(" ", $_);
   my $path = $vars[2];
   my $name = $vars[1];

   $name =~ s/\"//g;
   $path =~ s/^/\/ifs/;

   my $log = "c:\\" . $name . "";
   $log =~ s/ //;
   my $src = $source . $name;
   my $dst = $dest . $name;

   print "emcopy \"$src\" \"$  dst\" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:$log\n";
}

close(CMD);

The Output of the script looks like this (this is an excerpt from the screen output):

emcopy "\\celerra_path\Data01" "\\isilon_path\billing_tmip_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_tmip_01
emcopy "\\celerra_path\Data02" "\\isilon_path\billing_trxs_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_trxs_01
emcopy "\\celerra_path\Data03" "\\isilon_path\billing_vru_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_vru_01
emcopy "\\celerra_path\Data04" "\\isilon_path\billing_rpps_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_rpps_01

That’s it.  Good luck with your data migration, and I hope this has been of some assistance.  Special thanks to Mark May and his virtualstoragezone blog, he published the original versions of these scripts here.

Custom Reporting with Isilon OneFS API Calls

Have you ever wondered where the metrics you see in InsightIQ come from?  InsightIQ uses OneFS API calls to gather information, and you can use the same API calls for custom reporting and scripting.  Whether you’re interested in performance metrics relating to cluster capacity, CPU utilization, network latency & throughput, or disk activities, you have access to all of that information.

I spent a good deal of time already on how to make this work and investigating options that are available to make the gathered data be presentable in some useful manner.  This is really just the beginning, I’m hoping to take some more time later to work on additional custom script examples that gather specific info and have useful output options.  For now, this should get anyone started who’s interested in trying this out.  This post also includes a list of available API calls you can make.  I cover these basic steps in this post to get you started:

  1. How to authenticate to the Isilon Cluster using cookies.
  2. How to make the API call to the Isilon to generate the JSON output.
  3. How to install the jq utility to parse JSON output files.
  4. Some examples of using the jq utility to parse the JSON output.

Authentication

First I’ll go over how to authenticate to the Isilon cluster using cookies. You’ll have to create a credentials file first.  Name the file auth.json and enter the following info into it:

{
 "username":"root",
 "password":"<password>",
 "services":["platform","namespace"]
 }

Note that I am using root for this example, but it would certainly be possible to create a separate account on the Isilon to use for this purpose.  Just give the account the Platform API and Statistics roles.

Once the file is created, you can make a session call to get a cookie:

curl -v -k –insecure -H “Content-Type: application/json” -c cookiefile -X POST -d @auth.json https://10.10.10.10:8080/session/1/session

The output will be over one page long, but you’re looking to verify that the cookie was generated.  You should see two lines similar to this:

* Added cookie isisessid="123456-xxxx-xxxx-xxxx-a193333a99bc" for domain 10.10.10.10, path /, expire 0
< Set-Cookie: isisessid=123456-xxxx-xxxx-xxxx-a193333a99bc; path=/; HttpOnly; Secure

Run a test command to gather data

Below is a sample string to gather some sample statistics.  Later in this document I’ll review all of the possible strings you can use to gather info on CPU, disk, performance, etc.

curl -k –insecure -b @cookiefile ‘https://10.10.10.10:8080/platform/1/statistics/current?key=ifs.bytes.total’

The command above generates the following json output:

{
"stats" :
[

{
"devid" : 0,
"error" : null,
"error_code" : null,
"key" : "ifs.bytes.total",
"time" : 1398840008,
"value" : 433974304096256
}
]
}

Install JQ

Now that we have data in json format, we need to be able to parse it and change it into a more readable format.  I’m looking to convert it to csv.  There are many different scripts, tools, and languages available for that purpose online.  I personally looked for a method that can be used in a simple bash script and jq is a good solution for that.  I use Cygwin on a windows box for my scripts, but you can download any version you like for your flavor of OS.  You can download the JQ parser here: https://github.com/stedolan/jq/releases.

Instructions for the installation of jq for Cygwin:

  1. Download the latest source tarball for jq from https://stedolan.github.io/jq/download/
  2. Open Cygwin to create the folder you’d like to extract it in
  3. Copy the ‘jq-1.5.tar.gz’ file into your folder to make it available within Cygwin
  4. From a Cygwin command shell, enter the following to uncompress the tarball file : ‘tar -xvzf jq-1.5.tar.gz’
  5. Change folder location to the uncompressed folder e.g. ‘cd /jq-1.5’
  6. Next enter ‘./configure’ and wait for the command to complete (about 5 minutes)
  7. Then enter the commands ‘make’, followed by ‘make install’
  8. You’re done.

Once jq is installed, we can play around with using it to make our json output more readable.  One simple way to make it into a comma separated output, is with this command:

cat sample.json | jq “.stats | .[]” –compact-output

It will turn json output like this:

{
"stats" :
[

{
"devid" : 8,
"error" : null,
"error_code" : null,
"key" : "node.ifs.bytes.used",
"values" :
[

{
"time" : 1498745964,
"value" : 51694140276736
},
{
"time" : 1498746264,
"value" : 51705407610880
}
]
},

Into single line, comma separated output like this:

{"devid":8,"error":null,"error_code":null,"key":"node.ifs.bytes.used","values":[{"time":1498745964,"value":51694140276736},{"time":1498746264,"value":51705407610880}]}

You can further improve the output by removing the quote marks with sed:

cat sample.json | jq “.stats | .[]” –compact-output | sed ‘s/\”//g’

At this point the data is formatted well enough to easily modify it to suit my needs in Excel.

{devid:8,error:null,error_code:null,key:node.ifs.bytes.used,values:[{time:1498745964,value:51694140276736},{time:1498746264,value:51705407610880}]}

JQ CSV

Using the –compact-output switch isn’t the only way to manipulate the data, and probably not the best way.  I haven’t had much time to work with the @csv option in JQ, but it looks very promising. for this.  Below are a few notes on using it, I will include more samples in an edit to this post or a new post in the future that relate this more directly to using it with the Isilon-generated output.  I prefer to use csv files for report output due to the ease of working with them and manipulating them with scripts.

Order is significant for csv, but not for JSON fields.  Specify the mapping from JSON named fields to csv positional fields by constructing an array of those fields, using [.date,.count,.title]:

input: { "date": "2011-01-12 13:14", "count": 17, "title":"He's dead, Jim!" }
jq -r '[.date,.count,.title] | @csv'
"2017-06-12 08:19",17,"You're going too fast!"

You also may want to apply this to an array of objects, in which case you’ll need to use the .[] operator, which streams each item of an array in turn:

jq -r '.[] | [.date, .count, .title] | @csv'
"2017-06-12 08:19",17,"You're going too fast!"
"2017-06-15 11:50",4711,"That's impossible"?"
"2017-06-19 00:01",,"I can't drive 55!"

You’ll likely also want the csv file to populate the csv field names at the top. The easiest way to do this is to add them in manually:

jq -r '["date", "count", "title"], (.[] | [.date, .count, .title]) | @csv'
"date","count","title"
"2017-06-12 08:19",17,"You're going too fast!"
"2017-06-15 11:50",4711,"That's impossible"?"
"2017-06-19 00:01",,"I can't drive 55!"

We can avoid repeating the same list of field names by reusing the header array to lookup the fields in each object.

jq -r '["date", "count", "title"] as $fields| $fields, (.[] | [.[$fields[]]]) | @csv'

Here it is as a function, with a slightly nicer field syntax, using path():

def csv(fs): [path(null|fs)[]] as $fields| $fields, (.[] | [.[$fields[]]]) | @csv;
USAGE: csv(.date, .count, .title)

If the input is not an array of objects but just a sequence of objects  then we can omit the .[] – but then we can’t get the header at the top.  It’s best to convert it to an array using the –slurp/-s option (or put [] around it if it’s generated within jq).

More to come on formatting JSON for Isilon in the future…

Isilon API calls

All of these specific API calls were pulled from the EMC community forum, I didn’t compose this list myself.  It’s a list of the calls that InsightIQ makes to the OneFS API.  They can be queried in exactly the same way that I demonstrated in the examples earlier in this post.

Please note the following about the API calls regarding time ranges:

  1. Every call to the “/platform/1/statistics/current” APIs do not contain query parameters for &begin and &end time range.
  2. Every call to the “/platform/1/statistics/history” APIs always contain query parameters for &begin and &end POSIX time range.

Capacity

https://10.10.10.10:8080/platform/1/statistics/current?key=ifs.bytes.total&key=ifs.ssd.bytes.total&key=ifs.bytes.free&key=ifs.ssd.bytes.free&key=ifs.bytes.avail&key=ifs.ssd.bytes.avail&devid=all

https://10.10.10.10:8080/platform/1/statistics/current?key=node.ifs.bytes.total&devid=all

https://10.10.10.10:8080/platform/1/statistics/current?key=node.ifs.bytes.total&key=node.ifs.bytes.used&key=node.disk.count&key=node.cpu.count&key=node.uptime&devid=all

https://10.10.10.10:8080/platform/1/statistics/history?key=ifs.bytes.avail&key=ifs.bytes.total&key=ifs.bytes.free&key=ifs.ssd.bytes.free&key=ifs.ssd.bytes.avail&key=ifs.ssd.bytes.total&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.ifs.bytes.used.all&key=node.disk.ifs.bytes.total.all&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.bytes.out.rate&key=node.ifs.bytes.in.rate&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.bytes.total&key=node.ifs.ssd.bytes.used&key=node.ifs.ssd.bytes.total&key=node.ifs.bytes.used&devid=all&degraded=true&interval=300&memory_only=true

CPU

https://10.10.10.10:8080/platform/1/statistics/history?key=node.cpu.idle.avg&devid=all&degraded=true&interval=30&memory_only=true

Network

https://10.10.10.10:8080/platform/1/statistics/current?key=node.net.iface.name.0&key=node.net.iface.name.1&key=node.net.iface.name.2&key=node.net.iface.name.3&key=node net.iface.name.4&key=node.net.iface.name.5&key=node.net.iface.name.6&key=node.net.iface.name.7&key=node.net.iface.name.8&key=node.net.iface.name.9&devid=all

https://10.10.10.10:8080/platform/1/statistics/history?key=node.net.ext.packets.in.rate&key=node.net.ext.errors.in.rate&key=node.net.ext.bytes.out.rate&key=node.net.ext.errors.out.rate&key=node.net.ext.bytes.in.rate&key=node.net.ext.packets.out.rate&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.net.iface.bytes.out.rate.0&key=node.net.iface.bytes.out.rate.1&key=node.net.iface.bytes.out.rate.2&key=node.net.iface.bytes.out.rate.3&key=node.net.iface.bytes.out.rate.4&key=node.net.iface.bytes.out.rate.5&key=node.net.iface.bytes.out.rate.6&key=node.net.iface.bytes.out.rate.7&key=node.net.iface.bytes.out.rate.8&key=node.net.iface.bytes.out.rate.9&key=node.net.iface.errors.in.rate.0&key=node.net.iface.errors.in.rate.1&key=node.net.iface.errors.in.rate.2&key=node.net.iface.errors.in.rate.3&key=node.net.iface.errors.in.rate.4&key=node.net.iface.errors.in.rate.5&key=node.net.iface.errors.in.rate.6&key=node.net.iface.errors.in.rate.7&key=node.net.iface.errors.in.rate.8&key=node.net.iface.errors.in.rate.9&key=node.net.iface.errors.out.rate.0&key=node.net.iface.errors.out.rate.1&key=node.net.iface.errors.out.rate.2&key=node.net.iface.errors.out.rate.3&key=node.net.iface.errors.out.rate.4&key=node.net.iface.errors.out.rate.5&key=node.net.iface.errors.out.rate.6&key=node.net.iface.errors.out.rate.7&key=node.net.iface.errors.out.rate.8&key=node.net.iface.errors.out.rate.9&key=node.net.iface.packets.in.rate.0&key=node.ne .iface.packets.in.rate.1&key=node.net.iface.packets.in.rate.2&key=node.net.iface.packets.in.rate.3&key=node.net.iface.packets.in.rate.4&key=node.net.iface.packets.in.rate.5&key=node.net.iface.packets.in.rate.6&key=node.net.iface.packets.in.rate.7&key=node.net.iface.packets.in.rate.8&key=node.net.iface.packets.in.rate.9&key=node.net.iface.bytes.in.rate.0&key=node.net.iface.bytes.in.rate.1&key=node.net.iface.bytes.in.rate.2&key=node.net.iface.bytes.in.rate.3&key=node.net.iface.bytes.in.rate.4&key=node.net.iface.bytes.in.rate.5&key=node.net.iface.bytes.in.rate.6&key=node.net.iface.bytes.in.rate.7&key=node.net.iface.bytes.in.rate.8&key=node.net.iface.bytes.in.rate.9&key=node.net.iface.packets.out.rate.0&key=node.net.iface.packets.out.rate.1&key=node.net.iface.packets.out.rate.2&key=node.net.iface.packets.out.rate.3&key=node.net.iface.packets.out.rate.4&key=node.net.iface.packets.out.rate.5&key=node.net.iface.packets.out.rate.6&key=node.net.iface.packets.out.rate.7&key=node.net.iface.packets.out.rate.8&key=node.net.iface.packets.out.rate.9&devid=all&degraded=true&interval=30&memory_only=true

Disk

https://10.10.10.10:8080/platform/1/statistics/current?key=node.disk.count&devid=all

https://10.10.10.10:8080/platform/1/statistics/current?key=node.disk.name.0&key=node.disk.name.1&key=node.disk.name.2&key=node.disk.name.3&key=node.disk.name.4&key=node.disk.name.5&key=node.disk.name.6&key=node.disk.name.7&key=node.disk.name.8&key=node.disk.name.9&key=node.disk.name.10&key=node.disk.name.11&key=node.disk.name.12&key=node.disk.name.13&key=node.disk.name.14&key=node.disk.name.15&key=node.disk.name.16&key=node.disk.name.17&key=node.disk.name.18&key=node.disk.name.19&key=node.disk.name.20&key=node.disk.name.21&key=node.disk.name.22&key=node.disk.name.23&key=node.disk.name.24&key=node.disk.name.25&key=node.disk.name.26&key=node.disk.name.27&key=node.disk.name.28&key=node.disk.name.29&key=node.disk.name.30&key=node.disk.name.31&key=node.disk.name.32&key=node.disk.name.33&key=node.disk.name.34&key=node.disk.name.35&devid=all

https://10.10.10.10:8080/platform/1/statistics/current?key=node.ifs.bytes.total&key=node.ifs.bytes.used&key=node.disk.count&key=node.cpu.count&key=node.uptime&devid=all

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.access.latency.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.access.slow.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.busy.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.bytes.in.rate.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.bytes.out.rate.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.ifs.bytes.used.all&key=node.disk.ifs.bytes.total.all&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.iosched.latency.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.iosched.queue.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.xfer.size.in.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.xfer.size.out.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.xfers.in.rate.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.xfers.out.rate.all&devid=all&degraded=true&interval=30&memory_only=true

Complete List of API calls made by InsightIQ

Here is a complete a list of all of the API calls that InsightIQ makes to the Isilon cluster using OneFS API. For complete reference of what these APIs actually do, you can refer to the OneFS API Info Hub and the OneFS API Reference documentation.

https://10.10.10.10:8080/platform/1/cluster/config

https://10.10.10.10:8080/platform/1/cluster/identity

https://10.10.10.10:8080/platform/1/cluster/time

https://10.10.10.10:8080/platform/1/dedupe/dedupe-summary

https://10.10.10.10:8080/platform/1/dedupe/reports

https://10.10.10.10:8080/platform/1/fsa/path

https://10.10.10.10:8080/platform/1/fsa/results

https://10.10.10.10:8080/platform/1/job/types

https://10.10.10.10:8080/platform/1/license/licenses

https://10.10.10.10:8080/platform/1/license/licenses/InsightIQ

https://10.10.10.10:8080/platform/1/quota/reports

https://10.10.10.10:8080/platform/1/snapshot/snapshots-summary

https://10.10.10.10:8080/platform/1/statistics/current?key=cluster.health&devid=all

https://10.10.10.10:8080/platform/1/statistics/current?key=ifs.bytes.total&key=ifs.ssd.bytes.total&key=ifs.bytes.free&key=ifs.ssd.bytes.free&key=ifs.bytes.avail&key=ifs.ssd.bytes.avail&devid=all

https://10.10.10.10:8080/platform/1/statistics/current?key=node.disk.count&devid=all

https://10.10.10.10:8080/platform/1/statistics/current?key=node.disk.name.0&key=node.disk.name.1&key=node.disk.name.2&key=node.disk.name.3&key=node.disk.name.4&key=node.disk.name.5&key=node.disk.name.6&key=node.disk.name.7&key=node.disk.name.8&key=node.disk.name.9&key=node.disk.name.10&key=node.disk.name.11&key=node.disk.name.12&key=node.disk.name.13&key=node.disk.name.14&key=node.disk.name.15&key=node.disk.name.16&key=node.disk.name.17&key=node.disk.name.18&key=node.disk.name.19&key=node.disk.name.20&key=node.disk.name.21&key=node.disk.name.22&key=node.disk.name.23&key=node.disk.name.24&key=node.disk.name.25&key=node.disk.name.26&key=node.disk.name.27&key=node.disk.name.28&key=node.disk.name.29&key=node.disk.name.30&key=node.disk.name.31&key=node.disk.name.32&key=node.disk.name.33&key=node.disk.name.34&key=node.disk.name.35&devid=all

https://10.10.10.10:8080/platform/1/statistics/current?key=node.ifs.bytes.total&devid=all

https://10.10.10.10:8080/platform/1/statistics/current?key=node.ifs.bytes.total&key=node.ifs.bytes.used&key=node.disk.count&key=node.cpu.count&key=node.uptime&devid=all

https://10.10.10.10:8080/platform/1/statistics/current?key=node.net.iface.name.0&key=node.net.iface.name.1&key=node.net.iface.name.2&key=node.net.iface.name.3&key=node.net.iface.name.4&key=node.net.iface.name.5&key=node.net.iface.name.6&key=node.net.iface.name.7&key=node.net.iface.name.8&key=node.net.iface.name.9&devid=all

https://10.10.10.10:8080/platform/1/statistics/history?key=cluster.dedupe.estimated.saved.bytes&key=cluster.dedupe.logical.deduplicated.bytes&key=cluster.dedupe.logical.saved.bytes&key=cluster.dedupe.estimated.deduplicated.bytes&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=ifs.bytes.avail&key=ifs.bytes.total&key=ifs.bytes.free&key=ifs.ssd.bytes.free&key=ifs.ssd.bytes.avail&key=ifs.ssd.bytes.total&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.active.ftp&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.active.hdfs&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.active.http&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.active.nfs3&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.active.nfs4&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.active.nlm&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.active.papi&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.active.siq&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.active.smb1&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.active.smb2&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.connected.ftp&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.connected.hdfs&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.connected.http&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.connected.nfs&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.connected.nlm&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.connected.papi&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.connected.siq&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.connected.smb&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.proto.ftp&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.proto.hdfs&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.proto.http&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.proto.nfs3&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.proto.nfs4&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.proto.nlm&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.proto.papi&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.proto.siq&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.proto.smb1&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.clientstats.proto.smb2&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.cpu.idle.avg&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.access.latency.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.access.slow.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.busy.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.bytes.in.rate.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.bytes.out.rate.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.ifs.bytes.used.all&key=node.disk.ifs.bytes.total.all&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.iosched.latency.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.iosched.queue.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.xfer.size.in.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.xfer.size.out.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.xfers.in.rate.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.disk.xfers.out.rate.all&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.bytes.out.rate&key=node.ifs.bytes.in.rate&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.bytes.total&key=node.ifs.ssd.bytes.used&key=node.ifs.ssd.bytes.total&key=node.ifs.bytes.used&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.cache&key=node.ifs.cache.l3.data.read.miss&key=node.ifs.cache.l3.meta.read.hit&key=node.ifs.cache.l3.data.read.hit&key=node.ifs.cache.l3.data.read.start&key=node.ifs.cache.l3.meta.read.start&key=node.ifs.cache.l3.meta.read.miss&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.blocked&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.blocked.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.contended&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.contended.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.deadlocked&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.deadlocked.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.getattr&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.getattr.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.link&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.link.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.lock&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.lock.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.lookup&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.lookup.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.read&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.read.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.rename&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.rename.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.setattr&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.setattr.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.unlink&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.unlink.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.write&devid=all&degraded=true&interval=300&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.ifs.heat.write.total&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.je.num_workers&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.net.ext.packets.in.rate&key=node.net.ext.errors.in.rate&key=node.net.ext.bytes.out.rate&key=node.net.ext.errors.out.rate&key=node.net.ext.bytes.in.rate&key=node.net.ext.packets.out.rate&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.net.iface.bytes.out.rate.0&key=node.net.iface.bytes.out.rate.1&key=node.net.iface.bytes.out.rate.2&key=node.net.iface.bytes.out.rate.3&key=node.net.iface.bytes.out.rate.4&key=node.net.iface.bytes.out.rate.5&key=node.net.iface.bytes.out.rate.6&key=node.net.iface.bytes.out.rate.7&key=node.net.iface.bytes.out.rate.8&key=node.net.iface.bytes.out.rate.9&key=node.net.iface.errors.in.rate.0&key=node.net.iface.errors.in.rate.1&key=node.net.iface.errors.in.rate.2&key=node.net.iface.errors.in.rate.3&key=node.net.iface.errors.in.rate.4&key=node.net.iface.errors.in.rate.5&key=node.net.iface.errors.in.rate.6&key=node.net.iface.errors.in.rate.7&key=node.net.iface.errors.in.rate.8&key=node.net.iface.errors.in.rate.9&key=node.net.iface.errors.out.rate.0&key=node.net.iface.errors.out.rate.1&key=node.net.iface.errors.out.rate.2&key=node.net.iface.errors.out.rate.3&key=node.net.iface.errors.out.rate.4&key=node.net.iface.errors.out.rate.5&key=node.net.iface.errors.out.rate.6&key=node.net.iface.errors.out.rate.7&key=node.net.iface.errors.out.rate.8&key=node.net.iface.errors.out.rate.9&key=node.net.iface.packets.in.rate.0&key=node.net.iface.packets.in.rate.1&key=node.net.iface.packets.in.rate.2&key=node.net.iface.packets.in.rate.3&key=node.net.iface.packets.in.rate.4&key=node.net.iface.packets.in.rate.5&key=node.net.iface.packets.in.rate.6&key=node.net.iface.packets.in.rate.7&key=node.net.iface.packets.in.rate.8&key=node.net.iface.packets.in.rate.9&key=node.net.iface.bytes.in.rate.0&key=node.net.iface.bytes.in.rate.1&key=node.net.iface.bytes.in.rate.2&key=node.net.iface.bytes.in.rate.3&key=node.net.iface.bytes.in.rate.4&key=node.net.iface.bytes.in.rate.5&key=node.net.iface.bytes.in.rate.6&key=node.net.iface.bytes.in.rate.7&key=node.net.iface.bytes.in.rate.8&key=node.net.iface.bytes.in.rate.9&key=node.net.iface.packets.out.rate.0&key=node.net.iface.packets.out.rate.1&key=node.net.iface.packets.out.rate.2&key=node.net.iface.packets.out.rate.3&key=node.net.iface.packets.out.rate.4&key=node.net.iface.packets.out.rate.5&key=node.net.iface.packets.out.rate.6&key=node.net.iface.packets.out.rate.7&key=node.net.iface.packets.out.rate.8&key=node.net.iface.packets.out.rate.9&devid=all&degraded=true&interval=30&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.protostats.ftp&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.protostats.hdfs&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.protostats.http&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.protostats.nfs3&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.protostats.nfs4&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.protostats.nlm&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.protostats.papi&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.protostats.siq&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.protostats.smb1&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/history?key=node.protostats.smb2&devid=all&degraded=true&interval=120&memory_only=true

https://10.10.10.10:8080/platform/1/statistics/keys

https://10.10.10.10:8080/platform/1/statistics/protocols

https://10.10.10.10:8080/platform/1/storagepool/nodepools

https://10.10.10.10:8080/platform/1/storagepool/tiers

https://10.10.10.10:8080/platform/1/storagepool/unprovisioned

https://10.10.10.10:8080/session/1/session

Scripting automatic reports for Isilon from the CLI

We recently set up a virtual demo of an Isilon system on our network as we are evaluating Isilon for a possible purchase.  You can obtain a virtual node that runs on ESX from your local EMC Isilon representative, along with temporary licenses to test everything out.  As part of the test I wanted to see if it was possible to create custom CLI scripts in the same way that I create them on the Celerra or VNX File.  On the Celerra, I run daily scripts that output file pool sizes, file systems & disk space, failover status, logs, health check info, checkpoint info, etc. to my web report page.  Can you do the same thing on an Isilon?

Well, to start with, Isilon’s commands are completely different.  The first step of course was to look at what was available to me.  All of the Isilon administration commands appear to begin with ‘isi’.  If you type isi by itself it will show you the basic list of commands and what they do:

isilon01-1% isi
Description:
OneFS cluster administration.
Usage:
isi  <subcommand>
[--timeout <integer>]
[{--help | -h}]
Subcommands:
Cluster Monitoring:
alert*           An alias for "isi events".
audit            Manage audit configuration.
events*          Manage cluster events.
perfstat*        View cluster performance statistics.
stat*            An alias for "isi status".
statistics*      View general cluster statistics.
status*          View cluster status.
Cluster Configuration:
config*          Manage general cluster settings.
email*           Configure email settings.
job              Isilon job management commands.
license*         Manage software licenses.
networks*        Manage network settings.
services*        Manage cluster services.
update*          Update OneFS system software.
pkg*             Manage OneFS system software patches.
version*         View system version information.
remotesupport    Manage remote support settings.
Hardware & Devices:
batterystatus*   View battery status.
devices*         Manage cluster hardware devices.
fc*              Manage Fibre Channel settings.
firmware*        Manage system firmware.
lun*             Manage iSCSI logical units (LUNs).
target*          Manage iSCSI targets.
readonly*        Toggle node read-write state.
servicelight*    Toggle node service light.
tape*            Manage tape and media changer devices.
File System Configuration:
get*             View file system object properties.
set*             Manage file system object settings.
quota            Manage SmartQuotas, notifications and reports.
smartlock*       Manage SmartLock settings.
domain*          Manage file system domains.
worm*            Manage SmartLock WORM settings.
dedupe           Manage Dedupe settings.
Access Management:
auth             Manage authentication, identities and role-based access.
zone             Manage access zones.
Data Protection:
avscan*          Manage antivirus service settings.
ndmp*            Manage backup (NDMP) settings.
snapshot         Manage SnapshotIQ file system snapshots and settings.
sync             SyncIQ management interface.
Protocols:
ftp*             Manage FTP settings.
hdfs*            Manage HDFS settings.
iscsi*           Manage iSCSI settings.
nfs              Manage NFS exports and protocol settings.
smb              Manage SMB shares and protocol settings.
snmp*            Manage SNMP settings.
Utilities:
exttools*        External tools.
Other Subcommands:
filepool         Manage filepools on the cluster.
storagepool      Configure and monitor storage pools
Options:
Display Options:
--timeout <integer>
Number of seconds for a command timeout.
--help | -h
Display help for this command.

Noe that subcommands or actions that are marked with an asterisk(*) require root login.  If you log in with a normal admin account you’ll get the following error message when you run the command:

isilon01-1% isi status
Commands not enabled for role-based administration require root user access.

Isilon’s OneFS operating system is based on FreeBSD as opposed to Linux for the Celerra/VNX DART OS.   Since it’s a unix based OS with console access, you can create shell scripts and cron job schedules just like the Celerra/VNX File.

Because all of the commands I want to run require root access, I had to create the scripts logged in as root.  Be careful doing this!  This is only a test for me, I would likely look for a workaround for a prod system.    Knowing that I’m going to be FTPing the output files to my web report server, I started by creating the .netrc file in the /root folder.  This is where I store the default login and password for the FTP server. Permissions must be changed for it to work, use chmod 600 on the file after you create it.  It didn’t work for me at first as the syntax is different on FreeBSD than on Linux, so looking at my Celerra notes didn’t help  (For Celerra/VNX File I used “machine login password ”).

For Isilon, the correct syntax is this:

default login <ftp_username> password <ftp_password>
 

The script that FTP’s the files would then look like this for Isilon:

HOST=”10.1.1.1″
ftp $HOST <<SCRIPT
put /root/isilon_stats.txt
put /root/isilon_repl.txt
put /root/isilon_df.txt
put /root/isilon_dedupe.txt
put /root/isilon_perf.txt
SCRIPT
 

For this demo, I created a script that generates reports for File system utilization, Deduplication Status, Performance Stats, Replication stats, and Array Status.  For the Filesystem utilization report, I used two different methods as I wasn’t sure which I’d like better.   Using ‘df –h –a /ifs’ will get you similar information to ‘isi storagepool list’, but the output format is different.   I used cron to schedule the job directly on the Isilon.

Here is the reporting script:

TODAY=$(date)
HOST=$(hostname)
sleep 15
echo “———————————————————————————” > /root/isilon_stats.txt
echo “Date: $TODAY  Host:  $HOST” >> /root/isilon_stats.txt
echo “———————————————————————————” >> /root/isilon_stats.txt
/usr/bin/isi status >> /root/isilon_stats.txt
echo “———————————————————————————” > /root/isilon_repl.txt
echo “Date: $TODAY  Host:  $HOST” >> /root/isilon_repl.txt
echo “———————————————————————————” >> /root/isilon_repl.txt
/usr/bin/isi sync reports list >> /root/isilon_repl.txt
echo “———————————————————————————” > /root/isilon_df.txt
echo “Date: $TODAY  Host:  $HOST” >> /root/isilon_df.txt
echo “———————————————————————————” >> /root/isilon_df.txt
df -h -a /ifs >> /root/isilon_df.txt
echo ” ” >> /root/isilon_df.txt
/usr/bin/isi storagepool list >> /root/isilon_df.txt
echo “———————————————————————————” > /root/isilon_dedupe.txt
echo “Date: $TODAY  Host:  $HOST” >> /root/isilon_dedupe.txt
echo “———————————————————————————” >> /root/isilon_dedupe.txt
/usr/bin/isi dedupe stats  >> /root/isilon_dedupe.txt
echo “———————————————————————————” > /root/isilon_perf.txt
echo “Date: $TODAY  Host:  $HOST” >> /root/isilon_perf.txt
echo “———————————————————————————” >> /root/isilon_perf.txt
sleep 1
echo ”  ” >> /root/isilon_perf.txt
echo “–System Stats–” >> /root/isilon_perf.txt
echo ”  ” >> /root/isilon_perf.txt
/usr/bin/isi statistics system  >> /root/isilon_perf.txt
sleep 1
echo ”  ” >> /root/isilon_perf.txt
echo “–Client Stats–” >> /root/isilon_perf.txt
echo ”  ” >> /root/isilon_perf.txt
/usr/bin/isi statistics client  >> /root/isilon_perf.txt
sleep 1
echo ”  ” >> /root/isilon_perf.txt
echo “–Protocol Stats–” >> /root/isilon_perf.txt
echo ”  ” >> /root/isilon_perf.txt
/usr/bin/isi statistics protocol  >> /root/isilon_perf.txt
sleep 1
echo ”  ” >> /root/isilon_perf.txt
echo “–Protocol Data–” >> /root/isilon_perf.txt
echo ”  ” >> /root/isilon_perf.txt
/usr/bin/isi statistics pstat  >> /root/isilon_perf.txt
sleep 1
echo ”  ” >> /root/isilon_perf.txt
echo “–Drive Stats–” >> /root/isilon_perf.txt
echo ”  ” >> /root/isilon_perf.txt
/usr/bin/isi statistics drive  >> /root/isilon_perf.txt
 

Once the ouput files are FTP’d to the web server, I have a basic HTML page that uses iframes to show the text files.  The web page is then automatically updated as soon as the new text files are FTP’d.  Below is a screenshot of my demo report page.  It doesn’t show the entire page, but you’ll get the idea.

IsilonReports