Category Archives: scripting-vnx

Scripting a VNX/Celerra to Isilon Data Migration with EMCOPY and Perl

datamigration

Below is a collection of perl scripts that make data migration from VNX/Celerra file systems to an Isilon system much easier.  I’ve already outlined the process of using isi_vol_copy_vnx in a prior post, however using EMCOPY may be more appropriate in a specific use case, or simply more familiar to administrators for support and management of the tool.  Note that while I have tested these scripts in my environment, they may need some modification for your use.  I recommend running them in a test environment prior to using them in production.

EMCOPY can be downloaded directly from DellEMC with the link below.  You will need to be a registered user in order to download it.

https://download.emc.com/downloads/DL14101_EMCOPY_File_migration_tool_4.17.exe

What is EMCOPY?

For those that haven’t used it before, EMCOPY is an application that allows you to copy a file, directory, and subdirectories between NTFS partitions while maintaining security information, an improvement over the similar robocopy tool that many veteran system administrators are familiar with. It allows you to back up the file and directory security ACLs, owner information, and audit information from a source directory to a destination directory.

Notes about using EMCOPY:

1) In my testing, EMCopy has shown up to a 25% performance improvement when copying CIFS data compared to Robocopy while using the same number of threads. I recommend using EMCopy over Robocopy as it has other feature improvements as well, for instance sidmapfile, which allows migrating local user data to Active Directory users. It’s available in version 4.17 or later.  Robocopy is also not an EMC supported tool, while EMCOPY is.

2) Unlike isi_vol_copy_vnx, EMCOPY is a windows application and must be run from a windows host.  I highly recommend a dedicated server for any migration tasks.  The isi_vol_copy_vnx utility runs directly on the Isilon OneFS CLI which eliminates any intermediary copy hosts, theoretically providing a much faster solution.

3) There are multiple methods to compare data sizes between the source and destination. I would recommend maintaining a log of each EMCopy session as that log indicates how much data was copied and if there were any errors.

4) If you are migrating over a WAN connection, I recommend first restoring from tape and then using an incremental data sync with EMCOPY.

Getting Started

I’ve divided this post up into a four step process.  Each step includes the relevant script and a description of the process.

  • Export File System information (export_fs.pl  Script)

Export file system information from the Celerra & generate the Isilon commands to re-create them.

  • Export SMB information (export_smb.pl Script)

Export SMB share information from the Celerra & generate the Isilon commands to re-create them.

  • Export NFS information (export_nfs.pl Script)

Export NFS information from the Celerra & generate the Isilon commands to re-create them.

  • Create the EMCOPY migration script (EMCOPY_create.pl Script)

Perform the data migration with EMCOPY using the output from this script.

Exporting information from the Celerra to run on the Isilon

These Perl scripts are designed to be run directly on the Control Station and will subsequently create shell scripts that will run on the Isilon to assist with the migration.  You will need to manually copy the output files from the VNX/Celerra to the Isilon. The first three steps I’ve outlined do not move the data or permissions, they simply run a nas_fs query on the Celerra to generate the Isilon script files that actually make the directories, create quotas, and create the NFS and SMB shares. They are “scripts that generate scripts”. 🙂

Before you run the scripts, make sure you edit them to correctly specify the appropriate Data Mover.  Once complete, You’ll end up with three .sh files created for you to move to your Isilon cluster.  They should be run in the same order as they were created.

Note that EMC occasionally changes the syntax of certain commands when they update OneFS.  Below is a sample of the isilon specific commands that are generated by the first three scripts.  I’d recommend verifying that the syntax is still correct with your version of OneFS, and then modify the scripts if necessary with the new syntax.  I just ran a quick test with OneFS 8.0.0.2, and the base commands and switches appear to be compatible.

isi quota create –directory –path=”/ifs/data1″ –enforcement –hard-threshold=”1032575M” –container=1
isi smb share create –name=”Data01″ –path=”/ifs/Data01/data”
isi nfs exports create –path=”/Data01/data”  –roclient=”Data” –rwclient=”Data” –rootclient=”Data”

 

Step 1 – Export File system information

This script will generate a list of the file system names from the Celerra and place the appropriate Isilon commands that create the directories and quotes into a file named “create_filesystems_xx.sh”.

#!/usr/bin/perl

# Export_fs.pl – Export File system information
# Export file system information from the Celerra & generate the Isilon commands to re-create them.

use strict;
my $nas_fs="nas_fs -query:inuse=y:type=uxfs:isroot=false -fields:ServersNumeric,Id,Name,SizeValues -format:'%s,%s,%s,%sQQQQQQ'";
my @data;

open (OUTPUT, ">> create_filesystems_$$.sh") || die "cannot open output: $!\n\n";
open (CMD, "$nas_fs |") || die "cannot open $nas_fs: $!\n\n";

while ()

{
   chomp;
   @data = split("QQQQQQ", $_);
}

close(CMD);
foreach (@data)

{
   my ($dm, $id, $dir,$size,$free,$used_per, $inodes) = split(",", $_);
   print OUTPUT "mkdir /ifs/$dir\n";
   print OUTPUT "chmod 755 /ifs/$dir\n";
   print OUTPUT "isi quota create --directory --path=\"/ifs/$dir\" --enforcement --hard-threshold=\"${size}M\" --container=1\n";
}

The Output of the script looks like this (this is an excerpt from the create_filesystems_xx.sh file):

isi quota create --directory --path="/ifs/data1" --enforcement --hard-threshold="1032575M" --container=1
mkdir /ifs/data1
chmod 755 /ifs/data1
isi quota create --directory --path="/ifs/data2" --enforcement --hard-threshold="20104M" --container=1
mkdir /ifs/data2
chmod 755 /ifs/data2
isi quota create --directory --path="/ifs/data3" --enforcement --hard-threshold="100774M" --container=1
mkdir /ifs/data3
chmod 755 /ifs/data3

The output script can now be copied to and run from the Isilon.

Step 2 – Export SMB Information

This script will generate a list of the smb share names from the Celerra and place the appropriate Isilon commands into a file named “create_smb_exports_xx.sh”.

#!/usr/bin/perl

# Export_smb.pl – Export SMB/CIFS information
# Export SMB share information from the Celerra & generate the Isilon commands to re-create them.

use strict;

my $datamover = "server_8";
my $prot = "cifs";:wq!
my $nfs_cli = "server_export $datamover -list -P $prot -v |grep share";

open (OUTPUT, ">> create_smb_exports_$$.sh") || die "cannot open output: $!\n\n";
open (CMD, "$nfs_cli |") || die "cant open $nfs_cli: $!\n\n";

while ()
{
   chomp;
   my (@vars) = split(" ", $_);
   my $path = $vars[2];
   my $name = $vars[1];

   $path =~ s/^"/\"\/ifs/;
   print  OUTPUT "isi smb share create --name=$name --path=$path\n";
}

close(CMD);

The Output of the script looks like this (this is an excerpt from the create_smb_exports_xx.sh file):

isi smb share create --name="Data01" --path="/ifs/Data01/data"
isi smb share create --name="Data02" --path="/ifs/Data02/data"
isi smb share create --name="Data03" --path="/ifs/Data03/data"
isi smb share create --name="Data04" --path="/ifs/Data04/data"
isi smb share create --name="Data05" --path="/ifs/Data05/data"

 The output script can now be copied to and run from the Isilon.

Step 3 – Export NFS Information

This script will generate a list of the NFS export names from the Celerra and place the appropriate Isilon commands into a file named “create_nfs_exports_xx.sh”.

#!/usr/bin/perl

# Export_nfs.pl – Export NFS information
# Export NFS information from the Celerra & generate the Isilon commands to re-create them.

use strict;

my $datamover = "server_8";
my $prot = "nfs";
my $nfs_cli = "server_export $datamover -list -P $prot -v |grep export";

open (OUTPUT, ">> create_nfs_exports_$$.sh") || die "cannot open output: $!\n\n";
open (CMD, "$nfs_cli |") || die "cant open $nfs_cli: $!\n\n";

while ()
{
   chomp;
   my (@vars) = split(" ", $_);
   my $test = @vars;
   my $i=2;
   my ($ro, $rw, $root, $access, $name);
   my $path=$vars[1];

   for ($i; $i < $test; $i++)
   {
      my ($type, $value) = split("=", $vars[$i]);

      if ($type eq "ro") {
         my @tmp = split(":", $value);
         foreach(@tmp) { $ro .= " --roclient=\"$_\""; }
      }
      if ($type eq "rw") {
         my @tmp = split(":", $value);
         foreach(@tmp) { $rw .= " --rwclient=\"$_\""; }
      }

      if ($type eq "root") {
         my @tmp = split(":", $value);
         foreach(@tmp) { $root .= " --rootclient=\"$_\""; }
      }

      if ($type eq "access") {
         my @tmp = split(":", $value);
         foreach(@tmp) { $ro .= " --roclient=\"$_\""; }
      }

      if ($type eq "name") { $name=$value; }
   }
   print OUTPUT "isi nfs exports create --path=$path $ro $rw $root\n";
}

close(CMD);

The Output of the script looks like this (this is an excerpt from the create_nfs_exports_xx.sh file):

isi nfs exports create --path="/Data01/data" --roclient="Data" --roclient="BACKUP" --rwclient="Data" --rwclient="BACKUP" --rootclient="Data" --rootclient="BACKUP"
isi nfs exports create --path="/Data02/data" --roclient="Data" --roclient="BACKUP" --rwclient="Data" --rwclient="BACKUP" --rootclient="Data" --rootclient="BACKUP"
isi nfs exports create --path="/Data03/data" --roclient="Backup" --roclient="Data" --rwclient="Backup" --rwclient="Data" --rootclient="Backup" --rootclient="Data"
isi nfs exports create --path="/Data04/data" --roclient="Backup" --roclient="ProdGroup" --rwclient="Backup" --rwclient="ProdGroup" --rootclient="Backup" --rootclient="ProdGroup"
isi nfs exports create --path="/" --roclient="127.0.0.1" --roclient="127.0.0.1" --roclient="127.0.0.1" -rootclient="127.0.0.1"

The output script can now be copied to and run from the Isilon.

Step 4 – Generate the EMCOPY commands

Now that the scripts have been generated and run on the Isilon, the next step is the actual data migration using EMCOPY.  This script will generate the commands for a migration script, which should be run from a windows server that has access to both the source and destination locations. It should be run after the previous three scripts have successfully completed.

This script will output the commands directly to the screen, it can then be cut and pasted from the screen directly into a windows batch script on your migration server.

#!/usr/bin/perl

# EMCOPY_create.pl – Create the EMCOPY migration script
# Perform the data migration with EMCOPY using the output from this script.

use strict;

my $datamover = "server_4";
my $source = "\\\\celerra_path\\";
my $dest = "\\\\isilon_path\\";
my $prot = "cifs";
my $nfs_cli = "server_export $datamover -list -P $prot -v |grep share";

open (OUTPUT, ">> create_smb_exports_$$.sh") || die "cant open output: $!\n\n";
open (CMD, "$nfs_cli |") || die "cant open $nfs_cli: $!\n\n";

while ()
{
   chomp;
   my (@vars) = split(" ", $_);
   my $path = $vars[2];
   my $name = $vars[1];

   $name =~ s/\"//g;
   $path =~ s/^/\/ifs/;

   my $log = "c:\\" . $name . "";
   $log =~ s/ //;
   my $src = $source . $name;
   my $dst = $dest . $name;

   print "emcopy \"$src\" \"$  dst\" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:$log\n";
}

close(CMD);

The Output of the script looks like this (this is an excerpt from the screen output):

emcopy "\\celerra_path\Data01" "\\isilon_path\billing_tmip_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_tmip_01
emcopy "\\celerra_path\Data02" "\\isilon_path\billing_trxs_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_trxs_01
emcopy "\\celerra_path\Data03" "\\isilon_path\billing_vru_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_vru_01
emcopy "\\celerra_path\Data04" "\\isilon_path\billing_rpps_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_rpps_01

That’s it.  Good luck with your data migration, and I hope this has been of some assistance.  Special thanks to Mark May and his virtualstoragezone blog, he published the original versions of these scripts here.

Advertisements

Scripting an alert to verify the availability of individual VNX CIFS server shares

It was recently asked to come up with a method to alert on the availability of specific CIFS file shares in our environment.  This was due to a recent issue we had on our VNX with our data mover crashing and causing the corruption of a single file system when it came back up.  We were unaware for several hours of the one file system being unavailable on our CIFS server.

This particular script would require maintenance whenever a new file system share is added to a CIFS server.  A unique line must to be added for every file system share that you have configured.  If a file system is not mounted and the share is inaccessible, an email alert will be sent.  If the share is accessible the script does nothing when run from the scheduler.  If it’s run manually from the CLI, it will echo back to the screen that the path is active.

This is a bash shell script, I run it on a windows server with Cygwin installed using the ‘email’ package for SMTP.  It should also run fine from a linux server, and you could substitute the ‘email’ syntax for sendmail or whatever other mail application you use.   I have it scheduled to check the availability of CIFS shares every one hour.

DIR1=file_system_1; SRV1=cifs_servername;  echo -ne $DIR1 && echo -ne ": " && [ -d //$SRV1/$DIR1 ] && echo "Network Path is Active" || email -b -s "Network Path \\\\$SRV1\\$DIR1 is offline" emailaddress@email.com

DIR2=file_system_2; SRV1=cifs_servername;  echo -ne $DIR1 && echo -ne ": " && [ -d //$SRV1/$DIR2 ] && echo "Network Path is Active" || email -b -s "Network Path \\\\$SRV1\\$DIR2 is offline" emailaddress@email.com

DIR3=file_system_3; SRV1=cifs_servername;  echo -ne $DIR1 && echo -ne ": " && [ -d //$SRV1/$DIR3 ] && echo "Network Path is Active" || email -b -s "Network Path \\\\$SRV1\\$DIR3 is offline" emailaddress@email.com

DIR4=file_system_4; SRV1=cifs_servername;  echo -ne $DIR1 && echo -ne ": " && [ -d //$SRV1/$DIR4 ] && echo "Network Path is Active" || email -b -s "Network Path \\\\$SRV1\\$DIR4 is offline" emailaddress@email.com

DIR5=file_system_5; SRV1=cifs_servername;  echo -ne $DIR1 && echo -ne ": " && [ -d //$SRV1/$DIR5 ] && echo "Network Path is Active" || email -b -s "Network Path \\\\$SRV1\\$DIR5 is offline" emailaddress@email.com

Gathering performance data on a virtual windows server

When troubleshooting a potential storage related performance problem on a virtual windows server, it’s a bit more difficult to anaylze a because many virtual hosts share the same LUN for a datastore in ESX.  Using EMC’s analyzer or Control Center Performance Manager only gives me statistics on specific disks or LUNs, I have no visibility into a specific virtual server with those tools.  When this situation arises, I use a windows batch script to gather data with the typeperf command line utility for a specific time period and run it directly on the server.  Typically I’ll let it run for 24 hours and then analyze the data in Excel, where it’s easy to make charts and graphs to get a visual view of what’s going on.

Sometimes the most difficult thing to figure out is the correct syntax for the command and which parameters to use.  For reference, here is the command and it’s parameters:

Syntax:

Typeperf [Path [path ...]] [-cf FileName] [-f {csv|tsv|bin}] [-si interval] [-o FileName] [-q [object]] [-qx [object]] [-sc samples] [-config FileName] [-s computer_name]

Parameters:

-c { Path [ path ... ] | -cf   FileName } : Specifies the performance counter path to log. To list multiple counter paths, separate each command path by a space.
 -cf FileName : Specifies the file name of the file that contains the counter paths that you want to monitor, one per line.
 -f { csv | tsv | bin } : Specifies the output file format. File formats are csv (comma-delimited), tsv (tab-delimited), and bin (binary). Default format is csv.
 -si interval [ mm: ] ss   : Specifies the time between samples, in the [mm:] ss format. Default is one second.
 -o FileName   : Specifies the pathname of the output file. Defaults to stdout.
 -q [ object ] : Displays and queries available counters without instances. To display counters for one object, include the object name.
 -qx [ object ] : Displays and queries all available counters with instances. To display counters for one object, include the object name.
 -sc samples : Specifies the number of samples to collect. Default is to sample until you press CTRL+C.
 -config FileName : Specifies the pathname of the settings file that contains command line parameters.
 -s computer_name : Specifies the system to monitor if no server is specified in the counter path.
 /? : Displays help at the command prompt.

EMC’s Analyzer vs. Windows Perfmon Metrics

I tend to look at Response time, disk queue length, Total/Read/Write IO, and Service time first.   I dive into how to interpret many of the SAN performance metrics in my older post here. 

The counters you’ll choose in Windows performance monitor don’t precisely line up with what we commonly look at using EMC’s tools in how they are named, and in addition you can choose ‘LogicalDisk’ and ‘PhysicalDisk’ when selecting the counters.

What is the difference between the Physical Disk vs. Logical Disk performance objects in Perfmon, and why monitor both? Their counters are calculated the same way but their scope is different. I generally use both “\LogicalDisk(*)\” and “\PhysicalDisk(*)\” when I run my perfmon script.

The Physical Disk performance object monitors disk drives on the computer. It identifies the instances representing the physical hardware, and the counters are the sum of the access to all partitions on the physical instance.

The Logical Disk Performance object monitors logical partitions. Performance monitor identifies logical disks by their drive letter or mount point. If a physical disk contains multiple partitions, this counter will report the values just for the partition selected and not for the entire disk. On the other hand, when using Dynamic Disks the logical volumes may span more than one physical disk, in this scenario the counter values will include the access to the logical disk in all the physical disks it spans.

Here are the performance monitor counters that I frequently use, and how they compare to EMC’s navisphere analyzer (or ECC):

“\LogicalDisk(*)\Avg. Disk Queue Length” – (Named the same as EMC) The average number of outstanding requests when the disk was busy
“\LogicalDisk(*)\%% Disk Time” – (No direct EMC equivalent) The “% Disk Time” counter is the “Avg. Disk Queue Length” counter multiplied by 100. It is the same value displayed in a different scale.
“\LogicalDisk(*)\Disk Transfers/sec” – Total Throughput (IO/sec) – the total number of individual disk IO requests completed over a period of one second.  We’ll use this value to help determine Disk Service Time.
“\LogicalDisk(*)\Disk Reads/sec” – Read Throughput (IO/sec)
“\LogicalDisk(*)\Disk Writes/sec” – Write Throughput (IO/sec)
“\LogicalDisk(*)\%% Idle Time” –  (No direct EMC equivalent) This counter provides a very precise measurement of how much time the disk remained in idle state, meaning all the requests from the operating system to the disk have been completed and there are zero pending requests. We’ll also use this to calculate disk service time.
“\LogicalDisk(*)\Avg. Disk sec/Transfer” – Response time (sec) – EMC uses milliseconds, windows uses seconds, so you’ll see 8ms represented as .008 in the results.
“\LogicalDisk(*)\Avg. Disk sec/Read” – Response times for read IO
“\LogicalDisk(*)\Avg. Disk sec/Write” – Response times for write IO

Disk Service Time is caculated with this formula:  Disk Utilization = 100 – %Idle Time, then Disk Utilization  /  Disk Transfers/Sec. = Disk Service Time.

Configuring the Script

This batch script collects all of the relevant data for disk activity.  After 24 hours, it will dump the data into a csv file.  The length of time is controller by the combination of the “-sc” and “-si” parameters.  To collect data in one minute intervals for 24 hours, you’d set si to 60 (collect data every 60 seconds), and sc to 1440 (1440 minutes = 24 hours). To collect data every one minute for 30 minutes, you’d enter “-si 60 -sc 30”.  This script assumes you have a local directory on the C: Drive named ‘Collection’.

@echo off
cd c:\collection

@for /f "tokens=1,2,3,4 delims=/ " %%A in ('date /t') do @(set all=%%A%%B%%C%%D)
@for /f "tokens=1,2,3 delims=: " %%A in ('time /t') do @(set allm=%%A%%B%%C)

typeperf “\LogicalDisk(*)\Avg. Disk Queue Length” “\LogicalDisk(*)\%% Disk Time” “\LogicalDisk(*)\Disk 
Transfers/sec” “\LogicalDisk(*)\Disk Reads/sec” “\LogicalDisk(*)\Disk Writes/sec” “\LogicalDisk(*)\%% Idle Time” 
“\LogicalDisk(*)\Avg. Disk sec/Transfer” “\LogicalDisk(*)\Avg. Disk sec/Read” “\LogicalDisk(*)\Avg. Disk sec/Write” 
“\PhysicalDisk(*)\Avg. Disk Queue Length” “\PhysicalDisk(*)\%% Disk Time” “\PhysicalDisk(*)\Disk Transfers/sec” “\PhysicalDisk(*)\Disk Reads/sec” “\PhysicalDisk(*)\Disk Writes/sec” \PhysicalDisk(*)\%% Idle Time” “\PhysicalDisk(*)\Avg. Disk sec/Transfer” “\PhysicalDisk(*)\Avg. Disk sec/Read” "\PhysicalDisk(*)\Avg. Disk sec/Write” -si 60 -sc 1440 -o PerfCounters-%All%-%Allm%.csv
 

Alerting on VNX File SP Failover

You may see a Celerra alert description that states “Disk dx has been trespassed” or “Storage Processor is ready to restore”.  This would mean that one or more Celerra LUNs aren’t being accessed through the default SP.  It can happen during a DART upgrade, some other maintenance activity, or simply a temporary loss of connectivity to the default SP for the LUN.  I wanted to set up an email alert to let the storage team know when an SP failover occurs, so I wrote a script to send an alert email when a failover is detected.  It can be run directly on the data mover and scheduled as frequently as you like via cron.

Here’s the script:

#!/bin/bash

TODAY=$(date) 
HOST=$(hostname) 

STATUS=`cat /scripts/health/arrayFOstatus.txt`

# Checks and displays status of NAS storage, will show SP/disk Failover info. 
# We will use this info to include in the alert email if needed. 

/nas/bin/nas_storage -check -all > /scripts/health/backendcheck.txt 

#  [nasadmin@celerra]$ nas_storage -check -all     
#  Discovering storage (may take several minutes) 
#  Error 5017: storage health check failed #  CK900052700319 SPA is failed over #  CK900052700319 d6 is failed over

# Shows detailed info, I'm only pulling out failover info. 

/nas/bin/nas_storage -info -all | grep failed_over > /scripts/health/failovercheck.txt 

# The command above results in this output: 
#   failed_over = <True/False> 
#   failed_over = <True/False> 
#   failed_over = <True/False> 

# The first entry is the value for the array, second is SPA, third is SPB. 

# The next line pulls the True/False value for the entire array (the third value on the first line of output) 

echo `cat /scripts/health/failovercheck.txt | awk '{if (NR<2) print $3}'` > /scripts/health/arrayFOstatus.txt

# Now we check the value in the 'arrayFOstatus.txt' file, if it's 'True', we send an email notification that there is an SP failed over. 

# In addition to sending an email, you could also run the 'nas_storage -failback id=1' command to automatically fail it back.

if [ "$STATUS" == "False" ]; then  
   echo "Value is False" 
fi

if [ "$STATUS" == "True" ]; then  
   mail -s "SP Failover on $HOST" username@domain.com < /scripts/health/backendcheck.txt  

   #nas_storage -failback id=1 #Optionally fail it back, our team decided to alert only and fail back manually.

   echo "Value is True" 
fi

If a failover is detected, you can manually fail it back with the following commands:

Determine/Confirm the ID number:

[nasadmin@celerra]$ nas_storage -list
 id   acl    name                     serial_number
 1    0      CK900052700319 CK900052700319

Fail it back (will fail back Celerra/VNX File LUNs only):

[nasadmin@celerra]$ nas_storage -failback id=1
id  = 1  
serial_number   = CCK900052700319  
name  = CCK900052700319  
acl  = 0  
done

Archiving NAZ and NAR files from EMC VNX and Clariion arrays

It can be useful to copy and archive naz and nar files from all arrays to a local server.  It’s useful for helping EMC with troubleshooting efforts, general health checks, and researching historical trends.   I use them often with our EMC technical rep when a workload analysis is done, and it’s much faster to simply have them all copied somewhere automatically on a daily basis.

Not all of our arrays have an analyzer license, so the files are stored in “naz” format rather than “nar” format.  The naz files need to be sent to emc for decryption before they can be used by a customer.

The windows shell script below will store the current date in a variable, attempt to start analyzer and then pull the current file.  Arrays that don’t have an analyzer license will only run data collection for a maximum of 7 days.  The script attempts to start the service every day, so if it happens to have been 7 days it will start back up.  I set the archive interval to 600 seconds and run the script every 24 hours.

 

@ECHO OFF
 
For /f “tokens=2-4 delims=/ ” %%a in (‘date /t’) do (set date=%%a-%%b-%%c)
For /f “tokens=1-3 delims=: ” %%a in (‘time /t’) do (set time=%%a-%%b-%%c)
for /f “tokens=1-7 delims=:/-, ” %%i in (‘echo exit^|cmd /q /k”prompt $d $t”‘) do (
   for /f “tokens=2-4 delims=/-,() skip=1” %%a in (‘echo.^|date’) do (
      set dow=%%i
      set %%a=%%j
      set %%b=%%k
      set %%c=%%l
      set hh=%%m
      set min=%%n
      set ss=%%o
   )
)
 
echo Array01a
naviseccli -h Array01a analyzer -start
echo Array02a
naviseccli -h Array02a analyzer -start
echo Array03a
naviseccli -h Array03a analyzer -start
echo Array04a
naviseccli -h Array04a analyzer -start
echo Array05a
naviseccli -h Array05a analyzer -start
echo Array06a
naviseccli -h Array05a analyzer -start
 
NaviSECCli.exe -h Array01a analyzer -archiveretrieve -file APM00111100006_SPA_%date%-%time%.naz -Location D:\SAN\narcollection\Array01
NaviSECCli.exe -h Array01b analyzer -archiveretrieve -file APM00111100006_SPB_%date%-%time%.naz -Location D:\SAN\narcollection\Array01
 
NaviSECCli.exe -h Array02a analyzer -archiveretrieve -file APM00111000005_SPA_%date%-%time%.naz -Location D:\SAN\narcollection\Array02
NaviSECCli.exe -h Array02b analyzer -archiveretrieve -file APM00111000005_SPB_%date%-%time%.naz -Location D:\SAN\narcollection\Array02
 
NaviSECCli.exe -h Array03a analyzer -archiveretrieve -file APM00182700004_SPA_%date%-%time%.nar -Location D:\SAN\narcollection\Array03
NaviSECCli.exe -h Array03b analyzer -archiveretrieve -file APM00182700004_SPB_%date%-%time%.nar -Location D:\SAN\narcollection\Array03
 
NaviSECCli.exe -h Array04a analyzer -archiveretrieve -file APM00122600000_SPA_%date%-%time%.naz -Location D:\SAN\narcollection\Array04
NaviSECCli.exe -h Array04b analyzer -archiveretrieve -file APM00122600000_SPB_%date%-%time%.naz -Location D:\SAN\narcollection\Array04
 
NaviSECCli.exe -h Array05a analyzer -archiveretrieve -file APM00122700001_SPA_%date%-%time%.nar -Location D:\SAN\narcollection\Array05
NaviSECCli.exe -h Array05b analyzer -archiveretrieve -file APM00122700001_SPB_%date%-%time%.nar -Location D:\SAN\narcollection\Array05
 
NaviSECCli.exe -h Array06a analyzer -archiveretrieve -file APM00132900002_SPA_%date%-%time%.naz -Location D:\SAN\narcollection\Array06
NaviSECCli.exe -h Array06b analyzer -archiveretrieve -file APM00132900003_SPB_%date%-%time%.naz -Location D:\SAN\narcollection\Array06

 

Reporting on the Estimated Job Completion Times for FAST VP Data Relocation

Another one of the many daily reports I run reports on the current time remaining on the FAST VP data relocation times for all of our arrays.  I also make a single backup copy of the report to show the times for the previous day so I can get a quick view of progress that was made over the previous 24 hours.  Both reports are presented side by side on my intranet report page for easy comparison.

I made a post last year regarding how to deal with long running FAST VP data relocation jobs (http://emcsan.wordpress.com/2012/01/18/long-running-fast-vp-relocation-job/), and this report helps identify any arrays that could be falling behind.  If your estimated completion time is longer than the time window you have defined for your data relocation job you may need to make some changes, see my previous post for more information about that.

You can get the current status of the data relocation job at any time by running the following command:

naviseccli -h [array_hostname] autotiering -info -state -rate -schedule -opStatus -poolID [Pool_ID_Number]
 

The output looks like this:

Auto-Tiering State:  Enabled
Relocation Rate:  Medium
 
Schedule Name:  Default Schedule
Schedule State:  Enabled
Default Schedule:  Yes
Schedule Days:  Sun Mon Tue Wed Thu Fri Sat
Schedule Start Time:  20:00
Schedule Stop Time:  6:00
Schedule Duration:  10 hours
Storage Pools:  Array1_Pool1_SPB, Array1_Pool0_SPA
 
Storage Pool Name:  Array1_Pool0_SPA
Storage Pool ID:  0
Relocation Start Time:  08/15/13 20:00
Relocation Stop Time:  08/16/13 6:00
Relocation Status:  Inactive
Relocation Type:  Scheduled
Relocation Rate:  Medium
Data to Move Up (GBs):  8.00
Data to Move Down (GBs):  8.00
Data Movement Completed (GBs):  2171.00
Estimated Time to Complete:  4 minutes
Schedule Duration Remaining:  None
 
Storage Pool Name:  Array1_Pool1_SPB
Storage Pool ID:  1
Relocation Start Time:  08/15/13 20:00
Relocation Stop Time:  08/16/13 6:00
Relocation Status:  Inactive
Relocation Type:  Scheduled
Relocation Rate:  Medium
Data to Move Up (GBs):  14.00
Data to Move Down (GBs):  14.00
Data Movement Completed (GBs):  1797.00
Estimated Time to Complete:  5 minutes
Schedule Duration Remaining:  None
 

The output of the command is very verbose, I want to trim it down to only show me the pool name and the estimated time for the relocation job to complete.   This bash script will trim it down to only show the pool names and estimated completion times.

The final output of the script generated report looks like this: 

Runtime: Thu Aug 11 07:00:01 CDT 2013
Array1_Pool0:  9 minutes
Array1_Pool1:  6 minutes
Array2_Pool0:  1 hour, 47 minutes
Array2_Pool1:  3 minutes
Array2_Pool2:  2 days, 7 hours, 25 minutes
Array2_Pool3:  1 day, 9 hours, 58 minutes
Array3_Pool0:  1 minute
Array4_Pool0:  N/A
Array4_Pool1:  2 minutes
Array5_Pool1:  5 minutes
Array5_Pool0:  5 minutes
Array6_Pool0:  N/A
Array6_Pool1:  N/A

 

Below is the bash script that generates the report. The script is set up to report on six different arrays, it can be easily modified to suit your environment. 

TODAY=$(date)
echo “Runtime: $TODAY” > /reports/tierstatus.txt
echo $TODAY
#
naviseccli -h [array_hostname1] autotiering -info -state -rate -schedule -opStatus -poolID 0 > /reports/[array_hostname1]_tierstatus0.out
naviseccli -h [array_hostname1] autotiering -info -state -rate -schedule -opStatus -poolID 1 > /reports/[array_hostname1]_tierstatus1.out
#
echo `grep “Pool Name:” /reports/[array_hostname1]_tierstatus0.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname1]_tierstatus0.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
echo `grep “Pool Name:” /reports/[array_hostname1]_tierstatus1.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname1]_tierstatus1.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
#
naviseccli -h [array_hostname2] autotiering -info -state -rate -schedule -opStatus -poolID 0 > /reports/[array_hostname2]_tierstatus0.out
naviseccli -h [array_hostname2] autotiering -info -state -rate -schedule -opStatus -poolID 1 > /reports/[array_hostname2]_tierstatus1.out
naviseccli -h [array_hostname2] autotiering -info -state -rate -schedule -opStatus -poolID 2 > /reports/[array_hostname2]_tierstatus2.out
naviseccli -h [array_hostname2] autotiering -info -state -rate -schedule -opStatus -poolID 3 > /reports/[array_hostname2]_tierstatus3.out
#
echo `grep “Pool Name:” /reports/[array_hostname2]_tierstatus0.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname2]_tierstatus0.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
echo `grep “Pool Name:” /reports/[array_hostname2]_tierstatus1.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname2]_tierstatus1.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
echo `grep “Pool Name:” /reports/[array_hostname2]_tierstatus2.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname2]_tierstatus2.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
echo `grep “Pool Name:” /reports/[array_hostname2]_tierstatus3.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname2]_tierstatus3.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
#
naviseccli -h [array_hostname3] autotiering -info -state -rate -schedule -opStatus -poolID 0 > /reports/[array_hostname3]_tierstatus0.out
#
echo `grep “Pool Name:” /reports/[array_hostname3]_tierstatus0.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname3]_tierstatus0.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
#
naviseccli -h [array_hostname4] autotiering -info -state -rate -schedule -opStatus -poolID 0 > /reports/[array_hostname4]_tierstatus0.out
naviseccli -h [array_hostname4] autotiering -info -state -rate -schedule -opStatus -poolID 1 > /reports/[array_hostname4]_tierstatus1.out
#
echo `grep “Pool Name:” /reports/[array_hostname4]_tierstatus0.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname4]_tierstatus0.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
echo `grep “Pool Name:” /reports/[array_hostname4]_tierstatus1.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname4]_tierstatus1.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
#
naviseccli -h [array_hostname5] autotiering -info -state -rate -schedule -opStatus -poolID 0 > /reports/[array_hostname5]_tierstatus0.out
naviseccli -h [array_hostname5] autotiering -info -state -rate -schedule -opStatus -poolID 1 > /reports/[array_hostname5]_tierstatus1.out
#
echo `grep “Pool Name:” /reports/[array_hostname5]_tierstatus0.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname5]_tierstatus0.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
echo `grep “Pool Name:” /reports/[array_hostname5]_tierstatus1.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname5]_tierstatus1.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
#
naviseccli -h [array_hostname6] autotiering -info -state -rate -schedule -opStatus -poolID 0 > /reports/[array_hostname6]_tierstatus0.out
naviseccli -h [array_hostname6] autotiering -info -state -rate -schedule -opStatus -poolID 1 > /reports/[array_hostname6]_tierstatus1.out
#
echo `grep “Pool Name:” /reports/[array_hostname6]_tierstatus0.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname6]_tierstatus0.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
echo `grep “Pool Name:” /reports/[array_hostname6]_tierstatus1.out |awk ‘{print $4}’`”:  “`grep Complete: /reports/[array_hostname6]_tierstatus1.out |awk ‘{print $5,$6,$7,$8,$9,$10}’` >> /reports/tierstatus.txt
#
#Copy the current report to a new file and rename it, one prior day is saved.
cp /cygdrive/c/inetpub/wwwroot/tierstatus.txt /cygdrive/c/inetpub/wwwroot/tierstatus_yesterday.txt
#Remove the current report on the web page.
rm /cygdrive/c/inetpub/wwwroot/tierstatus.txt
#Copy the new report to the web page.
cp /reports/tierstatus.txt /cygdrive/c/inetpub/wwwroot

 

 

Reporting on Clariion / VNX Block Storage Pool capacity with a bash script

I recently added a post about how to report on Celerra & VNX File pool sizes with a bash script. I’ve also been doing that for a long time with our Clariion and VNX block pools so I thought I’d share that information as well.

I use a cron job to schedule the report daily and copy it to our internal web server. I then run the csv2html.pl perl script (from http://www.jpsdomain.org/source/perl.html) to convert it to an HTML output file to add to our intranet report page. This is likely the most viewed report I create on a daily basis as we always seem to be running low on available capacity.

The output of the script looks similar to this:

PoolReport

Here is the bash script that creates the report:

TODAY=$(date)
#Add the current time/date stamp to the top of the report 
echo $TODAY > /scripts/PoolReport.csv

#Create a file with the standard navisphere CLI output for each storage pool (to be processed later into the format I want)

naviseccli -h VNX_1 storagepool -list -id 0 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site1_0.csv

naviseccli -h VNX_1 storagepool -list -id 1 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site1_1.csv

naviseccli -h VNX_1 storagepool -list -id 2 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site1_2.csv

naviseccli -h VNX_1 storagepool -list -id 3 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site1_3.csv

naviseccli -h VNX_1 storagepool -list -id 4 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site1_6.csv

#

naviseccli -h VNX_2 storagepool -list -id 0 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site1_4.csv

naviseccli -h VNX_2 storagepool -list -id 1 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site1_5.csv

naviseccli -h VNX_2 storagepool -list -id 2 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site1_7.csv

naviseccli -h VNX_2 storagepool -list -id 3 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site1_8.csv

#

Naviseccli -h VNX_3 storagepool -list -id 0 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site1_6.csv

#

naviseccli -h VNX_4 storagepool -list -id 0 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site2_0.csv

naviseccli -h VNX_4 storagepool -list -id 1 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site2_1.csv

naviseccli -h VNX_4 storagepool -list -id 2 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site2_2.csv

naviseccli -h VNX_4 storagepool -list -id 3 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site2_3.csv

#

naviseccli -h VNX_5 storagepool -list -id 0 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site2_4.csv

#

naviseccli -h VNX_6 storagepool -list -id 0 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site3_0.csv

naviseccli -h VNX_6 storagepool -list -id 1 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site3_1.csv

#

naviseccli -h VNX_7 storagepool -list -id 0 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site3_0.csv

naviseccli -h VNX_7 storagepool -list -id 1 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site3_1.csv

#

naviseccli -h VNX_8 storagepool -list -id 0 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site4_0.csv

naviseccli -h VNX_8 storagepool -list -id 1 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site4_1.csv

#

naviseccli -h VNX_9 storagepool -list -id 0 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site5_0.csv

naviseccli -h VNX_9 storagepool -list -id 1 -availableCap -consumedCap -UserCap -prcntFull >/scripts/report_site5_1.csv

#

#Create a new file for each site's storage pool (from the file generated in the previous step) hat contains only the info that I want.

#

cat /scripts/report_site1_4.csv | grep Name > /scripts/Site1Pool4.csv

cat /scripts/report_site1_4.csv | grep GBs >>/scripts/Site1Pool4.csv

cat /scripts/report_site1_4.csv | grep Full >>/scripts/Site1Pool4.csv

#

cat /scripts/report_site1_5.csv | grep Name > /scripts/Site1Pool5.csv

cat /scripts/report_site1_5.csv | grep GBs >>/scripts/Site1Pool5.csv

cat /scripts/report_site1_5.csv | grep Full >>/scripts/Site1Pool5.csv

#

cat /scripts/report_site1_0.csv | grep Name > /scripts/Site1Pool0.csv

cat /scripts/report_site1_0.csv | grep GBs >>/scripts/Site1Pool0.csv

cat /scripts/report_site1_0.csv | grep Full >>/scripts/Site1Pool0.csv

#

cat /scripts/report_site1_1.csv | grep Name > /scripts/Site1Pool1.csv

cat /scripts/report_site1_1.csv | grep GBs >;;>;;/scripts/Site1Pool1.csv

cat /scripts/report_site1_1.csv | grep Full >;;>;;/scripts/Site1Pool1.csv

#

cat /scripts/report_site1_2.csv | grep Name > /scripts/Site1Pool2.csv

cat /scripts/report_site1_2.csv | grep GBs >>/scripts/Site1Pool2.csv

cat /scripts/report_site1_2.csv | grep Full >>/scripts/Site1Pool2.csv

#

cat /scripts/report_site1_7.csv | grep Name > /scripts/Site1Pool7.csv

cat /scripts/report_site1_7.csv | grep GBs >>/scripts/Site1Pool7.csv

cat /scripts/report_site1_7.csv | grep Full >>/scripts/Site1Pool7.csv

#

cat /scripts/report_site1_8.csv | grep Name > /scripts/Site1Pool8.csv

cat /scripts/report_site1_8.csv | grep GBs >>/scripts/Site1Pool8.csv

cat /scripts/report_site1_8.csv | grep Full >>/scripts/Site1Pool8.csv

#

cat /scripts/report_site1_3.csv | grep Name > /scripts/Site1Pool3.csv

cat /scripts/report_site1_3.csv | grep GBs >>/scripts/Site1Pool3.csv

cat /scripts/report_site1_3.csv | grep Full >>/scripts/Site1Pool3.csv

#

cat /scripts/report_site1_6.csv | grep Name > /scripts/Site1Pool6.csv

cat /scripts/report_site1_6.csv | grep GBs >>/scripts/Site1Pool6.csv

cat /scripts/report_site1_6.csv | grep Full >>/scripts/Site1Pool6.csv

#

cat /scripts/report_site1_6.csv | grep Name > /scripts/Site1Pool6.csv

cat /scripts/report_site1_6.csv | grep GBs >>/scripts/Site1Pool6.csv

cat /scripts/report_site1_6.csv | grep Full >>/scripts/Site1Pool6.csv

#

cat /scripts/report_site2_0.csv | grep Name > /scripts/Site2Pool0.csv

cat /scripts/report_site2_0.csv | grep GBs >>/scripts/Site2Pool0.csv

cat /scripts/report_site2_0.csv | grep Full >>/scripts/Site2Pool0.csv

#

cat /scripts/report_site2_1.csv | grep Name > /scripts/Site2Pool1.csv

cat /scripts/report_site2_1.csv | grep GBs >>/scripts/Site2Pool1.csv

cat /scripts/report_site2_1.csv | grep Full >>/scripts/Site2Pool1.csv

#

cat /scripts/report_site2_2.csv | grep Name > /scripts/Site2Pool2.csv

cat /scripts/report_site2_2.csv | grep GBs >>/scripts/Site2Pool2.csv

cat /scripts/report_site2_2.csv | grep Full >>/scripts/Site2Pool2.csv

#

cat /scripts/report_site2_3.csv | grep Name > /scripts/Site2Pool3.csv

cat /scripts/report_site2_3.csv | grep GBs >>/scripts/Site2Pool3.csv

cat /scripts/report_site2_3.csv | grep Full >>/scripts/Site2Pool3.csv

#

cat /scripts/report_site2_4.csv | grep Name > /scripts/Site2Pool4.csv

cat /scripts/report_site2_4.csv | grep GBs >>/scripts/Site2Pool4.csv

cat /scripts/report_site2_4.csv | grep Full >>/scripts/Site2Pool4.csv

#

cat /scripts/report_site3_0.csv | grep Name > /scripts/Site3Pool0.csv

cat /scripts/report_site3_0.csv | grep GBs >>/scripts/Site3Pool0.csv

cat /scripts/report_site3_0.csv | grep Full >>/scripts/Site3Pool0.csv

#

cat /scripts/report_site3_1.csv | grep Name > /scripts/Site3Pool1.csv

cat /scripts/report_site3_1.csv | grep GBs >>/scripts/Site3Pool1.csv

cat /scripts/report_site3_1.csv | grep Full >>/scripts/Site3Pool1.csv

#

cat /scripts/report_site3_0.csv | grep Name > /scripts/Site4Pool0.csv

cat /scripts/report_site3_0.csv | grep GBs >>/scripts/Site4Pool0.csv

cat /scripts/report_site3_0.csv | grep Full >>/scripts/Site4Pool0.csv

#

cat /scripts/report_site3_1.csv | grep Name > /scripts/Site4Pool1.csv

cat /scripts/report_site3_1.csv | grep GBs >>/scripts/Site4Pool1.csv

cat /scripts/report_site3_1.csv | grep Full >>/scripts/Site4Pool1.csv

#

cat /scripts/report_site4_0.csv | grep Name > /scripts/Site5Pool0.csv

cat /scripts/report_site4_0.csv | grep GBs >>/scripts/Site5Pool0.csv

cat /scripts/report_site4_0.csv | grep Full >>/scripts/Site5Pool0.csv

#

cat /scripts/report_site4_1.csv | grep Name > /scripts/Site5Pool1.csv

cat /scripts/report_site4_1.csv | grep GBs >>/scripts/Site5Pool1.csv

cat /scripts/report_site4_1.csv | grep Full >>/scripts/Site5Pool1.csv

#

cat /scripts/report_site5_0.csv | grep Name > /scripts/Site6Pool0.csv

cat /scripts/report_site5_0.csv | grep GBs >>/scripts/Site6Pool0.csv

cat /scripts/report_site5_0.csv | grep Full >>/scripts/Site6Pool0.csv

#

cat /scripts/report_site5_1.csv | grep Name > /scripts/Site6Pool1.csv

cat /scripts/report_site5_1.csv | grep GBs >>/scripts/Site6Pool1.csv

cat /scripts/report_site5_1.csv | grep Full >>/scripts/Site6Pool1.csv

#

#The last section creates the final output for the report before it is processed into an html table. It creates a single line for each storage pool with the total GB available, total GB used, available GB, and the percent utilization of the pool.

#

echo 'Pool Name','Total GB ','Used GB ','Available GB ','Percent Full ' >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site1Pool0.csv |awk '{print $3}'`","`grep -i User /scripts/Site1Pool0.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site1Pool0.csv |awk '{print $4}'`","`grep -i Available /scripts/Site1Pool0.csv |awk '{print $4}'`","`grep -i Full /scripts/Site1Pool0.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site1Pool1.csv |awk '{print $3}'`","`grep -i User /scripts/Site1Pool1.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site1Pool1.csv |awk '{print $4}'`","`grep -i Available /scripts/Site1Pool1.csv |awk '{print $4}'`","`grep -i Full /scripts/Site1Pool1.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site1Pool2.csv |awk '{print $3}'`","`grep -i User /scripts/Site1Pool2.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site1Pool2.csv |awk '{print $4}'`","`grep -i Available /scripts/Site1Pool2.csv |awk '{print $4}'`","`grep -i Full /scripts/Site1Pool2.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site1Pool3.csv |awk '{print $3}'`","`grep -i User /scripts/Site1Pool3.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site1Pool3.csv |awk '{print $4}'`","`grep -i Available /scripts/Site1Pool3.csv |awk '{print $4}'`","`grep -i Full /scripts/Site1Pool3.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site1Pool6.csv |awk '{print $3}'`","`grep -i User /scripts/Site1Pool6.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site1Pool6.csv |awk '{print $4}'`","`grep -i Available /scripts/Site1Pool6.csv |awk '{print $4}'`","`grep -i Full /scripts/Site1Pool6.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

#

echo " ",'Total GB','Used GB','Available GB','Percent Full' >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site1Pool4.csv |awk '{print $3}'`","`grep -i User /scripts/Site1Pool4.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site1Pool4.csv |awk '{print $4}'`","`grep -i Available /scripts/Site1Pool4.csv |awk '{print $4}'`","`grep -i Full /scripts/Site1Pool4.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site1Pool5.csv |awk '{print $3}'`","`grep -i User /scripts/Site1Pool5.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site1Pool5.csv |awk '{print $4}'`","`grep -i Available /scripts/Site1Pool5.csv |awk '{print $4}'`","`grep -i Full /scripts/Site1Pool5.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site1Pool7.csv |awk '{print $3}'`","`grep -i User /scripts/Site1Pool7.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site1Pool7.csv |awk '{print $4}'`","`grep -i Available /scripts/Site1Pool7.csv |awk '{print $4}'`","`grep -i Full /scripts/Site1Pool7.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site1Pool8.csv |awk '{print $3}'`","`grep -i User /scripts/Site1Pool8.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site1Pool8.csv |awk '{print $4}'`","`grep -i Available /scripts/Site1Pool8.csv |awk '{print $4}'`","`grep -i Full /scripts/Site1Pool8.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

#

echo " ",'Total GB','Used GB','Available GB','Percent Full' >> /scripts/PoolReport.csv

#

#

echo `grep Name /scripts/Site1Pool6.csv |awk '{print $3}'`","`grep -i User /scripts/Site1Pool6.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site1Pool6.csv |awk '{print $4}'`","`grep -i Available /scripts/Site1Pool6.csv |awk '{print $4}'`","`grep -i Full /scripts/Site1Pool6.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

#

echo " ",'Total GB','Used GB','Available GB','Percent Full' >> /scripts/PoolReport.csv

#

#

echo `grep Name /scripts/Site2Pool0.csv |awk '{print $3}'`","`grep -i User /scripts/Site2Pool0.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site2Pool0.csv |awk '{print $4}'`","`grep -i Available /scripts/Site2Pool0.csv |awk '{print $4}'`","`grep -i Full /scripts/Site2Pool0.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site2Pool1.csv |awk '{print $3}'`","`grep -i User /scripts/Site2Pool1.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site2Pool1.csv |awk '{print $4}'`","`grep -i Available /scripts/Site2Pool1.csv |awk '{print $4}'`","`grep -i Full /scripts/Site2Pool1.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site2Pool2.csv |awk '{print $3}'`","`grep -i User /scripts/Site2Pool2.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site2Pool2.csv |awk '{print $4}'`","`grep -i Available /scripts/Site2Pool2.csv |awk '{print $4}'`","`grep -i Full /scripts/Site2Pool2.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site2Pool3.csv |awk '{print $3}'`","`grep -i User /scripts/Site2Pool3.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site2Pool3.csv |awk '{print $4}'`","`grep -i Available /scripts/Site2Pool3.csv |awk '{print $4}'`","`grep -i Full /scripts/Site2Pool3.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

#

echo " ",'Total GB','Used GB','Available GB','Percent Full' >> /scripts/PoolReport.csv

#

#

echo `grep Name /scripts/Site2Pool4.csv |awk '{print $3}'`","`grep -i User /scripts/Site2Pool4.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site2Pool4.csv |awk '{print $4}'`","`grep -i Available /scripts/Site2Pool4.csv |awk '{print $4}'`","`grep -i Full /scripts/Site2Pool4.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

#

echo " ",'Total GB','Used GB','Available GB','Percent Full' >> /scripts/PoolReport.csv

#

#

echo `grep Name /scripts/Site3Pool0.csv |awk '{print $3}'`","`grep -i User /scripts/Site3Pool0.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site3Pool0.csv |awk '{print $4}'`","`grep -i Available /scripts/Site3Pool0.csv |awk '{print $4}'`","`grep -i Full /scripts/Site3Pool0.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site3Pool1.csv |awk '{print $3}'`","`grep -i User /scripts/Site3Pool1.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site3Pool1.csv |awk '{print $4}'`","`grep -i Available /scripts/Site3Pool1.csv |awk '{print $4}'`","`grep -i Full /scripts/Site3Pool1.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

#

echo " ",'Total GB','Used GB','Available GB','Percent Full' >> /scripts/PoolReport.csv

#

#

echo `grep Name /scripts/Site4Pool0.csv |awk '{print $3}'`","`grep -i User /scripts/Site4Pool0.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site4Pool0.csv |awk '{print $4}'`","`grep -i Available /scripts/Site4Pool0.csv |awk '{print $4}'`","`grep -i Full /scripts/Site4Pool0.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site4Pool1.csv |awk '{print $3}'`","`grep -i User /scripts/Site4Pool1.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site4Pool1.csv |awk '{print $4}'`","`grep -i Available /scripts/Site4Pool1.csv |awk '{print $4}'`","`grep -i Full /scripts/Site4Pool1.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

#

echo " ",'Total GB','Used GB','Available GB','Percent Full' >> /scripts/PoolReport.csv

#

#

echo `grep Name /scripts/Site5Pool0.csv |awk '{print $3}'`","`grep -i User /scripts/Site5Pool0.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site5Pool0.csv |awk '{print $4}'`","`grep -i Available /scripts/Site5Pool0.csv |awk '{print $4}'`","`grep -i Full /scripts/Site5Pool0.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site5Pool1.csv |awk '{print $3}'`","`grep -i User /scripts/Site5Pool1.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site5Pool1.csv |awk '{print $4}'`","`grep -i Available /scripts/Site5Pool1.csv |awk '{print $4}'`","`grep -i Full /scripts/Site5Pool1.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

#

echo " ",'Total GB','Used GB','Available GB','Percent Full' >> /scripts/PoolReport.csv

#

#

echo `grep Name /scripts/Site6Pool0.csv |awk '{print $3}'`","`grep -i User /scripts/Site6Pool0.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site6Pool0.csv |awk '{print $4}'`","`grep -i Available /scripts/Site6Pool0.csv |awk '{print $4}'`","`grep -i Full /scripts/Site6Pool0.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

echo `grep Name /scripts/Site6Pool1.csv |awk '{print $3}'`","`grep -i User /scripts/Site6Pool1.csv |awk '{print $4}'`","`grep -i Consumed /scripts/Site6Pool1.csv |awk '{print $4}'`","`grep -i Available /scripts/Site6Pool1.csv |awk '{print $4}'`","`grep -i Full /scripts/Site6Pool1.csv |awk '{print $3}'` >> /scripts/PoolReport.csv

#

#Convert the file to HTML for use on the internal web server 

#

./csv2htm.pl -e -T -i /scripts/PoolReport.csv -o /webfolder/PoolReport.html

Reporting on Celerra / VNX NAS Pool capacity with a bash script

I recently created a script that I run on all of our celerras and VNX’s that reports on NAS pool size.   The output from each array is then converted to HTML and combined on a single intranet page to provide a quick at-a-glance view of our global NAS capacity and disk space consumption.  I made another post that shows how to create a block storage pool report as well:  http://emcsan.wordpress.com/2013/08/09/reporting-on-celerravnx-block-storage-pool-capacity-with-a-bash-script/

The default command unfortunately outputs in Megabytes with no option to change to GB or TB.  This script performs the MB to GB conversion and adds a comma as the numerical separator (what we use in the USA) to make the output much more readable.

First, identify the ID number for each of your NAS pools.  You’ll need to insert the ID numbers into the script itself.

[nasadmin@celerra]$ nas_pool -list
 id      inuse   acl     name                      storage system
 10      y       0       NAS_Pool0_SPA             AKM00111000000
 18      y       0       NAS_Pool1_SPB             AKM00111000000
Note that the default output of the command that provides the size of each pool is in a very hard to read format.  I wanted to clean it up to make it easier to read on our reporting page.  Here’s the default output:
[nasadmin@celerra]$ nas_pool -size -all
id           = 10
name         = NAS_Pool0_SPA
used_mb      = 3437536
avail_mb     = 658459
total_mb     = 4095995
potential_mb = 0
id           = 18
name         = NAS_Pool1_SPB
used_mb      = 2697600
avail_mb     = 374396
total_mb     = 3071996
potential_mb = 1023998
 My script changes the output to look like the example below.
Name (Site)   ; Total GB ; Used GB  ; Avail GB
 NAS_Pool0_SPA ; 4,000    ; 3,356.97 ; 643.03
 NAS_Pool1_SPB ; 3,000    ; 2,634.38 ; 365.62
 In this example there are two NAS pools and this script is set up to report on both.  It could be easily expanded or reduced depending on the number of pools on your array. The variable names I used include the Pool ID number from the output above, that should be changed to match your ID’s.  You’ll also need to update the ‘id=’ portion of each command to match your Pool ID’s.

Here’s the script:

#!/bin/bash

NAS_DB="/nas"
export NAS_DB

# Set the Locale to English/US, used for adding the comma as a separator in a cron job
export LC_NUMERIC="en_US.UTF-8"
TODAY=$(date)

 

# Gather Pool Name, Used MB, Avaialble MB, and Total MB for First Pool

# Set variable to pull the Name of the pool from the output of 'nas_pool -size'.
name18=`/nas/bin/nas_pool -size id=18 | /bin/grep name | /bin/awk '{print $3}'`

# Set variable to pull the Used MB of the pool from the output of 'nas_pool -size'.
usedmb18=`/nas/bin/nas_pool -size id=18 | /bin/grep used_mb | /bin/awk '{print $3}'`

# Set variable to pull the Available MB of the pool from the output of 'nas_pool -size'.
availmb18=`/nas/bin/nas_pool -size id=18 | /bin/grep avail_mb | /bin/awk '{print $3}'`
# Set variable to pull the Total MB of the pool from the output of 'nas_pool -size'.

totalmb18=`/nas/bin/nas_pool -size id=18 | /bin/grep total_mb | /bin/awk '{print $3}'`

# Convert MB to GB, Add Comma as separator in output

# Remove '...b' variables if you don't want commas as a separator

# Convert Used MB to Used GB
usedgb18=`/bin/echo $usedmb18/1024 | /usr/bin/bc -l | /bin/sed 's/^\./0./;s/0*$//;s/0*$//;s/\.$//'`

# Add comma separator
usedgb18b=`/usr/bin/printf "%'.2f\n" "$usedgb18" | /bin/sed 's/\.00$// ; s/\(\.[1-9]\)0$/\1/'`

# Convert Available MB to Available GB
availgb18=`/bin/echo $availmb18/1024 | /usr/bin/bc -l | /bin/sed 's/^\./0./;s/0*$//;s/0*$//;s/\.$//'`

# Add comma separator
availgb18b=`/usr/bin/printf "%'.2f\n" "$availgb18" | /bin/sed 's/\.00$// ; s/\(\.[1-9]\)0$/\1/'`

# Convert Total MB to Total GB
totalgb18=`/bin/echo $totalmb18/1024 | /usr/bin/bc -l | /bin/sed 's/^\./0./;s/0*$//;s/0*$//;s/\.$//'`

# Add comma separator
totalgb18b=`/usr/bin/printf "%'.2f\n" "$totalgb18" | /bin/sed 's/\.00$// ; s/\(\.[1-9]\)0$/\1/'`

# Gather Pool Name, Used MB, Avaialble MB, and Total MB for Second Pool

# Set variable to pull the Name of the pool from the output of 'nas_pool -size'.
name10=`/nas/bin/nas_pool -size id=10 | /bin/grep name | /bin/awk '{print $3}'`

# Set variable to pull the Used MB of the pool from the output of 'nas_pool -size'.
usedmb10=`/nas/bin/nas_pool -size id=10 | /bin/grep used_mb | /bin/awk '{print $3}'`

# Set variable to pull the Available MB of the pool from the output of 'nas_pool -size'.
availmb10=`/nas/bin/nas_pool -size id=10 | /bin/grep avail_mb | /bin/awk '{print $3}'`

# Set variable to pull the Total MB of the pool from the output of 'nas_pool -size'.
totalmb10=`/nas/bin/nas_pool -size id=10 | /bin/grep total_mb | /bin/awk '{print $3}'`
 
# Convert MB to GB, Add Comma as separator in output

# Remove '...b' variables if you don't want commas as a separator
 
# Convert Used MB to Used GB
usedgb10=`/bin/echo $usedmb10/1024 | /usr/bin/bc -l | /bin/sed 's/^\./0./;s/0*$//;s/0*$//;s/\.$//'`

# Add comma separator
usedgb10b=`/usr/bin/printf "%'.2f\n" "$usedgb10" | /bin/sed 's/\.00$// ; s/\(\.[1-9]\)0$/\1/'`

# Convert Available MB to Available GB
availgb10=`/bin/echo $availmb10/1024 | /usr/bin/bc -l | /bin/sed 's/^\./0./;s/0*$//;s/0*$//;s/\.$//'`

# Add comma separator
availgb10b=`/usr/bin/printf "%'.2f\n" "$availgb10" | /bin/sed 's/\.00$// ; s/\(\.[1-9]\)0$/\1/'`

# Convert Total MB to Total GB
totalgb10=`/bin/echo $totalmb10/1024 | /usr/bin/bc -l | /bin/sed 's/^\./0./;s/0*$//;s/0*$//;s/\.$//'`

# Add comma separator
totalgb10b=`/usr/bin/printf "%'.2f\n" "$totalgb10" | /bin/sed 's/\.00$// ; s/\(\.[1-9]\)0$/\1/'`

# Create Output File

# If you don't want the comma separator in the output file, substitute the variable without the 'b' at the end.

# I use the semicolon rather than the comma as a separator due to the fact that I'm using the comma as a numerical separator.

# The comma could be substituted here if desired.

/bin/echo $TODAY > /scripts/NasPool.txt
/bin/echo "Name" ";" "Total GB" ";" "Used GB" ";" "Avail GB" >> /scripts/NasPool.txt
/bin/echo $name18 ";" $totalgb18b ";" $usedgb18b ";" $availgb18b >> /scripts/NasPool.txt
/bin/echo $name10 ";" $totalgb10b ";" $usedgb10b ";" $availgb10b >> /scripts/NasPool.txt
 Here’s what the Output looks like:
Wed Jul 17 23:56:29 JST 2013
 Name (Site) ; Total GB ; Used GB ; Avail GB
 NAS_Pool0_SPA ; 4,000 ; 3,356.97 ; 643.03
 NAS_Pool1_SPB ; 3,000 ; 2,634.38 ; 365.62
 I use a cron job to schedule the report daily and copy it to our internal web server.  I then run the csv2html.pl perl script (from http://www.jpsdomain.org/source/perl.html) to convert it to an HTML output file to add to our intranet report page.

Note that I had to modify the csv2html.pl command to accomodate the use of a semicolon instead of the default comma in a csv file.  Here is the command I use to do the conversion:

./csv2htm.pl -e -T -D “;” -i /reports/NasPool.txt -o /reports/NasPool.html
 Below is what the output looks like after running the HTML conversion tool.

NASPool

Disk space reporting on sub folders on a VNX File CIFS shared file system

I recently had a comment on a different post asking how to report on the size of multiple folders on a single file system, so I thought I’d share the method I use to create reports and alerts on the space that sub folders on file systems consume. There is a way to navigate to all of the file systems from the control station, simply navigate to /nas/quota/slot_<x>. Slot_<x> refers to the data mover. The file systems on server_2 would be in the slot_2 folder.  Because we have access to those folders, we can simply run the standard unix ‘du’ command to get the amount of used disk space for each sub folder on any file system.

Running this command:

sudo du -h /nas/quota/slot_2/File_System/Sub_Folder

Will give an output that looks like this:

24K     /nas/quota/slot_2/File_System/Sub_Folder/1
0       /nas/quota/slot_2/File_System/Sub_Folder/2
16K     /nas/quota/slot_2/File_System/Sub_Folder

Each sub folder of the file system named “File_System” is listed with the space each sub folder uses. You’ll notice that I used the sudo command to run the du command. Unfortunately you need root access in order to run the du command on the /nas/quota directory, so you’ll have to add whatever account you log in to the celerra with to the /etc/sudoers file. If you don’t, you’ll get the error “Sorry, user nasadmin is not allowed to execute ‘/usr/bin/du -h -s /nas/quota/slot_2/File_System’ as root on “. I generally log in to the celerra as nasadmin, so I added that account to sudoers. To modify the file, su to root first and then (using vi) add the following to the very bottom of the file of /etc/sudoers (substitute nasadmin for the username you will be using):

nasadmin ALL=/usr/bin/du, /usr/bin/, /sbin/, /usr/sbin, /nas/bin, /nas/bin/, /nas/sbin, /nas/sbin/, /bin, /bin/
nasadmin ALL=/nas/bin/server_df

Once that is complete, you’ll have access to run the command.  You can now write a script that will report on specific folders for you. I have two scripts created that I’m going to share, one I use to send out a daily email report and the other will send an email only if the folder sizes have crossed a certain threshold of space utilization. I have them both scheduled as a cron job on the Celerra itself.  It should be easy to modify these scripts for anyone’s environment.

The following script will generate a report of folder and sub folder sizes for any given file system. In this example, I am reporting on three specific subfolders on a file system (Production, Development, and Test). I also use the grep and awk options at the beginning to pull out the percent full from the df command for the entire file system and include that in the subject line of the email.

#!/bin/bash
 NAS_DB="/nas"
 export NAS_DB
 TODAY=$(date)
 HOST=$(hostname)

PERCENT=`sudo df -h /nas/quota/slot_2/File_System | grep server_2 | awk '{print $5}'`

echo "File_System Folder Size Report Date: $TODAY Host:$HOST" > /home/nasadmin/report/fs_report.txt
 echo " " >> /home/nasadmin/report/fs_report.txt
 echo "Production" >> /home/nasadmin/report/fs_report.txt
 echo " " >> /home/nasadmin/report/fs_report.txt

sudo du -h -S /nas/quota/slot_2/File_System/Prod >> /home/nasadmin/report/fs_report.txt

echo " " >> /home/nasadmin/report/fs_report.txt
 echo "Development" >> /home/nasadmin/report/fs_report.txt
 echo " " >> /home/nasadmin/report/fs_report.txt

sudo du -h -S /nas/quota/slot_2/File_System/Dev >> /home/nasadmin/report/fs_report.txt

echo " " >> /home/nasadmin/report/fs_report.txt
 echo "Test" >> /home/nasadmin/report/fs_report.txt
 echo " " >> /home/nasadmin/report/fs_report.txt

sudo du -h -S /nas/quota/slot_2/File_System/Test >> /home/nasadmin/report/fs_report.txt

echo " " >> /home/nasadmin/report/fs_report.txt
 echo "% Remaining on File_System Filesystem" >> /home/nasadmin/report/fs_report.txt
 echo " " >> /home/nasadmin/report/fs_report.txt

sudo df -h /nas/quota/slot_2/File_System/ >> /home/nasadmin/report/fs_report.txt

echo $PERCENT

mail -s "Folder Size Report ($PERCENT In Use)" youremail@domain.com < /home/nasadmin/report/fs_report.txt

Below is what the output of that script looks like. It is included in the body of the email as plain text.

File_System Folder Size Report
Date: Fri Jun 28 08:00:02 CDT 2013
Host:<celerra_name>

Production

 24K     /nas/quota/slot_2/File_System/Production/1
 0       /nas/quota/slot_2/File_System/Production/2
 16K     /nas/quota/slot_2/File_System/Production

Development

 8.0K    /nas/quota/slot_2/File_System/Development/1
 108G    /nas/quota/slot_2/File_System/Development/2
 0          /nas/quota/slot_2/File_System/Development/3
 16K     /nas/quota/slot_2/File_System/Development

Test

 0       /nas/quota/slot_2/File_System/Test/1
 422G    /nas/quota/slot_2/File_System/Test/2
 0       /nas/quota/slot_2/File_System/Test/3
 16K     /nas/quota/slot_2/File_System/Test

% Remaining on File_System Filesystem

Filesystem Size Used Avail Use% Mounted on
 server_2:/ 2.5T 529G 1.9T 22% /nasmcd/quota/slot_2
 .

The following script will only send an email if the folder size has crossed a defined threshold. Simply change the path to the folder you want to report on and change the value of the ‘threshold’ variable in the script to whatever you want it to be, I have mine set at 95%.

#Get the % Disk Utilization from the server_df command
 percent=`sudo df -h /nas/quota/slot_2/File_System | grep server_2 | awk '{print $5}'`

#Strip the % Sign from the output value
 percentvalue=`echo $percent | awk -F"%" '{print $1}'`

#Set the critical threshold that will trigger an email
 threshold=95

#compare the threshold value to the reported value, send an email if needed
 if [ $percentvalue -eq 0 ] && [ $threshold -eq 0 ]
 then
  echo "Both are zero"
 elif [ $percentvalue -eq $threshold ]
 then
  echo "Both Values are equal"
 elif [ $percentvalue -gt $threshold ]
 then
 mail -s "File_System Critical Disk Space Alert: $percentvalue% utilization is above the threshold of $threshold%" youremail@domain.com < /home/nasadmin/report/fs_report.txt

else
  echo "$percentvalue is less than $threshold"
 fi

Automating VNX Storage Processor Percent Utilization Alerts

Note:  The original post describes a method that requires EMC Control Center and Performance Manager.  That tool has been deprecated by EMC in favor of ViPR SRM.  There is still a method you can use to gather CPU information for use in bash scripts. I don’t have script examples that use this command, but if anyone needs help send me a comment and I’ll help. The Navisphere CLI command to get busy/idle ticks for the Storage processors is naviseccli -h getcontrol -cbt.

The output looks like this:

Controller busy ticks: 1639432
Controller idle ticks: 1773844

The SP utilization statistics outputted are an average of the utilization across all the cores of the SP’s processors since the last reset. To get the actual point-in-time SP CPU utilization from this output requires a calculation. You need to poll twice, create a delta for the individual counters by subtracting the earlier value from the later, and apply this formula:

Utilization = Busy Ticks / (Busy Ticks + Idle Ticks)

What follows is the original method I posted that requries EMC Control Center.

I was tasked with coming up with a way to get email alerts whenever our SP utilization breaks a certain threshold.  Since none of the monitoring tools that we own will do that right now, I had to come up with a way using custom scripts.  This is my 2nd post on the same subject, I removed my post from yesterday as it didn’t work as I intended.  This time I used EMC’s Performance Manager rather than pulling data from the SP with the Navisphere CLI.

First, I’m running all of my bash scripts on a windows sever using cygwin.  These should run fine on any linux box as well, however.  Because I don’t have a native sendmail configuration set up on the windows server, I’m using the control station on the Celerra to actually do the comparison of the utilization numbers in the text files and then email out an alert.  The Celerra control station automatically pulls the file via FTP from the windows server every 30 minutes and sends out an email alert if the numbers cross the threshold.  A description of each script and the schedule is below.

Windows Server:

Export.cmd:

This first windows batch script runs an export (with pmcli) from EMC Performance Manager that does a dump of all the performance stats for the current day.

For /f "tokens=2-4 delims=/ " %%a in ('date /t') do (set date=%%c%%a%%b)

C:\ECC\Client.610\PerformanceManager\pmcli.exe -export -out c:\cygwin\home\scripts\sputil999_interval.csv -type interval -class clariion -date %date% -id APM00400500999

Data.cmd:

This cygwin/bash script manipulates the file export from above and ultimately creates two single text files (one for SPA and one for SPB) with a single numerical value of the most recent SP Utilization.  There are a few extra steps at the beginning of the script that are irrelevant to the SP utilization, they’re there for other purposes.

#This will pull only the timestamp line from the top

grep -m 1 "/" /home/scripts/sputil/0999_interval.csv > /home/scripts/sputil/timestamp.csv

# This will pull out only the "disk utilization" line.

grep -i "^% Utilization" /home/scripts/sputil/0999_interval.csv >> /home/scripts/sputil/stats.csv

# This will pull out the disk/LUN title info for the first column

grep -i "Data Collected for DiskStats -" /home/scripts/sputil/0999_interval.csv > /home/scripts/sputil/diskstats.csv

grep -i "Data Collected for LUNStats -" /home/scripts/sputil/0999_interval.csv > /home/scripts/sputil/lunstats.csv

# This will create a column with the disk/LUN number

cat /home/scripts/sputil/diskstats.csv /home/scripts/sputil/lunstats.csv > /home/scripts/sputil/data.csv

# This combines the disk/LUN column with the data column

paste /home/scripts/sputil/data.csv /home/scripts/sputil/stats.csv > /home/scripts/sputil/combined.csv

cp /home/scripts/sputil/combined.csv /home/scripts/sputil/utilstats.csv
 

#  This removes all the temporary files
rm /home/scripts/sputil/timestamp.csv
rm /home/scripts/sputil/stats.csv
rm /home/scripts/sputil/diskstats.csv
rm /home/scripts/sputil/lunstats.csv
rm /home/scripts/sputil/data.csv
rm /home/scripts/sputil/combined.csv

# This next line strips the file of all but the last two rows, which are SP Utilization.

# The 1 looks at the first character in the row, the D specifies "starts with D", then deletes rows meeting those conditions.

awk -v FS="" -v OFS="" '$1 != "D"' < /home/scripts/sputil/utilstats.csv > /home/scripts/sputil/sputil.csv

#This pulls the values from the last column, which would be the most recent.

awk -F, '{print $(NF-1)}' < /home/scripts/sputil/sputil.csv > /home/scripts/sputil/sp_util.csv

#pull 1st line (SPA) into separate file

sed -n 1,1p < /home/scripts/sputil/sp_util.csv > /home/scripts/sputil/spAutil.txt

#pull 2nd line (SPB) into separate file

sed -n 2,2p < /home/scripts/sputil/sp_util.csv > /home/scripts/sputil/spButil.txt

#The spAutil.txt/spButil.txt files now contain only a single numerical value, which would be the most recent %utilization from the Control Center/Performance Manager dump file.

#Copy files to web server root directory

cp /home/scripts/sputil/*.txt /cygdrive/c/inetpub/wwwroot

Celerra Control Station:

CelerraArray:/home/nasadmin/sputil/ftpsp.sh

The script below connects to the windows server and grabs the current SP utilization text files via FTP every 30 minutes (via a cron job).

#!/bin/bash
cd /home/nasadmin/sputil
ftp windows_server.domain.net <<SCRIPT
get spAutil.txt
get spButil.txt
quit
SCRIPT
 CelerraArray:/home/nasadmin/sputil/spcheck.sh:

This script does the comparison check to see if the SP utilization is over our threshold. If it is, it sends an email alert that includes the %Utilization number in the subject line of the email. To change the threshold setting, you’d need to change the THRESHOLD=<XX> line in the script.  The line containing printf “%2.0f” converts the floating point value to an integer, as bash scripts don’t recognize floating point values.

#!/bin/bash

SPB=`cat /home/nasadmin/sputil/spButil.txt` 
SPBcheck= printf "%2.0f" $SPB > /home/nasadmin/sputil/spButil2.txt 
SPB=`cat /home/nasadmin/sputil/spButil2.txt`

echo $SPB
THRESHOLD=50
if [ $SPB -eq 0 ] && [ $THRESHOLD -eq 0 ] 
then 
        echo "Both are zero"
 elif [ $SPB -eq $THRESHOLD ]
 then         
        echo "Both Values are equal"
 elif [ $SPB -gt $THRESHOLD ]
 then          
        echo "SPB is greater than the threshold.  Sending alert" 

        uuencode spButil.txt | mail -s "<array_name> SPB Utilization Alert: $SPB % above threshold of $THRESHOLD %" notify@domain.com
else         
echo "$SPB is lesser than $THRESHOLD" 
fi

CelerraArray Crontab schedule:

The FTP script is currently set to pull SP utilization files.  Run “crontab –e” to edit the scheduler.  I’ve got the alert script set to run at the top of the hour and half past the hour, and the updated SP files from the web server are FTP’d in a few minutes prior.

[nasadmin@CelerraArray sputil]$ crontab –l
58,28 * * * * /home/nasadmin/sputil/ftpsp.sh
0,30 * * * * /home/nasadmin/sputil/spcheck.sh
 Overall Scheduling:

Windows Server:

Performance Manager Dump runs 15 minutes past the hour (exports data)
Data script runs at 20 minutes past the hour (processes data to get SP Utilization)

Celerra Server:

FTP script pulls new SP utilization text files at 28 minutes past the hour
Alert script runs at 30 minutes past the hour

The cycle then repeats at minute 45, minute 50, minute 58, and minute 0.

 

Reporting on the state of VNX auto-tiering

 

To go along with my previous post (reporting on LUN tier distribution) I also include information on the same intranet page about the current state of the auto-tiering job.  We run auto-tiering from 10PM to 6AM in the morning to avoid the movement of data during business hours or our normal backup window in the evening.

Sometimes the auto-tiering job will get very backed up and would theoretically never finish in the time slot that we have for data movement.  I like to keep tabs on the amount of data that needs to move up or down, and the amount of time that the array estimates until it’s completion.  If needed, I will sometimes modify the schedule to run 24 hours a day over the weekend and change it back early on Monday morning.  Unfortunately, EMC did not design the auto-tiering scheduler to allow for creating different time windows on different days. It’s a manual process.

This is a relatively simple, one line CLI command, but it provides very useful info and it’s convenient to add it to a daily report to see it at a glance.

I run this script at 6AM every day, immediately following the end of the window for data to move:

naviseccli -h clariion1_hostname autotiering -info -state -rate -schedule -opStatus > c:\inetpub\wwwroot\clariion1_hostname.autotier.txt

naviseccli -h clariion2_hostname autotiering -info -state -rate -schedule -opStatus > c:\inetpub\wwwroot\clariion2_hostname.autotier.txt

naviseccli -h clariion3_hostname autotiering -info -state -rate -schedule -opStatus > c:\inetpub\wwwroot\clariion3_hostname.autotier.txt

naviseccli -h clariion4_hostname autotiering -info -state -rate -schedule -opStatus > c:\inetpub\wwwroot\clariion4_hostname.autotier.txt

 ....
 The output for each individual clariion looks like this:
Auto-Tiering State: Enabled
Relocation Rate: Medium

Schedule Name: Default Schedule
Schedule State: Enabled
Default Schedule: Yes
Schedule Days: Sun Mon Tue Wed Thu Fri Sat
Schedule Start Time: 22:00
Schedule Stop Time: 6:00
Schedule Duration: 8 hours
Storage Pools: Clariion1_SPB, Clariion2_SPA

Storage Pool Name: Clariion2_SPA
Storage Pool ID: 0
Relocation Start Time: 12/05/11 22:00
Relocation Stop Time: 12/06/11 6:00
Relocation Status: Inactive
Relocation Type: Scheduled
Relocation Rate: Medium
Data to Move Up (GBs): 1854.11
Data to Move Down (GBs): 909.06
Data Movement Completed (GBs): 2316.00
Estimated Time to Complete: 9 hours, 12 minutes
Schedule Duration Remaining: None

Storage Pool Name: Clariion1_SPB
Storage Pool ID: 1
Relocation Start Time: 12/05/11 22:00
Relocation Stop Time: 12/06/11 6:00
Relocation Status: Inactive
Relocation Type: Scheduled
Relocation Rate: Medium
Data to Move Up (GBs): 1757.11
Data to Move Down (GBs): 878.05
Data Movement Completed (GBs): 1726.00
Estimated Time to Complete: 11 hours, 42 minutes
Schedule Duration Remaining: None
 
 

Reporting on LUN auto-tier distribution

We have auto-tiering turned on in all of our storage pools, which all use EFD, FC, and SATA disks.  I created a script that will generate a list of all of our LUNs and the current tier distribution for each LUN.  Note that this script is designed to run in unix.  It can be run using cygwin installed on a Windows server if you don’t have access to a unix based server.

You will first need to create a text file with a list of the hostnames for your arrays (or the IP to one of the storage processors for each array).  Separate lists must be made for VNX vs. older Clariion arrays, as the naviseccli output was changed for VNX.  For example, “Flash” in the text output on a CX was changed to “Extreme Performance” as the output from a VNX when you run the same command.  I have one file named san.list for the older arrays, and another named san2.list for the VNX arrays.

As I mentioned in my previous post, our naming convention for LUNs includes the pool ID, LUN number, server name, filesystem/drive letter, last four digits of the array’s serial number, and size (in GB). Having all of this information in the LUN name makes for very easy reporting.  This information is what truly makes this report useful, as simply having a list of LUNs gives me all the information I need for reporting.  If I need to look at tier distribution for a certain server from this report, I simply filter the list in the spreadsheet for the server name (which is included in the LUN name).

Here’s what our LUN names looks like: P1_LUN100_SPA_0000_servername_filesystem_150G

As I said earlier, because of output differences from the naviseccli command on VNX arrays vs. older CX’s, I have two separate scripts.  I’ll include the complete scripts first, then explain in more detail what each section does.

Here is the script for CX series arrays:

for san in `/bin/cat /reports/tiers/san.list`
do
naviseccli -h $san lun -list -tiers |grep LUN |awk '{print $2}' > $san.out 
     for lun in `cat $san.out`
        do
        sleep 2
        echo $san
        naviseccli -h $san -np lun -list -name $lun -tiers > $lun.$san.dat &
     done 

mv $san.report.csv $san.report.`date +%j`.csv 
echo "LUN Name","FLASH","FC","SATA" > $san.report.csv 
     for lun in `cat  $san.out`
        do
        echo $lun
        echo `grep Name $lun.$san.dat |awk '{print $2}'`","`grep -i flash $lun.$san.dat |awk '{print $2}'`","`grep -i fc $lun.$san.dat |awk '{print $2}'`","`grep -i sata $lun.$san.dat |awk '{print $2}'` >> $san.report.csv
     done
 done

./csv2htm.pl -e -T -i /reports/clariion1_hostname.report.csv -o /reports/clariion1_hostname.report.html

./csv2htm.pl -e -T -i /reports/clariion2_hostname.report.csv -o /reports/clariion2_hostname.report.html

./csv2htm.pl -e -T -i /reports/clariion3_hostname.report.csv -o /reports/clariion3_hostname.report.html

Here is the script for VNX series arrays:

for san in `/bin/cat /reports/tiers2/san2.list`
do
naviseccli -h $san lun -list -tiers |grep LUN |awk '{print $2}' > $san.out
   for lun in `cat $san.out`
     do
     sleep 2
     echo $san.Generating-LUN-List
     naviseccli -NoPoll -h $san lun -list -name $lun -tiers > $lun.$san.dat &
  done

mv $san.report.csv $san.report.`date +%j`.csv
echo "LUN Name","FLASH","FC","SATA" > $san.report.csv
   for lun in `cat  $san.out`
      do
      echo $lun
      echo `grep Name $lun.$san.dat |awk '{print $2}'`","`grep -i extreme $lun.$san.dat |awk '{print $3}'`","`grep -i Performance $lun.$san.dat |grep -v Extreme|awk '{print $2}'`","`grep -i Capacity $lun.$san.dat |awk '{print $2}'` >> $san.report.csv
   done
 done

./csv2htm.pl -e -T -i /reports/VNX1_hostname.report.csv -o /reports/VNX1_hostname.report.html

./csv2htm.pl -e -T -i /reports/VNX2_hostname.report.csv -o /reports/VNX2_hostname.report.html

./csv2htm.pl -e -T -i /reports/VNX3_hostname.report.csv -o /reports/VNX3_hostname.report.html
 Here is a more detailed explanation of the script.

Section 1:

The entire script runs in a loop based on the SAN hostname entries.   We’ll use this list in the next section to get the LUN information from each SAN that needs to be monitored.

for san in `/bin/cat /reports/tiers/san.list`

do

naviseccli -h $san lun -list -tiers |grep LUN |awk '{print $2}' > $san.out
 Section 2:

This section will run the naviseccli command for every lun in each of the <san_hostname>.out files, and output a single text file with the tier distribution for every LUN.  If you have 500 LUNs, then 500 text files will be created in the same directory that your run the script in.

     for lun in `cat $san.out`
        do
        sleep 2
        echo $san
        naviseccli -h $san -np lun -list -name $lun -tiers > $lun.$san.dat &
     done
 Each file will be named <lun_name>.dat, and the contents of the file looks like this:
LOGICAL UNIT NUMBER 962
Name:  P1_LUN962_0000_SPB_servername_filesystem_350G
Tier Distribution: 
Flash:  4.74%
FC:  95.26%
 Section 3:

This line simply makes a copy of the previous day’s output file for archiving purposes.  The %j adds the Julian date to the file (which is 1-365, the day of the year), so the files will automatically be overwritten after one year.  It’s a self cleaning archive directory.  🙂

mv $san.report.csv $san.report.`date +%j`.csv

Section 4:

This section then processes each individual LUN file pulling out only the tier information that we need, and then combines the list into one large output file in csv format.

The first line creates a blank CSV file with the appropriate column headers.

echo "LUN Name","FLASH","FC","SATA" > $san.report.csv

This block of code parses each individual LUN file, doing a grep for each column item that we need added to the report, and awk to only grab the specific text that we want from that line.  For example, if the LUN output file has “Flash:  4.74%” in one line, and we only want the “4.74%” and the word “Flash:” stripped off, we would do an awk ‘{print $2}’ to grab only the second line item.

     for lun in `cat  $san.out`
        do
        echo $lun
        echo `grep Name $lun.$san.dat |awk '{print $2}'`","`grep -i flash $lun.$san.dat |awk '{print $2}'`","`grep -i fc $lun.$san.dat |awk '{print $2}'`","`grep -i sata $lun.$san.dat |awk '{print $2}'` >> $san.report.csv
     done
done
 Once every LUN file has been processed and added to the report, I run the csv2html.pl perl script (from http://www.jpsdomain.org/source/perl.html) to add to our intranet website.  The csv files are also added as download links on the site.
./csv2htm.pl -e -T -i /reports/clariion1_hostname.report.csv -o /reports/clariion1_hostname.report.html

./csv2htm.pl -e -T -i /reports/clariion2_hostname.report.csv -o /reports/clariion2_hostname.report.html

./csv2htm.pl -e -T -i /reports/clariion3_hostname.report.csv -o /reports/clariion3_hostname.report.html
 And finally, the output looks like this:
LUN Name FLASH FC SATA
P0_LUN101_0000_SPA_servername_filesystem_100G

24.32%

67.57%

8.11%

P0_LUN102_0000_SPA_servername_filesystem_100G

5.92%

58.77%

35.31%

P1_LUN103_0000_SPA_servername_filesystem_100G

7.00%

81.79%

11.20%

P1_LUN104_0000_SPA_servername_filesystem_100G

1.40%

77.20%

21.40%

P0_LUN200_0000_SPA_servername_filesystem_100G

5.77%

75.06%

19.17%

P0_LUN201_0000_SPA_servername_filesystem_100G

6.44%

71.21%

22.35%

P0_LUN202_0000_SPA_servername_filesystem_100G

4.55%

90.91%

4.55%

P0_LUN203_0000_SPA_servername_filesystem_100G

10.73%

80.76%

8.52%

P0_LUN204_0000_SPA_servername_filesystem_100G

8.62%

88.31%

3.08%

P0_LUN205_0000_SPA_servername_filesystem_100G

10.88%

82.65%

6.46%

P0_LUN206_0000_SPA_servername_filesystem_100G

7.00%

81.79%

11.20%

P0_LUN207_0000_SPA_servername_filesystem_100G

1.40%

77.20%

21.40%

P0_LUN208_0000_SPA_servername_filesystem_100G

5.77%

75.06%

19.17%

Reporting on Trespassed LUNs

 

All of our production clariions are configured with two large tiered storage pools, one for LUNs on SPA and one for LUNs on SPB.  When storage is created on a server, two identical LUNs are created (one in each pool) and are striped at the host level.  I do it that way to more evenly balance the load on the storage processors.

I’ve noticed that LUNs will occassionally trespass to the other SP.  In order to keep the SP’s balanced how I want them, I will routinely check and trespass them back to their default owner.  Our naming convention for LUNs includes the SP that the LUN was initially configured to use, as well as the pool ID, server name, filesystem/drive letter, last four digits of serial number, and size.  Having all of this information in the LUN name makes for very easy reporting.  Having the default SP in the LUN name is required for this script to work as written.

Here’s what our LUN names looks like:     P1_LUN100_SPA_0000_servername_filesystem_150G

To quickly check on the status of any mismatched LUNs every morning, I created a script that generates a daily report.  The script first creates output files that list all of the LUNs on each SP, then uses simple grep commands to output only the LUNs whose SP designation in the name does not match the current owner.   The csv output files are then parsed by the csv2html perl script, which converts the csv into easy to read HTML files that are automatically posted on our intranet web site.  The csv2html perl script is from http://www.jpsdomain.org/source/perl.html and is under a GNU General Public License.  Note that this script is designed to run in unix.  It can be run using cygwin installed on a Windows server if you don’t have access to a unix based server.

Here’s the shell script (I have one for each clariion/VNX):

naviseccli -h clariion_hostname getlun -name -owner |grep -i name > /reports/sp/lunname.out

sleep 5

naviseccli -h clariion_hostname getlun -name -owner |grep -i current >  /reports/sp/currentsp.out

sleep 5

paste -d , /reports/sp/lunname.out /reports/sp/currentsp.out >  /reports/sp/clariion_hostname.spowner.csv

./csv2htm.pl -e -T -i /reports/sp/clariion_hostname.spowner.csv -o /reports/sp/clariion_hostname.spowner.html

#Determine SP mismatches between LUNs and SPs, output to separate files

cat /reports/sp/clariion_hostname.spowner.csv | grep 'SP B' > /reports/sp/clariion_hostname_spb.csv

grep SPA /reports/sp/clariion_hostname_spb.csv > /reports/sp/clariion_hostname_spb_mismatch.csv

cat /reports/sp/clariion_hostname.spowner.csv | grep 'SP A' > /reports/sp/clariion_hostname_spa.csv

grep SPB /reports/sp/clariion_hostname_spa.csv > /reports/sp/clariion_hostname_spa_mismatch.csv

#Convert csv output files to HTML for intranet site

./csv2htm.pl -e -d -T -i /reports/sp/clariion_hostname_spa_mismatch.csv -o /reports/sp/clariion_hostname_spa_mismatch.html

./csv2htm.pl -e -d -T -i /reports/sp/clariion_hostname_spb_mismatch.csv -o /reports/sp/clariion_hostname_spb_mismatch.html
 The output files look like this (clariion_hostname_spa_mismatch.html from the script):
Name: P1_LUN100_SPA_0000_servername_filesystem1_150G       Current Owner: SPB

Name: P1_LUN101_SPA_0000_servername_filesystem2_250G      Current Owner: SPB

Name: P1_LUN102_SPA_0000_servername_filesystem3_350G      Current Owner: SPB

Name: P1_LUN103_SPA_0000_servername_filesystem4_450G
Current Owner: SPB

Name: P1_LUN104_SPA_0000_servername_filesystem5_550G      
Current Owner: SPB
 The 0000 represents the last four digits of the serial number of the Clariion.

That’s it, a quick and easy way to report on trespassed LUNs in our environment.

VNX replication monitoring script

This script allows me to quickly monitor and verify the status of my replication jobs every morning.  It will generate a csv file with six columns for file system name, interconnect, estimated completion time, current transfer size,current transfer size remaining, and current write speed.

I recently added two more remote offices to our replication topology and I like to keep a daily tab on how much longer they have to complete the initial seeding, and it will also alert me to any other jobs that are running too long and might need my attention.

Step 1:

Log in to your Celerra and create a directory for the script.  I created a subdirectory called “scripts” under /home/nasadmin.

Create a text file named ‘replfs.list’ that contains a list of your replicated file systems.  You can cut and paste the list out of Unisphere.

The contents of the file should should look something like this:

Filesystem01
Filesystem02
Filesystem03
Filesystem04
Filesystem05
 Step 2:

Copy and paste all of the code into a text editor and modify it for your needs (the complete code is at the bottom of this post).  I’ll go through each section here with an explanation.

1: The first section will create a text file ($fs.dat) for each filesystem in the replfs.list file you made eariler.

for fs in `cat replfs.list`
         do
         nas_replicate -info $fs | egrep 'Celerra|Name|Current|Estimated' > $fs.dat
         done
 The output will look like this:
Name                                        = Filesystem_01
Source Current Data Port            = 57471
Current Transfer Size (KB)          = 232173216
Current Transfer Remain (KB)     = 230877216
Estimated Completion Time        = Thu Nov 24 06:06:07 EST 2011
Current Transfer is Full Copy      = Yes
Current Transfer Rate (KB/s)       = 160
Current Read Rate (KB/s)           = 774
Current Write Rate (KB/s)           = 3120
 2: The second section will create a blank csv file with the appropriate column headers:
echo 'Name,System,Estimated Completion Time,Current Transfer Size (KB),Current Transfer Remain (KB),Write Speed (KB)' > replreport.csv

3: The third section will parse all of the output files created by the first section, pulling out only the data that we’re interested in.  It places it in columns in the csv file.

         for fs in `cat replfs.list`

         do

         echo $fs","`grep Celerra $fs.dat | awk '{print $5}'`","`grep -i Estimated $fs.dat |awk '{print $5,$6,$7,$8,$9,$10}'`","`grep -i Size $fs.dat |awk '{print $6}'`","`grep -i Remain $fs.dat |awk '{print $6}'`","`grep -i Write $fs.dat |awk '{print $6}'` >> replreport.csv

        done
 If you’re not familiar with awk, I’ll give a brief explanation here.  When you grep for a certain line in the output code, awk will allow you to output only one word in the line.

For example, if you want the output of “Yes” put into a column in the csv file, but the output code line looks like “Current Transfer is Full Copy      = Yes”, then you could pull out only the “Yes” by typing in the following:

 nas_replicate -info Filesystem01 | grep  Full | awk '{print $7}'

Because the word ‘Yes’ is the 7th item in the line, the output would only contain the word Yes.

4: The final section will send an email with the csv output file attached.

uuencode replreport.csv replreport.csv | mail -s "Replication Status Report" user@domain.com

Step 3:

Copy and paste the modified code into a script file and save it.  I have mine saved in the /home/nasadmin/scripts folder. Once the file is created, make it executable by typing in chmod +X scriptfile.sh, and change the permissions with chmod 755 scriptfile.sh.

Step 4:

You can now add the file to crontab to run automatically.  Add it to cron by typing in crontab –e, to view your crontab entries type crontab –l.  For details on how to add cron entries, do a google search as there is a wealth of info available on your options.

Script Code:

for fs in `cat replfs.list`

         do

         nas_replicate -info $fs | egrep 'Celerra|Name|Current|Estimated' > $fs.dat

        done

 echo 'Name,System,Estimated Completion Time,Current Transfer Size (KB),Current Transfer Remain (KB),Write Speed (KB)' > replreport.csv

         for fs in `cat replfs.list`

         do

         echo $fs","`grep Celerra $fs.dat | awk '{print $5}'`","`grep -i Estimated $fs.dat |awk '{print $5,$6,$7,$8,$9,$10}'`","`grep -i Size $fs.dat |awk '{print $6}'`","`grep -i Remain $fs.dat |awk '{print $6}'`","`grep -i Write $fs.dat |awk '{print $6}'` >> replreport.csv

         done

 uuencode replreport.csv replreport.csv | mail -s "Replication Status Report" user@domain.com
 The final output of the script generates a report that looks like the sample below.  Filesystems that have all zeros and no estimated completion time are caught up and not currently performing a data synchronization.
Name System Estimated Completion Time Current Transfer Size (KB) Current Transfer Remain (KB) Write Speed (KB)
SA2Users_03 SA2VNX5500 0 0 0
SA2Users_02 SA2VNX5500 Wed Dec 16 01:16:04 EST 2011 211708152 41788152 2982
SA2Users_01 SA2VNX5500 Wed Dec 16 18:53:32 EST 2011 229431488 59655488 3425
SA2CommonFiles_04 SA2VNX5500 0 0 0
SA2CommonFiles_03 SA2VNX5500 Wed Dec 16 10:35:06 EST 2011 232173216 53853216 3105
SA2CommonFiles_02 SA2VNX5500 Mon Dec 14 15:46:33 EST 2011 56343592 12807592 2365
SA2commonFiles_01 SA2VNX5500 0 0 0

Reporting on Soft media errors

 

Ah, soft media errors.  The silent killer.  We had an issue with one of our Clariion LUNs that had many uncorrectable sector errors.  Prior to the LUN failure, there were hundreds of soft media errors reported in the navisphere logs.  Why weren’t we alerted about them?  Beats me.  I created my own script to pull and parse the alert logs so I can manually check for these type of errors.

What exactly is a soft media error?  Soft Media errors indicate that the SAN has identified a bad sector on the disk and is reconstructing the data from RAID parity data  in order to fulfill the read request.   It can indicate a failing disk.

To run a report that pulls only soft media errors from the SP log, put the following in a windows batch file:

naviseccli -h <SP IP Address> getlog >textfile.txt

for /f "tokens=1,2,3,4,5,6,7,8,9,10,11,12,13,14" %%i in ('findstr Soft textfile.txt') do (echo %%i %%j %%k %%l %%m %%n %%o %%p %%q %%r %%s %%t %%u %%v)  >>textfile_mediaerrors.txt

The text file output looks like this:

10/25/2010 19:40:17 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:22 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:22 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:27 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:27 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:33 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:33 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:38 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:38 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:44 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:44 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:49 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:49 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5

If you see lots of soft media errors, do yourself a favor and open a case with EMC.  Too many can lead to the failure of one of your LUNs.

The script can be automated to run and send an email with daily alerts, if you so choose.  I just run it manually about once a week for review.

Tiering reports for EMC’s FAST VP

Note: On a separate blog post, I shared a script to generate a report of the tiering status of all LUNs.

One of the items that EMC did not implement along with FAST VP is the ability to run a canned report on how your LUNs are being allocated among the different tiers of storage.  While there is no canned report, alas, it is possible to get this information from the CLI.

The naviseccli –h {SP IP or hostname} lun –list –tiers command fits the bill. It shows how a specific LUN is distributed across the different drive types.  I still need to come up with a script to pull out only the information that I want, but the info is definitely in the command’s output.

Here’s the sample output:

LOGICAL UNIT NUMBER 6
 Name:  LUN 6
 Tier Distribution:
 Flash:  13.83%
 FC:  86.17%

The storagepool report gives some good info as well.  Here’s an excerpt of what you see with the naviseccli –h {SP IP or hostname} storagepool –list –tiers command:

SPA

Tier Name:  Flash
 Raid Type:  r_5
 User Capacity (GBs):  1096.07
 Consumed Capacity (GBs):  987.06
 Available Capacity (GBs):  109.01
 Percent Subscribed:  90.05%
 Data Targeted for Higher Tier (GBs):  0.00
 Data Targeted for Lower Tier (GBs):  11.00

Tier Name:  FC
 Raid Type:  r_5
 User Capacity (GBs):  28981.77
 Consumed Capacity (GBs):  10592.65
 Available Capacity (GBs):  18389.12
 Percent Subscribed:  36.55%

Tier Name:  SATA
 Raid Type:  r_5
 User Capacity (GBs):  11004.67
 Consumed Capacity (GBs):  260.02
 Available Capacity (GBs):  10744.66
 Percent Subscribed:  2.36%
 Data Targeted for Higher Tier (GBs):  3.00
 Data Targeted for Lower Tier (GBs):  0.00
 Disks (Type):

SPB

Tier Name:  Flash
 Raid Type:  r_5
 User Capacity (GBs):  1096.07
 Consumed Capacity (GBs):  987.06
 Available Capacity (GBs):  109.01
 Percent Subscribed:  90.05%
 Data Targeted for Higher Tier (GBs):  0.00
 Data Targeted for Lower Tier (GBs):  25.00

Tier Name:  FC
 Raid Type:  r_5
 User Capacity (GBs):  28981.77
 Consumed Capacity (GBs):  10013.61
 Available Capacity (GBs):  18968.16
 Percent Subscribed:  34.55%
 Data Targeted for Higher Tier (GBs):  25.00
 Data Targeted for Lower Tier (GBs):  0.00

Tier Name:  SATA
 Raid Type:  r_5
 User Capacity (GBs):  11004.67
 Consumed Capacity (GBs):  341.02
 Available Capacity (GBs):  10663.65
 Percent Subscribed:  3.10%
 Data Targeted for Higher Tier (GBs):  20.00
 Data Targeted for Lower Tier (GBs):  0.00

Good stuff in there.   It’s on my to-do list to run these commands periodically, and then parse the output to filter out only what I want to see.  Once I get that done I’ll post the script here too.

Note: I did create and post a script to generate a report of the tiering status of all LUNs.