Optimizing Java Memory for Navisphere / Unisphere

If you have a CLARiiON system with a large configuration in terms of disks, LUNs, initiator records, etc, you may experience a slowdown when managing the system with Navisphere or Unisphere.  If you increase the amount of memory that Java can use, you can significantly improve the response time when using the management console.

Here are the steps:

  1. Log in to the CLARiiON setup page (http://<clariion IP>/setup).  Go to Set Update Parameters > Update Interval.  Change it to 300.
  2. On the Management Server (or your local PC/laptop) go to Control Panel and launch the Java icon.
  3. Go to the Java tab and click view.
  4. Enter -Xmx128m under Java Runtime Parameter, which allocates 128MB for Java.  This number can be increased as you see fit, you may see better results with 512 or 1024.
Advertisements

Celerra Health Check with CLI Commands

Here are the first commands I’ll type when I suspect there is a problem with the Celerra, or if I want to do a simple health check.

1. <watch> /nas/sbin/getreason.  This will quickly give you the current status of each data mover. 5=up, 0=down/rebooting.  Typing watch before the command will run the command with continuous updates so you can monitor a datamover if you are purposely rebooting it.

10 – slot_0 primary control station
5 – slot_2 contacted
5 – slot_3 contacted

2. nas_server -list.  This lists all of the datamovers and their current state.  It’s a good way to quickly tell which datamovers are active and which are standby.

1=nas, 2=unused, 3=unused, 4=standby, 5=unused, 6=rdf

id      type  acl  slot groupID  state  name
1        1    0     2                         0    server_2
2        4    0     3                        0    server_3

3. server_sysstat.  This will give you a quick overview of memory and CPU utilization.

server_2 :
threads runnable = 6
threads blocked  = 4001
threads I/J/Z    = 1
memory  free(kB) = 2382807
cpu     idle_%   = 70

4. nas_checkup.   This runs a system health check.

Check Version:  5.6.51.3
Check Command:  /nas/bin/nas_checkup
Check Log    :  /nas/log/checkup-run.110608-143203.log

————————————-Checks————————————-
Control Station: Checking if file system usage is under limit………….. Pass
Control Station: Checking if NAS Storage API is installed correctly…….. Pass

5. server_log server_2.  This shows the current alert log.  Alert logs are also stored in /nas/log/webui.

6. vi /nas/jserver/logs/system_log.   This is the java system log.

7. vi /var/log/messages.  This displays system messages.

Easy File Extension filtering with EMC Celerra

Are your users filling up your CIFS fileserver with MP3 files?  Sick of sending out emails outlining IT policies, asking for their removal?  However your manage it now, the best way to avoid the problem in the first place is to set up filtering on your CIFS server file shares.

So, to use the same example, lets say you don’t want your users to store MP3 files on your \\PRODFILES\Public share.

1. Navigate to the \\PRODFILES\C$ administrative share.

2. Open the folder in the root directory called .filefilter

3. Create an empty text file called mp3@public in the .filefilter folder.

4. Change the windows security on the file to restrict access to certain active directory groups or individuals.

That’s it!  Once the file is created and security is set, users who are restricted by the file security will no longer be able to copy MP3 files to the public share.  Note that this will not remove any existsing MP3 files from the share, it will only prevent new ones from being copied.

A guide for troubleshooting CIFS issues on the Celerra

In my experience, every CIFS issue you may have will fall into 8 basic areas, the first five being the most common.   Check all of these things and I can almost guarantee you will resolve your problem. 🙂

1. CIFS Service.  Check and make sure the CIFS Service is running:  server_cifs server_2 -protocol CIFS -option start

2. DNS.  Check and make sure that your DNS server entries on the Celerra are correct, that you’re configured to point to at least two, and that they are up and running with the DNS Service running.

3. NTP.  Make sure your NTP server entry is correct on the Celerra, and that the IP is reachable on the network and is actively providing NTP services.

4. User Mapping.

5. Default Gateway.  Double check your default gateway in the Celerra’s routing table.  Get the network team involved if you’re not sure.

6. Interfaces.  Make sure the interfaces are physically connected and properly configured.

7. Speed/Duplex.  Make sure the speed and duplex settings on the Celerra match those of the switch port that the interfaces are plugged in to.

8. VLAN.  Double check your VLAN settings on the interfaces, make sure it matches what is configured on the connected switch.

Auto generating daily performance graphs with EMC Control Center / Performance Manager

This document describes the process I used to pull performance data using the ECC pmcli command line tool, parse the data to make it more usable with a graphing tool, and then use perl scripts to automatically generate graphs.

You must install Perl.  I use ActiveState Perl (Free Community Edition) (http://www.activestate.com/activeperl/downloads).

You must install Cygwin.  Link: http://www.cygwin.com/install.html. I generally choose all packages.

I use the follow CPAN Perl modules:

Step 1:

Once you have the software set up, the first step is to use the ECC command line utility to extract the interval performance data that you’re interested in graphing.  Below is a sample PMCLI command line script that could be used for this purpose.

:Get the current date

For /f “tokens=2-4 delims=/” %%a in (‘date /t’) do (set date=%%c%%a%%b)

:Export the interval file for today’s date.

D:\ECC\Client.610\PerformanceManager\pmcli.exe -export -out D:\archive\interval.csv -type interval -class clariion -date %date% -id APM00324532111

:Copy all the export data to my cygwin home directory for processing later.

copy /y e:\san712_interval.csv C:\cygwin\home\<userid>

You can schedule the command script above to run using windows task scheduler.  I run it at 11:46PM every night, as data is collected on our SAN in 15 minute intervals, and that gives me a file that reports all the way up to the end of one calendar day.

Note that there are 95 data collection points from 00:15 to 23:45 every day if you collect data at 15 minute intervals.  The storage processor data resides in the last two lines of the output file.

Here is what the output file looks like:

EMC ControlCenter Performance manager generated file from: <path>

Data Collected for DiskStats

Data Collected for DiskStats – 0_0_0

                                                             3/28/11 00:15       3/28/11 00:30      3/28/11  00:45      3/28/11 01:00 

Number of Arrivals with Non Zero Queue     12                         20                        23                      23 

% Utilization                                                30.2                     33.3                     40.4                  60.3

Response Time                                              1.8                        3.3                        5.4                     7.8

Read Throughput IO per sec                        80.6                    13.33                   90.4                    10.3

Great information in there, but the format of the data makes it very hard to do anything meaningful with the data in an excel chart.  If I want to chart only % utilization, that data is difficult to chart because there are so many counters around it that are also have data collected on them.   My next goal was to write a script to reformat the data in a much more usable format to automatically create a graph for one specific counter that I’m interested in (like daily utilization numbers), which could then be emailed daily or auto-uploaded to an internal website.

Step 2:

Once the PMCLI data is exported, the next step is to use cygwin bash scripts to parse the csv file and pull out only the performance data that is needed.  Each SAN will need a separate script for each type of performance data.  I have four scripts configured to run based on the data that I want to monitor.  The scripts are located in my cygwin home directory.

The scripts I use:

  • Iostats.sh (for total IO throughput)
  • Queuestats.sh (for disk queue length)
  • Resptime.sh (for disk response time in ms)
  • Utilstats.sh (for % utilization)

Here is a sample shell script for parsing the CSV export file (iostats.sh):

#!/usr/bin/bash

#This will pull only the timestamp line from the top of the CSV output file. I’ll paste it back in later.

grep -m 1 “/” interval.csv > timestamp.csv

#This will pull out only lines that begin with “total througput io per sec”.

grep -i “^Total Throughput IO per sec” interval.csv >> stats.csv

#This will pull out the disk/LUN title info for the first column.  I’ll add this back in later.

grep -i “Data Collected for DiskStats -” interval.csv > diskstats.csv

grep -i “Data Collected for LUNStats -” interval.csv > lunstats.csv

#This will create a column with the disk/LUN number .  I’ll paste it into the first column later.

cat diskstats.csv lunstats.csv > data.csv

#This adds the first column (disk/LUN) and combines it with the actual performance data columns.

paste data.csv stats.csv > combined.csv

#This combines the timestamp header at the top with the combined file from the previous step to create the final file we’ll use for the graph.  There is also a step to append the current date and copy the csv file to an archive directory.

cat timestamp.csv combined.csv > iostats.csv

cp iostats.csv /cygdrive/e/SAN/csv_archive/iostats_archive_$(date +%y%m%d).csv

#  This removes all the temporary files created earlier in the script.  They’re no longer needed.

rm timestamp.csv

rm stats.csv

rm diskstats.csv

rm lunstats.csv

rm data.csv

rm combined.csv

#This strips the last two lines of the CSV (Storage Processor data).  The resulting file is used for the “all disks” spreadsheet.  We don’t want the SP
data to skew the graph.  This CSV file is also copied to the archive directory.

sed ‘$d’ < iostats.csv > iostats2.csv

sed ‘$d’ < iostats2.csv > iostats_disk.csv

rm iostats2.csv

cp iostats_disk.csv /cygdrive/e/SAN/csv_archive/iostats_disk_archive_$(date +%y%m%d).csv

Note: The shell script above can be run in the windows task scheduler as long as you have cygwin installed.  Here’s the syntax:

c:\cygwin\bin\bash.exe -l -c “/home/<username>/iostats.sh”

After running the shell script above, the resulting CSV file contains only Total Throughput (IO per sec) data for each disk and lun.  It will contain data from 00:15 to 23:45 in 15 minute increments.  After the cygwin scripts have run we will have csv datasets that are ready to be exported to a graph.

The Disk and LUN stats are combined into the same CSV file.  It is entirely possible to rewrite the script to only have one or the other.  I put them both in there to make it easier to manually create a graph in excel for either disk or lun stats at a later time (if necessary).  The “all disks graph” does not look any different with both disk and lun stats in there, I tried it both ways and they overlap in a way that makes the extra data indistinguishable in the image.

The resulting data output after running the iostats.sh script is shown below.  I now have a nice, neat excel spreadsheet that lists the total throughput for each disk in the array for the entire day in 15 minute increments.   Having the data formatted in this way makes it super easy to create charts.  But I don’t want to have to do that manually every day, I want the charts to be created automatically.

                                                             3/28/11 00:15       3/28/11 00:30      3/28/11  00:45      3/28/11 01:00

Total Throughput IO per sec   – 0_0_0          12                             20                             23                           23 

Total Throughput IO per sec    – 0_0_1        30.12                        33.23                        40.4                         60.23

Total Throughput IO per sec    – 0_0_2         1.82                          3.3                           5.4                              7.8

Total Throughput IO per sec    -0_0_3         80.62                        13.33                        90.4                         10.3 

Step 3:

Now I want to automatically create the graphs every day using a Perl script.  After the CSV files are exported to a more usable format from the previous step, I Use the GD::Graph library from CPAN (http://search.cpan.org/~mverb/GDGraph-1.43/Graph.pm) to auto-generate the graphs.

Below is a sample Perl script that will autogenerate a great looking graph based on the CSV ouput file from the previous step.

#!/usr/bin/perl

#Declare the libraries that will be used.

use strict;

use Text::ParseWords;

use GD::Graph::lines;

use Data::Dumper;

#Specify the csv file that will be used to create the graph

my $file = ‘C:\cygwin\home\<username>\iostats_disk.csv’;

#my $file  = $ARGV[0];

my ($output_file) = ($file =~/(.*)\./);

#Create the arrays for the data and the legends

my @data;

my @legends;

#parse csv, generate an error if it fails

open(my $fh, ‘<‘, $file) or die “Can’t read csv file ‘$file’ [$!]\n”;

my $countlines = 0;

while (my $line = <$fh>) {

chomp $line;

my @fields = Text::ParseWords::parse_line(‘,’, 0, $line);

#There are 95 fields generated to correspond to the 95 data collection points in each
of the output files.

my @field =

(@fields[1],@fields[2],@fields[3],@fields[4],@fields[5],@fields[6],@fields[7],@fields[8],@fields[9],@fields[10],@fields[11],@fields[12],@fields[13],@fields[14],@fields[15],@fields[16],@fields[17],@fields[18],@fields[19],@fields[20],@fields[21],@fields[22],@fields[23],@fields[24],@fields[25],@fields[26],@fields[27],@fields[28],@fields[29],@fields[30],@fields[31],@fields[32],@fields[33],@fields[34],@fields[35],@fields[36],@fields[37],@fields[38],@fields[39],@fields[40],@fields[41],@fields[42],@fields[43],@fields[44],@fields[45],@fields[46],@fields[47],@fields[48],@fields[49],@fields[50],@fields[51],@fields[52],@fields[53],@fields[54],@fields[55],@fields[56],@fields[57],@fields[58],@fields[59],@fields[60],@fields[61],@fields[62],@fields[63],@fields[64],@fields[65],@fields[66],@fields[67],@fields[68],@fields[69],@fields[70],@fields[71],@fields[72],@fields[3],@fields[74],@fields[75],@fields[76],@fields[77],@fields[78],@fields[79],@fields[80],@fields[81],@fields[82],@fields[83],@fields[84],@fields[85],@fields[86],@fields[87],@fields[88],@fields[89],@fields[90],@fields[91],@fields[92],@fields[93],@fields[94],@fields[95]);
push @data, \@field;

if($countlines >= 1){

push @legends, @fields[0];

}

$countlines++;

}

#The data and legend arrays will read 820 lines of the CSV file.  This number will change based on the number of disks in the SAN, and will be different depending on the SAN being reported on.  The legend info will read the first column of the spreadsheet and create a color box that corresponds to the graph line.  For the purpose of this graph, I won’t be using it because 820+ legend entries look like a mess on the screen.

splice @data, 1, -820;

splice @legends, 0, -820;

#Set Graphing Options

my $mygraph = GD::Graph::lines->new(1024, 768);

# There are many graph options that can be changed using the GD::Graph library.  Check the website (and google) for lots of examples.

$mygraph->set(

title => ‘SP IO Utilization (00:15 – 23:45)’,

y_label => ‘IOs Per Second’,

y_tick_number => 4,

values_vertical => 6,

show_values => 0,

x_label_skip => 3,

) or warn $mygraph->error;

#As I said earlier, because of the large number of legend entries for this type of graph, I change the legend to simply read “All disks”.  If you want the legend to actually put the correct entries and colors, use this line instead:  $mygraph->set_legend(@legends);

$mygraph->set_legend(‘All Disks’);

#Plot the data

my $myimage = $mygraph->plot(\@data) or die $mygraph->error;

# Export the graph as a gif image.  The images are currently moved to the IIS folder (c:\inetpub\wwwroot) with one of the scripts.  The could also be emailed using a sendmail utility.

my $format = $mygraph->export_format;

open(IMG,”>$output_file.$format”) or die $!;

binmode IMG;

print IMG $myimage->gif;

close IMG;

After this script runs the resulting image file will be saved in the cygwin home directory (It saves it in the same directory that the CSV file is located in).  One of the nightly scripts I run will copy the image to our interal IIS server’s image directory, and sendmail will email the graph to the SAN Admin team.

That’s it!  You now have lots of pretty graphs with which you can impress your management team. 🙂

Here is a sample graph that was generated with the Perl script:

Filesystem Alignment

You’re likely to have seen the filesystem alignment check fail on most, if not all, of the EMC HEAT reports that you run on your windows 2003 servers.  The starting offset for partition 1 should optimally be a multiple of 128 sectors.  So, how do you fix this problem, and what does it mean?

If you align the partition to 128 blocks (or 64KB as each block is 512bytes) then you don’t cross a track boundary and thereby issue the minimum number of IOs.   Issuing the minimum number of IOs sounds good, right? 🙂

Because NTFS reserves 31.5 KB of signature space, if a LUN has an element size of 64 KB with the default alignment offset of 0 (both are default Navisphere settings), a 64 KB write to that LUN would result in a disk crossing even though it would seem to fit perfectly on
the disk.  A disk crossing can also be referred to as a split IO because the read or write must be split into two or more segments. In this case, 32.5 KB would be written to the first disk and 31.5 KB would be written to the following disk, because the beginning of the stripe is offset by 31.5 KB of signature space. This problem can be avoided by providing the correct alignment offset.  Each alignment offset value represents one block.  Therefore, EMC recommends setting the alignment offset value to 63, because 63 times 512 bytes is 31.5 KB.

Checking your offset:

1. Launch System Information in windows (msinfo32.exe)

2. Select Components -> Storage -> Disks.

3. Scroll to the bottom and you will see the partition starting offset information.  This number needs to be perfectly divisible by 4096, if it’s not then your partition is not properly aligned.

Correcting your starting offset:

Launch diskpart:

C:\>diskpart

DISKPART> list disk

Two disks should be listed

DISKPART> select disk 1

This selects the second disk drive

DISKPART> list partitions

This step should give a message “There are no partitions on this disk to show”.  This confirms a blank disk.

DISKPART> create partition primary align=64

That’s it.  You now have a perfectly aligned disk.

A guide for troubleshooting CIFS issues on the Celerra

In my experience, every CIFS issue you may have will fall into 8 basic areas, the first five being the most common. Check all of these things and I can almost guarantee you will resolve your problem. 🙂

1. CIFS Service. Check and make sure the CIFS Service is running: server_cifs server_2 -protocol CIFS -option start

2. DNS. Check and make sure that your DNS server entries on the Celerra are correct, that you’re configured to point to at least two, and that they are up and running with the DNS Service running.

3. NTP. Make sure your NTP server entry is correct on the Celerra, and that the IP is reachable on the network and is actively providing NTP services.

4. User Mapping.

5. Default Gateway. Double check your default gateway in the Celerra’s routing table. Get the network team involved if you’re not sure.

6. Interfaces. Make sure the interfaces are physically connected and properly configured.

7. Speed/Duplex. Make sure the speed and duplex settings on the Celerra match those of the switch port that the interfaces are plugged in to.

8. VLAN. Double check your VLAN settings on the interfaces, make sure it matches what is configured on the connected switch.

Reporting on Soft media errors

 

Ah, soft media errors.  The silent killer.  We had an issue with one of our Clariion LUNs that had many uncorrectable sector errors.  Prior to the LUN failure, there were hundreds of soft media errors reported in the navisphere logs.  Why weren’t we alerted about them?  Beats me.  I created my own script to pull and parse the alert logs so I can manually check for these type of errors.

What exactly is a soft media error?  Soft Media errors indicate that the SAN has identified a bad sector on the disk and is reconstructing the data from RAID parity data  in order to fulfill the read request.   It can indicate a failing disk.

To run a report that pulls only soft media errors from the SP log, put the following in a windows batch file:

naviseccli -h <SP IP Address> getlog >textfile.txt

for /f "tokens=1,2,3,4,5,6,7,8,9,10,11,12,13,14" %%i in ('findstr Soft textfile.txt') do (echo %%i %%j %%k %%l %%m %%n %%o %%p %%q %%r %%s %%t %%u %%v)  >>textfile_mediaerrors.txt

The text file output looks like this:

10/25/2010 19:40:17 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:22 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:22 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:27 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:27 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:33 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:33 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:38 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:38 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:44 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:44 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:49 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5
 10/25/2010 19:40:49 Enclosure 6 Disk 7 (820) Soft Media Error [0x00] 0 5

If you see lots of soft media errors, do yourself a favor and open a case with EMC.  Too many can lead to the failure of one of your LUNs.

The script can be automated to run and send an email with daily alerts, if you so choose.  I just run it manually about once a week for review.

Tiering reports for EMC’s FAST VP

Note: On a separate blog post, I shared a script to generate a report of the tiering status of all LUNs.

One of the items that EMC did not implement along with FAST VP is the ability to run a canned report on how your LUNs are being allocated among the different tiers of storage.  While there is no canned report, alas, it is possible to get this information from the CLI.

The naviseccli –h {SP IP or hostname} lun –list –tiers command fits the bill. It shows how a specific LUN is distributed across the different drive types.  I still need to come up with a script to pull out only the information that I want, but the info is definitely in the command’s output.

Here’s the sample output:

LOGICAL UNIT NUMBER 6
 Name:  LUN 6
 Tier Distribution:
 Flash:  13.83%
 FC:  86.17%

The storagepool report gives some good info as well.  Here’s an excerpt of what you see with the naviseccli –h {SP IP or hostname} storagepool –list –tiers command:

SPA

Tier Name:  Flash
 Raid Type:  r_5
 User Capacity (GBs):  1096.07
 Consumed Capacity (GBs):  987.06
 Available Capacity (GBs):  109.01
 Percent Subscribed:  90.05%
 Data Targeted for Higher Tier (GBs):  0.00
 Data Targeted for Lower Tier (GBs):  11.00

Tier Name:  FC
 Raid Type:  r_5
 User Capacity (GBs):  28981.77
 Consumed Capacity (GBs):  10592.65
 Available Capacity (GBs):  18389.12
 Percent Subscribed:  36.55%

Tier Name:  SATA
 Raid Type:  r_5
 User Capacity (GBs):  11004.67
 Consumed Capacity (GBs):  260.02
 Available Capacity (GBs):  10744.66
 Percent Subscribed:  2.36%
 Data Targeted for Higher Tier (GBs):  3.00
 Data Targeted for Lower Tier (GBs):  0.00
 Disks (Type):

SPB

Tier Name:  Flash
 Raid Type:  r_5
 User Capacity (GBs):  1096.07
 Consumed Capacity (GBs):  987.06
 Available Capacity (GBs):  109.01
 Percent Subscribed:  90.05%
 Data Targeted for Higher Tier (GBs):  0.00
 Data Targeted for Lower Tier (GBs):  25.00

Tier Name:  FC
 Raid Type:  r_5
 User Capacity (GBs):  28981.77
 Consumed Capacity (GBs):  10013.61
 Available Capacity (GBs):  18968.16
 Percent Subscribed:  34.55%
 Data Targeted for Higher Tier (GBs):  25.00
 Data Targeted for Lower Tier (GBs):  0.00

Tier Name:  SATA
 Raid Type:  r_5
 User Capacity (GBs):  11004.67
 Consumed Capacity (GBs):  341.02
 Available Capacity (GBs):  10663.65
 Percent Subscribed:  3.10%
 Data Targeted for Higher Tier (GBs):  20.00
 Data Targeted for Lower Tier (GBs):  0.00

Good stuff in there.   It’s on my to-do list to run these commands periodically, and then parse the output to filter out only what I want to see.  Once I get that done I’ll post the script here too.

Note: I did create and post a script to generate a report of the tiering status of all LUNs.

VNX NAS CLI Command Reference Guide

vnx1.jpg

Other CLI Reference Guides:
Isilon CLI  |  EMC ECS CLI  |  VNX NAS CLI  |  ViPR Controller CLI  NetApp Clustered ONTAP CLI  |  Data Domain CLI  |  Brocade FOS CLI

This VNX NAS CLI reference guide includes command syntax samples for more commonly used commands at the top, and a list of available commands at the bottom with a brief description of their function.  Here are some other posts on my blog that provide more specific examples of using some CLI commands, with additional detail and some scripting examples:

Undocumented VNX CLI Commands
Using the Database Query option with the VNX NAS CLI
Celerra Health Check CLI Commands
Testing Disaster Recovery with VDM’s and CIFS Servers
Checking Replication Job Throughput with the CLI
Collecting info on Actives Shares, Clients, Protocols, & Authentication with the CLI
Listing and Counting Multiprotocol File Systems from the CLI

VNX NAS CLI Command Reference (Updated January 2018):

NAS Commands:
nas_disk   -list Lists the disk table
nas_checkup Runs a system health check.
nas_pool   -size -all Lists available space on each defined storage pool
nas_replicate  -info –all | grep <fs> Info about each filesystem’s replication status, grep to view just one.
nas_replicate  -list A list of all current replications
nas_server  -list Lists all datamovers. 1=primary,4=standby,6=rdf (remote data facility)
<watch> /nas/sbin/getreason Shows current status of each datamover. 5=up, 0=down or rebooting
nas_fs Creates, deletes, extends, modifies, and lists filesystems.
nas_config Control station configuration (requires root login)
nas_version View current nas revision
nas_ckpt_schedule Manage  checkpoint schedule
nas_storage -list List the attached backend storage systems (with ID’s)
nas_storage -failback id=<x> Fail back failed over SP’s or disks
nas_server  -vdm <vdm_name> -setstate loaded Loads a VDM
nas_server  -vdm <vdm_name> -setstate mounted Unloads a VDM
/nas/sbin/t2reset pwron -s Powers on a data mover that has been shut down.
Server commands:
server_cpu server_<x> -r now Reboots a datamover
server_ping <IP> Ping any IP from the control station
server_ifconfig server_2 –all View all configured interfaces
server_route server_2 {-list,flush,add,delete} Routing table commands
server_mount Mount a filesystem
server_export Export a filesystem
server_stats Provides realtime stats for a datamover, many different options.
server_sysconfig Modifies hardware config of the data movers.
server_devconfig Configures devices on the data movers.
server_sysstat Shows current Memory, CPU, and thread utilization
server_log server_2 Shows current log
vi /nas/jserver/logs/system_log Java System log
vi /var/log/messages System Messages
server_ifconfig server_2 <interface_name> up Bring up a specific interface
server_ifconfig server_2 <interface_name> down Take a specific interface down
server_date Sets system time and NTP server settings
server_date <server_X> timesvc start ntp Starts NTP on a data mover
server_date <server_X> timesvc stats ntp To view the status of NTP.
server_date <server_X> timesvc update ntp Forces an update of NTP
server_file FTP equivalent command
server_dns Configure DNS
server_cifssupport Support services for CIFS users
nas_ckpt_schedule -create <ckpt_fs_name> -filesystem -recurrence once To create a single Checkpoint
fs_ckpt <ckpt_fs_name> -name -Create -readonly n To create a Read/Write copy of a single Checkpoint
server_export [vdm] -P cifs -name [filesystem]_ckpt1 -option netbios=[cifserver] [filesystem]_ckpt1_writeable1 To export a Read/Write checkpoint copy to a CIFS Share
server_cifs server_2 -Join compname=SERVERNAME,domain=DOMAIN.COM,admin=ADMINID Join a CIFS Server to the domain
server_cifs server_2 -Unjoin compname=SERVERNAME,domain=DOMAIN.COM,admin=ADMINID Unjoin a CIFS Server to the domain
 .server_config server_2 -v “pdc dump” To view the current domain controllers visible on the data mover
.server_config server_2 -v “pdc enable=<ip_address>” Enable a domain controller
.server_config server_2 -v “pdc disable=<ip_address>” Disable a domain controller
server_setup server_2 -P cifs -o stop Stop CIFS Service
server_setup server_2 -P cifs -o start Start CIFS Service
server_iscsi server_2 -service -start Start iSCSI service
server_iscsi server_2 -service -stop Stop iSCSI service
server_iscsi server_2 -service -status Check the status of the iSCSI service
.server_config  server_x  “logsys set  severity  NDMP=LOG_DBG2” Enable NDMP Logging [run both of these commands]
.server_config  server_x  “logsys set  severity  PAX=LOG_DBG2” Enable NDMP Logging [run both of these commands]
.server_config  server_x  “logsys  set severity  NDMP=LOG_ERR” Disable NDMP Logging [run both of these commands]
.server_config  server_x  “logsys set severity   PAX=LOG_ERR” Disable NDMP Logging [run both of these commands]
server_netstat server_x -i               Interface statistics Gather interface performance statistics
server_sysconfig server_x -v         Lists virtual devices List Virtual Devices
server_sysconfig server_x -v -i vdevice_name  Informational stats on the virtual device Informational stats on the virtual device
server_netstat server_x -s -a tcp  Retransmissions Retransmissions
server_nfsstat server_x                    NFS SRTs NFS SRTs
server_nfsstat server_x -zero        Reset NFS stats Reset NFS Stats
To view HBA Statistics:
.server_config server_2 -v “printstats fcp reset” View HBA Stats:  Toggles the service on/off
.server_config server_2 -v “printstats fcp full” View HBA Stats:  View stats table (must wait for some stats to collect before viewing)
Filesystem specific commands:
fs_ckpt Manage Checkpoints
fs_dhsm Manage File Mover
fs_group Manage filesystem groups
Complete List of  “nas_”  Commands:
nas_acl Creates, lists, and displays information for access control level entries within the table
nas_ckpt_schedule Manages SnapSure checkpoint scheduling for the VNX
nas_dbtable Displays the table records of the Control Station.
nas_emailuser Manages email notifications for serious system events
nas_inventory Provides detailed information about hardware components
nas_pool Manages the user-defined and system-defined storage pools
nas_slice Manage Slices
nas_task Manages in-progress or completed tasks
nas_automountmap Creates and displays an automount map containing all permanently exported file systems
nas_cmd nas_cmd
nas_devicegroup Manages an established MirrorView/Synchronous consistency group
nas_event Provides a user interface to system-wide events
nas_license Enables software packages.
nas_quotas Manages quotas for mounted file systems.
nas_stats Manages Statistics Groups.
nas_version Displays the software version running on the Control Station.
nas_cel Performs management of remotely linked VNX or a linked pair of Data Movers.
nas_copy Creates a replication session for a one-time copy of a file system.
nas_disk Manages the disk table.
nas_fs Manages local file systems for the VNX.
nas_logviewer Displays the content of nas_eventlog generated log files.
nas_replicate Manages loopback, local, and remote VNX Replicator sessions.
nas_storage Controls storage system access and performs some management tasks
nas_volume Manages the volume table.
nas_checkup Provides a system health checkup for the VNX.
nas_cs Manages the configuration properties of the Control Station.
nas_diskmark Queries the system, manages and lists the SCSI devices configuration.
nas_fsck Manages fsck and aclchk utilities on specified file systems.
nas_message Displays message description.
nas_server Manages the Data Mover (server) table.
nas_symm nas_symm
nas_xml nas_xml
Complete list of  “server_”  Commands:
server_archive Reads and writes file archives, and copies directory hierarchies.
server_cifssupport Provides support services for CIFS users.
server_file Copies files between the Control Station and the specified Data Movers.
server_log Displays the log generated by the specified Data Mover.
server_name Manages the name for the specified Data Movers.
server_ping6 Checks the IPv6 network connectivity for the specified Data Movers.
server_sysconfig Manages the hardware configuration for the specified Data Mover(s).
server_vtlu Configures a virtual tape library unit (VTLU) on the specified Data Movers
server_arp Manages the Address Resolution Protocol (ARP) table for the Data Movers.
server_cpu Performs an orderly, timed, or immediate halt or reboot of a Data Mover.
server_ftp Configures the FTP server configuration for the specified Data Movers.
server_mgr server_mgr (deprecated?)
server_netstat Displays the network statistics for the specified Data Mover.
server_rip Manages the Routing Information Protocol (RIP) configuration
server_sysstat Displays the operating system statistics for the specified Data Movers.
server_cdms Provides File Migration Service for VNX functionality
server_date Displays or sets the date and time for a Data Mover, and synchronizes time
server_http Configures the HTTP configuration file for independent services
server_mount Mounts file systems and manages mount options
server_nfs Manages the NFS service, including secure NFS and NVSv4
server_route Manages the routing table for the specified Data Movers.
server_tftp Manages the Trivial File Transfer Protocol (TFTP)
server_cepp Manages the Common Event Publishing Agent (CEPA) service
server_dbms Enables backup and restore of databases, displays database environment statistics.
server_ifconfig Manages the network interface configuration
server_mountpoint Manages mount points for the specified Data Movers.
server_nfsstat server_nfsstat (deprecated?)
server_security Manages GPO Policy settings for CIFS Servers
server_umount Unmounts file systems
server_certificate Manages VNX for file system’s Public Key Infrastructure (PKI)
server_devconfig Queries, saves, and displays the SCSI over Fibre Channel device configuration
server_ip Manages the IPv6 neighbor cache and route table for VNX.
server_mpfs Sets up and configures MPFS protocol.
server_nis Manages the Network Information Service (NIS) configuration
server_setup Manages the type and protocol component for the specified Data Movers.
server_uptime Displays the length of time that a specified Data Mover has been running since the last reboot
server_checkup Checks the configuration parameters, and state of a Data Mover and its dependencies
server_df Reports free and used disk space and inodes for mounted file systems
server_iscsi server_iscsi (deprecated?)
server_mpfsstat server_mpfsstat (deprecated?)
server_param Manages parameter information for the specified Data Movers.
server_snmpd Manages the Simple Network Management Protocol (SNMP) config values
server_usermapper Provides an interface to manage the Internal Usermapper service.
server_cifs Manages the CIFS configuration for the specified Data Movers or VDMs
server_dns Manages the Domain Name System (DNS) lookup server config
server_kerberos Manages the Kerberos configuration within the specified Data Movers.
server_mt Manages the magnetic tape drive for the specified Data Mover.
server_pax Displays and resets backup and restore statistics and file system information for a backup session already in progress.
server_standby Manages the standby and RDF relationships for the specified Data Movers.
server_version Displays the software version running on the specified Data Movers.
server_cifsstat server_cifsstat (deprecated?)
server_export Exports file systems and manages access on the specified Data Movers for NFS/CIFS clients
server_ldap Manages the LDAP-based directory client configuration and LDAP over SSL
server_muxconfig server_muxconfig (deprecated?)
server_ping Checks the network connectivity for the specified Data Movers.
server_stats Displays sets of statistics that are running on the specified Data Mover.
server_viruschk Manages the virus checker configuration for the specified Data Movers.
Complete list of  “fs_” Commands:
fs_ckpt Manages checkpoints using the EMCSnapSure functionality.
fs_dedupe Manages filesystem deduplication state.
fs_dhsm Manages the VNX FileMover file system connections.
fs_group Creates a file system group from the specified file systems or a single file system
fs_rdf Manages the Remote Data Facility (RDF) functionality for a file system residing on RDF drives.
fs_timefinder Manages the TimeFinderTM/FS functionality for the specified filesystem

A Roundup of Storage Startups

Blockchain and Enterprise Storage

The SAN Guy

I’ve been in IT for over 17 years now, and have a few tricks up my sleeve after all this time.  My most recent job transition in this field has been to a full time SAN Administrator.  It’s a job function I’ve performed for 8+ years, but for the past four years it’s been my exclusive responsibility.

I work for a global company with EMC SAN hardware deployed in many countries around the globe.  I’m responsible for the management and administration of all of this hardware, primarily Clariion and Celerra hardware.  Because EMC doesn’t always provide all of the tools you need, sometimes administrators like us have to get a bit creative to get the job done. 😉

I’ll be posting some tips and tricks I’ve picked up about EMC SAN administration here, as well as some simple, “work smarter not harder” tips for every day tasks, and maybe just some general info as well.