Category Archives: Celerra / VNX File specific

DM Interconnect failure with Celerra Replicator

We just installed a new VNX 5500 a few weeks ago in the UK, and i intially set up a VDM replication job between it and it’s replication partner, an NS-960 in Canada.  The setup went fine with no errors, and replication of the VDM has completed successfully every day up until yesterday when I noticed that the status on the main replications screen says “network communication has been lost”.   I am able to use the server_ping command to ping the data mover/replication interface from UK to Canada, so network connectivity appears to be ok.

I was attempting to set up new replication jobs for the filesystems on this VDM, and the background tasks to create the replication jobs are stuck at “Establishing communication with secondary side for Create task” with a status of “Incomplete”.

I went to the DM interconnect next to validate that it was working, and the validation test failed with the following message: “Validate Data Mover Interconnect server_2:<SAN_name>. The following interfaces cannot connect: source interface=10.x.x.x destination interface=10.x.x.x, Message_ID=13160415446: Authentication failed for DIC communication.”

So, why is the DM Interconnect is failing?   It was working fine for several weeks!

My next trip was to the server log (>server_log server_2) where I spotted another issue.  Hundreds of entries that looked just like these:

2011-07-07 16:32:07: CMD: 6: CmdReplicatev2ReversePri::startSecondary dicSt 16 cmdSt 214
2011-07-07 16:32:10: CIC: 3: <DicXmlSyncMsgService> Sending Cmd to 10.x.x.x failed (16=Bad authentication)
2011-07-07 16:32:10: CMD: 3: DicXmlSyncRequest::sendMessage sendCmd failed:16
2011-07-07 16:32:12: CIC: 3: <DicXmlSyncMsgService> Sending Cmd to 10.x.x.x failed (16=Bad authentication)
2011-07-07 16:32:12: CMD: 3: DicXmlSyncRequest::sendMessage sendCmd failed:16

Bad Authentication? Hmmm.  There is something amiss with the trusted relationship between the VNX and the NS960.  I did a quick read of EMC’s VNX replication manual (yep, rtfm!) and found the command to update the interconnect, nas_cel.

First, run nas_cel -list to view all of your interconnects, noting the ID number of the one you’re having difficulty with.

[nasadmin@<name> ~]$ nas_cel -list
id    name          owner mount_dev  channel    net_path                                      CMU
0     <name_1>  0                               10.x.x.x                                   APM007039002350000
2     <name_2>      0                           10.x.x.x                                   APM001052420000000
4     <name_3>      0                           10.x.x.x                                   APM009015016510000
5     <name_4>       0                           10.x.x.x                                  APM000827205690000

In this case, I was having trouble with <name_3>, which is ID 4.

Run this command next:  nas_cel -update id=4.   After that command completed, my interconnect immediately started working and I was able to create new replication jobs.

Celerra Health Check with CLI Commands

Here are the first commands I’ll type when I suspect there is a problem with the Celerra, or if I want to do a simple health check.

1. <watch> /nas/sbin/getreason.  This will quickly give you the current status of each data mover. 5=up, 0=down/rebooting.  Typing watch before the command will run the command with continuous updates so you can monitor a datamover if you are purposely rebooting it.

10 – slot_0 primary control station
5 – slot_2 contacted
5 – slot_3 contacted

2. nas_server -list.  This lists all of the datamovers and their current state.  It’s a good way to quickly tell which datamovers are active and which are standby.

1=nas, 2=unused, 3=unused, 4=standby, 5=unused, 6=rdf

id      type  acl  slot groupID  state  name
1        1    0     2                         0    server_2
2        4    0     3                        0    server_3

3. server_sysstat.  This will give you a quick overview of memory and CPU utilization.

server_2 :
threads runnable = 6
threads blocked  = 4001
threads I/J/Z    = 1
memory  free(kB) = 2382807
cpu     idle_%   = 70

4. nas_checkup.   This runs a system health check.

Check Version:  5.6.51.3
Check Command:  /nas/bin/nas_checkup
Check Log    :  /nas/log/checkup-run.110608-143203.log

————————————-Checks————————————-
Control Station: Checking if file system usage is under limit………….. Pass
Control Station: Checking if NAS Storage API is installed correctly…….. Pass

5. server_log server_2.  This shows the current alert log.  Alert logs are also stored in /nas/log/webui.

6. vi /nas/jserver/logs/system_log.   This is the java system log.

7. vi /var/log/messages.  This displays system messages.

Easy File Extension filtering with EMC Celerra

Are your users filling up your CIFS fileserver with MP3 files?  Sick of sending out emails outlining IT policies, asking for their removal?  However your manage it now, the best way to avoid the problem in the first place is to set up filtering on your CIFS server file shares.

So, to use the same example, lets say you don’t want your users to store MP3 files on your \\PRODFILES\Public share.

1. Navigate to the \\PRODFILES\C$ administrative share.

2. Open the folder in the root directory called .filefilter

3. Create an empty text file called mp3@public in the .filefilter folder.

4. Change the windows security on the file to restrict access to certain active directory groups or individuals.

That’s it!  Once the file is created and security is set, users who are restricted by the file security will no longer be able to copy MP3 files to the public share.  Note that this will not remove any existsing MP3 files from the share, it will only prevent new ones from being copied.

A guide for troubleshooting CIFS issues on the Celerra

In my experience, every CIFS issue you may have will fall into 8 basic areas, the first five being the most common.   Check all of these things and I can almost guarantee you will resolve your problem. 🙂

1. CIFS Service.  Check and make sure the CIFS Service is running:  server_cifs server_2 -protocol CIFS -option start

2. DNS.  Check and make sure that your DNS server entries on the Celerra are correct, that you’re configured to point to at least two, and that they are up and running with the DNS Service running.

3. NTP.  Make sure your NTP server entry is correct on the Celerra, and that the IP is reachable on the network and is actively providing NTP services.

4. User Mapping.

5. Default Gateway.  Double check your default gateway in the Celerra’s routing table.  Get the network team involved if you’re not sure.

6. Interfaces.  Make sure the interfaces are physically connected and properly configured.

7. Speed/Duplex.  Make sure the speed and duplex settings on the Celerra match those of the switch port that the interfaces are plugged in to.

8. VLAN.  Double check your VLAN settings on the interfaces, make sure it matches what is configured on the connected switch.

VNX NAS CLI Command Reference Guide

vnx1.jpg

Other CLI Reference Guides:
Isilon CLI  |  EMC ECS CLI  |  VNX NAS CLI  |  ViPR Controller CLI  NetApp Clustered ONTAP CLI  |  Data Domain CLI  |  Brocade FOS CLI

This VNX NAS CLI reference guide includes command syntax samples for more commonly used commands at the top, and a list of available commands at the bottom with a brief description of their function.  Here are some other posts on my blog that provide more specific examples of using some CLI commands, with additional detail and some scripting examples:

Undocumented VNX CLI Commands
Using the Database Query option with the VNX NAS CLI
Celerra Health Check CLI Commands
Testing Disaster Recovery with VDM’s and CIFS Servers
Checking Replication Job Throughput with the CLI
Collecting info on Actives Shares, Clients, Protocols, & Authentication with the CLI
Listing and Counting Multiprotocol File Systems from the CLI

VNX NAS CLI Command Reference (Updated January 2018):

NAS Commands:
nas_disk   -list Lists the disk table
nas_checkup Runs a system health check.
nas_pool   -size -all Lists available space on each defined storage pool
nas_replicate  -info –all | grep <fs> Info about each filesystem’s replication status, grep to view just one.
nas_replicate  -list A list of all current replications
nas_server  -list Lists all datamovers. 1=primary,4=standby,6=rdf (remote data facility)
<watch> /nas/sbin/getreason Shows current status of each datamover. 5=up, 0=down or rebooting
nas_fs Creates, deletes, extends, modifies, and lists filesystems.
nas_config Control station configuration (requires root login)
nas_version View current nas revision
nas_ckpt_schedule Manage  checkpoint schedule
nas_storage -list List the attached backend storage systems (with ID’s)
nas_storage -failback id=<x> Fail back failed over SP’s or disks
nas_server  -vdm <vdm_name> -setstate loaded Loads a VDM
nas_server  -vdm <vdm_name> -setstate mounted Unloads a VDM
/nas/sbin/t2reset pwron -s Powers on a data mover that has been shut down.
Server commands:
server_cpu server_<x> -r now Reboots a datamover
server_ping <IP> Ping any IP from the control station
server_ifconfig server_2 –all View all configured interfaces
server_route server_2 {-list,flush,add,delete} Routing table commands
server_mount Mount a filesystem
server_export Export a filesystem
server_stats Provides realtime stats for a datamover, many different options.
server_sysconfig Modifies hardware config of the data movers.
server_devconfig Configures devices on the data movers.
server_sysstat Shows current Memory, CPU, and thread utilization
server_log server_2 Shows current log
vi /nas/jserver/logs/system_log Java System log
vi /var/log/messages System Messages
server_ifconfig server_2 <interface_name> up Bring up a specific interface
server_ifconfig server_2 <interface_name> down Take a specific interface down
server_date Sets system time and NTP server settings
server_date <server_X> timesvc start ntp Starts NTP on a data mover
server_date <server_X> timesvc stats ntp To view the status of NTP.
server_date <server_X> timesvc update ntp Forces an update of NTP
server_file FTP equivalent command
server_dns Configure DNS
server_cifssupport Support services for CIFS users
nas_ckpt_schedule -create <ckpt_fs_name> -filesystem -recurrence once To create a single Checkpoint
fs_ckpt <ckpt_fs_name> -name -Create -readonly n To create a Read/Write copy of a single Checkpoint
server_export [vdm] -P cifs -name [filesystem]_ckpt1 -option netbios=[cifserver] [filesystem]_ckpt1_writeable1 To export a Read/Write checkpoint copy to a CIFS Share
server_cifs server_2 -Join compname=SERVERNAME,domain=DOMAIN.COM,admin=ADMINID Join a CIFS Server to the domain
server_cifs server_2 -Unjoin compname=SERVERNAME,domain=DOMAIN.COM,admin=ADMINID Unjoin a CIFS Server to the domain
 .server_config server_2 -v “pdc dump” To view the current domain controllers visible on the data mover
.server_config server_2 -v “pdc enable=<ip_address>” Enable a domain controller
.server_config server_2 -v “pdc disable=<ip_address>” Disable a domain controller
server_setup server_2 -P cifs -o stop Stop CIFS Service
server_setup server_2 -P cifs -o start Start CIFS Service
server_iscsi server_2 -service -start Start iSCSI service
server_iscsi server_2 -service -stop Stop iSCSI service
server_iscsi server_2 -service -status Check the status of the iSCSI service
.server_config  server_x  “logsys set  severity  NDMP=LOG_DBG2” Enable NDMP Logging [run both of these commands]
.server_config  server_x  “logsys set  severity  PAX=LOG_DBG2” Enable NDMP Logging [run both of these commands]
.server_config  server_x  “logsys  set severity  NDMP=LOG_ERR” Disable NDMP Logging [run both of these commands]
.server_config  server_x  “logsys set severity   PAX=LOG_ERR” Disable NDMP Logging [run both of these commands]
server_netstat server_x -i               Interface statistics Gather interface performance statistics
server_sysconfig server_x -v         Lists virtual devices List Virtual Devices
server_sysconfig server_x -v -i vdevice_name  Informational stats on the virtual device Informational stats on the virtual device
server_netstat server_x -s -a tcp  Retransmissions Retransmissions
server_nfsstat server_x                    NFS SRTs NFS SRTs
server_nfsstat server_x -zero        Reset NFS stats Reset NFS Stats
To view HBA Statistics:
.server_config server_2 -v “printstats fcp reset” View HBA Stats:  Toggles the service on/off
.server_config server_2 -v “printstats fcp full” View HBA Stats:  View stats table (must wait for some stats to collect before viewing)
Filesystem specific commands:
fs_ckpt Manage Checkpoints
fs_dhsm Manage File Mover
fs_group Manage filesystem groups
Complete List of  “nas_”  Commands:
nas_acl Creates, lists, and displays information for access control level entries within the table
nas_ckpt_schedule Manages SnapSure checkpoint scheduling for the VNX
nas_dbtable Displays the table records of the Control Station.
nas_emailuser Manages email notifications for serious system events
nas_inventory Provides detailed information about hardware components
nas_pool Manages the user-defined and system-defined storage pools
nas_slice Manage Slices
nas_task Manages in-progress or completed tasks
nas_automountmap Creates and displays an automount map containing all permanently exported file systems
nas_cmd nas_cmd
nas_devicegroup Manages an established MirrorView/Synchronous consistency group
nas_event Provides a user interface to system-wide events
nas_license Enables software packages.
nas_quotas Manages quotas for mounted file systems.
nas_stats Manages Statistics Groups.
nas_version Displays the software version running on the Control Station.
nas_cel Performs management of remotely linked VNX or a linked pair of Data Movers.
nas_copy Creates a replication session for a one-time copy of a file system.
nas_disk Manages the disk table.
nas_fs Manages local file systems for the VNX.
nas_logviewer Displays the content of nas_eventlog generated log files.
nas_replicate Manages loopback, local, and remote VNX Replicator sessions.
nas_storage Controls storage system access and performs some management tasks
nas_volume Manages the volume table.
nas_checkup Provides a system health checkup for the VNX.
nas_cs Manages the configuration properties of the Control Station.
nas_diskmark Queries the system, manages and lists the SCSI devices configuration.
nas_fsck Manages fsck and aclchk utilities on specified file systems.
nas_message Displays message description.
nas_server Manages the Data Mover (server) table.
nas_symm nas_symm
nas_xml nas_xml
Complete list of  “server_”  Commands:
server_archive Reads and writes file archives, and copies directory hierarchies.
server_cifssupport Provides support services for CIFS users.
server_file Copies files between the Control Station and the specified Data Movers.
server_log Displays the log generated by the specified Data Mover.
server_name Manages the name for the specified Data Movers.
server_ping6 Checks the IPv6 network connectivity for the specified Data Movers.
server_sysconfig Manages the hardware configuration for the specified Data Mover(s).
server_vtlu Configures a virtual tape library unit (VTLU) on the specified Data Movers
server_arp Manages the Address Resolution Protocol (ARP) table for the Data Movers.
server_cpu Performs an orderly, timed, or immediate halt or reboot of a Data Mover.
server_ftp Configures the FTP server configuration for the specified Data Movers.
server_mgr server_mgr (deprecated?)
server_netstat Displays the network statistics for the specified Data Mover.
server_rip Manages the Routing Information Protocol (RIP) configuration
server_sysstat Displays the operating system statistics for the specified Data Movers.
server_cdms Provides File Migration Service for VNX functionality
server_date Displays or sets the date and time for a Data Mover, and synchronizes time
server_http Configures the HTTP configuration file for independent services
server_mount Mounts file systems and manages mount options
server_nfs Manages the NFS service, including secure NFS and NVSv4
server_route Manages the routing table for the specified Data Movers.
server_tftp Manages the Trivial File Transfer Protocol (TFTP)
server_cepp Manages the Common Event Publishing Agent (CEPA) service
server_dbms Enables backup and restore of databases, displays database environment statistics.
server_ifconfig Manages the network interface configuration
server_mountpoint Manages mount points for the specified Data Movers.
server_nfsstat server_nfsstat (deprecated?)
server_security Manages GPO Policy settings for CIFS Servers
server_umount Unmounts file systems
server_certificate Manages VNX for file system’s Public Key Infrastructure (PKI)
server_devconfig Queries, saves, and displays the SCSI over Fibre Channel device configuration
server_ip Manages the IPv6 neighbor cache and route table for VNX.
server_mpfs Sets up and configures MPFS protocol.
server_nis Manages the Network Information Service (NIS) configuration
server_setup Manages the type and protocol component for the specified Data Movers.
server_uptime Displays the length of time that a specified Data Mover has been running since the last reboot
server_checkup Checks the configuration parameters, and state of a Data Mover and its dependencies
server_df Reports free and used disk space and inodes for mounted file systems
server_iscsi server_iscsi (deprecated?)
server_mpfsstat server_mpfsstat (deprecated?)
server_param Manages parameter information for the specified Data Movers.
server_snmpd Manages the Simple Network Management Protocol (SNMP) config values
server_usermapper Provides an interface to manage the Internal Usermapper service.
server_cifs Manages the CIFS configuration for the specified Data Movers or VDMs
server_dns Manages the Domain Name System (DNS) lookup server config
server_kerberos Manages the Kerberos configuration within the specified Data Movers.
server_mt Manages the magnetic tape drive for the specified Data Mover.
server_pax Displays and resets backup and restore statistics and file system information for a backup session already in progress.
server_standby Manages the standby and RDF relationships for the specified Data Movers.
server_version Displays the software version running on the specified Data Movers.
server_cifsstat server_cifsstat (deprecated?)
server_export Exports file systems and manages access on the specified Data Movers for NFS/CIFS clients
server_ldap Manages the LDAP-based directory client configuration and LDAP over SSL
server_muxconfig server_muxconfig (deprecated?)
server_ping Checks the network connectivity for the specified Data Movers.
server_stats Displays sets of statistics that are running on the specified Data Mover.
server_viruschk Manages the virus checker configuration for the specified Data Movers.
Complete list of  “fs_” Commands:
fs_ckpt Manages checkpoints using the EMCSnapSure functionality.
fs_dedupe Manages filesystem deduplication state.
fs_dhsm Manages the VNX FileMover file system connections.
fs_group Creates a file system group from the specified file systems or a single file system
fs_rdf Manages the Remote Data Facility (RDF) functionality for a file system residing on RDF drives.
fs_timefinder Manages the TimeFinderTM/FS functionality for the specified filesystem

A Roundup of Storage Startups

Blockchain and Enterprise Storage