Tag Archives: commands

Undocumented Celerra / VNX File commands

vnx1

The .server_config command is undocumented from EMC, I assume they don’t want customers messing with it. Use these commands at your own risk. ūüôā

Below is a list of some¬†of those undocumented¬†commands,¬†most are¬†meant¬†for¬†viewing performance stats. I’ve had EMC support use the fcp command during a support call in the past.¬†¬†¬†When using the command for fcp stats, ¬†I believe you need to run the ‘reset’ command first as it enables the collection of statistics.

There are likely other parameters that can be used with .server_config but I haven’t discovered them yet.

TCP Stats:

To view TCP info:
.server_config server_x -v “printstats tcpstat”
.server_config server_x -v “printstats tcpstat full”
.server_config server_x -v “printstats tcpstat reset”

Sample Output (truncated):
TCP stats :
connections initiated 8698
connections accepted 1039308
connections established 1047987
connections dropped 524
embryonic connections dropped 3629
conn. closed (includes drops) 1051582
segs where we tried to get rtt 8759756
times we succeeded 11650825
delayed acks sent 537525
conn. dropped in rxmt timeout 0
retransmit timeouts 823

SCSI Stats:

To view SCSI IO info:
.server_config server_x -v “printstats scsi”
.server_config server_x -v “printstats scsi reset”

Sample Output:
This output needs to be in a fixed width font to view properly.¬† I can’t seem to adjust the font, so I’ve attempted to add spaces to align it.
Ctlr: IO-pending Max-IO IO-total Idle(ms) Busy(ms) Busy(%)
0:      0         53    44925729       122348758     19159954   13%
1:      0                                           1 1 141508682       0          0%
2:      0                                           1 1 141508682       0          0%
3:      0                                           1 1 141508682       0          0%
4:      0                                           1 1 141508682       0          0%

File Stats:

.server_config server_x -v “printstats filewrite”
.server_config server_x -v “printstats filewrite full”
.server_config server_x -v “printstats filewrite reset”

Sample output (Full Output):
13108 writes of 1 blocks in 52105250 usec, ave 3975 usec
26 writes of 2 blocks in 256359 usec, ave 9859 usec
6 writes of 3 blocks in 18954 usec, ave 3159 usec
2 writes of 4 blocks in 2800 usec, ave 1400 usec
4 writes of 13 blocks in 6284 usec, ave 1571 usec
4 writes of 18 blocks in 7839 usec, ave 1959 usec
total 13310 blocks in 52397489 usec, ave 3936 usec

FCP Stats:

To view FCP stats, useful for checking SP balance:
.server_config server_x -v “printstats fcp”
.server_config server_x -v “printstats fcp full”
.server_config server_x -v “printstats fcp reset”

Sample Output (Truncated):
This output needs to be in a fixed width font to view properly.¬† I can’t seem to adjust the font, so I’ve attempted to add spaces to align it.
Total I/O Cmds: +0%——25%——-50%——-75%—–100%+ Total 0
FCP HBA 0 |                                                                                            | 0%  0
FCP HBA 1 |                                                                                            | 0%  0
FCP HBA 2 |                                                                                            | 0%  0
FCP HBA 3 |                                                                                            | 0%  0
# Read Cmds: +0%——25%——-50%——-75%—–100%+ Total 0
FCP HBA 0 |                                                                                            | 0% 0
FCP HBA 1 |                                                                                            | 0% 0
FCP HBA 2 |                                                                                            | 0% 0
FCP HBA 3 |  XXXXXXXXXXX                                                          | 25% 0

Usage:

‘fcp’ options are:¬†¬†¬†¬†¬†¬† bind …, flags, locate, nsshow, portreset=n, rediscover=n
rescan, reset, show, status=n, topology, version

‘fcp bind’ options are:¬† clear=n, read, rebind, restore=n, show
showbackup=n, write

Description:

Commands for ‘fcp’ operations:
fcp bind <cmd> ……… Further fibre channel binding commands
fcp flags ………….. Show online flags info
fcp locate …………. Show ScsiBus and port info
fcp nsshow …………. Show nameserver info
fcp portreset=n …….. Reset fibre port n
fcp rediscover=n ……. Force fabric discovery process on port n
Bounces the link, but does not reset the port
fcp rescan …………. Force a rescan of all LUNS
fcp reset ………….. Reset all fibre ports
fcp show …………… Show fibre info
fcp status=n ……….. Show link status for port n
fcp status=n clear ….. Clear link status for port n and then Show
fcp topology ……….. Show fabric topology info
fcp version ………… Show firmware, driver and BIOS version

Commands for ‘fcp bind’ operations:
fcp bind clear=n ……. Clear the binding table in slot n
fcp bind read ………. Read the binding table
fcp bind rebind …….. Force the binding thread to run
fcp bind restore=n ….. Restore the binding table in slot n
fcp bind show ………. Show binding table info
fcp bind showbackup=n .. Show Backup binding table info in slot n
fcp bind write ……… Write the binding table

NDMP Stats:

To Check NDMP Status:
.server_config server_x -v “printstats vbb show”

CIFS Stats:

This will output a CIFS report, including all servers, DC’s, IP’s, interfaces,¬†Mac addresses, and more.

.server_config server_x -v “cifs”

Sample Output:

1327007227: SMB: 6: 256 Cifs threads started
1327007227: SMB: 6: Security mode = NT
1327007227: SMB: 6: Max protocol = SMB2
1327007227: SMB: 6: I18N mode = UNICODE
1327007227: SMB: 6: Home Directory Shares DISABLED
1327007227: SMB: 6: Usermapper auto broadcast enabled
1327007227: SMB: 6:
1327007227: SMB: 6: Usermapper[0] = [127.0.0.1] state:active (auto discovered)
1327007227: SMB: 6:
1327007227: SMB: 6: Default WINS servers = 172.168.1.5
1327007227: SMB: 6: Enabled interfaces: (All interfaces are enabled)
1327007227: SMB: 6:
1327007227: SMB: 6: Disabled interfaces: (No interface disabled)
1327007227: SMB: 6:
1327007227: SMB: 6: Unused Interface(s):
1327007227: SMB: 6:  if=172-168-1-84 l=172.168.1.84 b=172.168.1.255 mac=0:60:48:1c:46:96
1327007227: SMB: 6:  if=172-168-1-82 l=172.168.1.82 b=172.168.1.255 mac=0:60:48:1c:10:5d
1327007227: SMB: 6:  if=172-168-1-81 l=172.168.1.81 b=172.168.1.255 mac=0:60:48:1c:46:97
1327007227: SMB: 6:
1327007227: SMB: 6:
1327007227: SMB: 6: DOMAIN DOMAIN_NAME FQDN=DOMAIN_NAME.net SITE=STL-Colo RC=24
1327007227: SMB: 6:  SID=S-1-5-15-7c531fd3-6b6745cb-ff77ddb-ffffffff
1327007227: SMB: 6:  DC=DCAD01(172.168.1.5) ref=2 time=0 ms
1327007227: SMB: 6:  DC=DCAD02(172.168.29.8) ref=2 time=0 ms
1327007227: SMB: 6:  DC=DCAD03(172.168.30.8) ref=2 time=0 ms
1327007227: SMB: 6:  DC=DCAD04(172.168.28.15) ref=2 time=0 ms
1327007227: SMB: 6: >DC=SERVERDCAD01(172.168.1.122) ref=334 time=1 ms (Closest Site)
1327007227: SMB: 6: >DC=SERVERDCAD02(172.168.1.121) ref=273 time=1 ms (Closest Site)
1327007227: SMB: 6:
1327007227: SMB: 6: CIFS Server SERVERFILESEMC[DOMAIN_NAME] RC=603
1327007227: UFS: 7: inc ino blk cache count: nInoAllocs 361: inoBlk 0x0219f2a308
1327007227: SMB: 6:  Full computer name=SERVERFILESEMC.DOMAIN_NAME.net realm=DOMAIN_NAME.NET
1327007227: SMB: 6:¬† Comment=’EMC-SNAS:T6.0.41.3′
1327007227: SMB: 6:  if=172-168-1-161 l=172.168.1.161 b=172.168.1.255 mac=0:60:48:1c:46:9c
1327007227: SMB: 6:   FQDN=SERVERFILESEMC.DOMAIN_NAME.net (Updated to DNS)
1327007227: SMB: 6:  Password change interval: 0 minutes
1327007227: SMB: 6:  Last password change: Fri Jan  7 19:25:30 2011 GMT
1327007227: SMB: 6:  Password versions: 2, 2
1327007227: SMB: 6:
1327007227: SMB: 6: CIFS Server SERVERBKUPEMC[DOMAIN_NAME] RC=2 (local users supported)
1327007227: SMB: 6:  Full computer name=SERVERbkupEMC.DOMAIN_NAME.net realm=DOMAIN_NAME.NET
1327007227: SMB: 6:¬† Comment=’EMC-SNAS:T6.0.41.3′
1327007227: SMB: 6:  if=172-168-1-90 l=172.168.1.90 b=172.168.1.255 mac=0:60:48:1c:10:54
1327007227: SMB: 6:   FQDN=SERVERbkupEMC.DOMAIN_NAME.net (Updated to DNS)
1327007227: SMB: 6:  Password change interval: 0 minutes
1327007227: SMB: 6:  Last password change: Thu Sep 30 16:23:50 2010 GMT
1327007227: SMB: 6:  Password versions: 2
1327007227: SMB: 6:
 

Domain Controller Commands:

These commands are useful for troubleshooting a windows domain controller connection issue on the control station.  Use these commands along with checking the normal server log (server_log server_2) to troubleshoot that type of problem.

To view the current domain controllers visible on the data mover:

.server_config server_2 -v ‚Äúpdc dump‚ÄĚ

Sample Output (Truncated):

1327006571: SMB: 6: Dump DC for dom='<domain_name>’ OrdNum=0
1327006571: SMB: 6: Domain=<domain_name> Next trusted domains update in 476 seconds1327006571: SMB: 6:  oldestDC:DomCnt=1,179531 Time=Sat Oct 15 15:32:14 2011
1327006571: SMB: 6:¬† Trusted domain info from DC='<Windows_DC_Servername>’ (423 seconds ago)
1327006571: SMB: 6:   Trusted domain:<domain_name>.net [<Domain_Name>]
   GUID:00000000-0000-0000-0000-000000000000
1327006571: SMB: 6:    Flags=0x20 Ix=0 Type=0x2 Attr=0x0
1327006571: SMB: 6:    SID=S-1-5-15-d1d612b1-87382668-9ba5ebc0
1327006571: SMB: 6:¬†¬†¬† DC=’-‘
1327006571: SMB: 6:    Status Flags=0x0 DCStatus=0x547,1355
1327006571: SMB: 6:   Trusted domain: <Domain_Name>
1327006571: SMB: 6:    Flags=0x22 Ix=0 Type=0x1 Attr=0x1000000
1327006571: SMB: 6:    SID=S-1-5-15-76854ac0-4c527104-321d5138
1327006571: SMB: 6:¬†¬†¬† DC=’\\<Windows_DC_Servername>’
1327006571: SMB: 6:    Status Flags=0x0 DCStatus=0x0,0
1327006571: SMB: 6:   Trusted domain:<domain_name>.net [<domain_name>]
1327006571: SMB: 6:    Flags=0x20 Ix=0 Type=0x2 Attr=0x0
1327006571: SMB: 6:    SID=S-1-5-15-88d60754-f3ed4f9d-b3f2cbc4
1327006571: SMB: 6:¬†¬†¬† DC=’-‘
1327006571: SMB: 6:    Status Flags=0x0 DCStatus=0x547,1355
DC=DC0x0067a82c18 <Windows_DC_Servername>[<domain_name>](10.3.0.5) ref=2 time(getdc187)=0 ms LastUpdt=Thu Jan 19 20:45:14 2012
    Pid=1000 Tid=0000 Uid=0000
    Cnx=UNSUCCESSFUL,DC state Unknown
    logon=Unknown 0 SecureChannel(s):
    Capa=0x0 Nego=0x0000000000,L=0 Chal=0x0000000000,L=0,W2kFlags=0x0
    refCount=2 newElectedDC=0x0000000000 forceInvalid=0
    Discovered from: WINS

To enable or disable a domain controller on the data mover:

.server_config server_2 -v ‚Äúpdc enable=<ip_address>‚Ä̬† Enable a domain controller

.server_config server_2 -v ‚Äúpdc disable=<ip_address>‚Ä̬† Disable a domain controller

MemInfo:

¬†.server_config server_2 -v “meminfo”

Sample Output (truncated):

CPU=0
3552907011 calls to malloc, 3540029263 to free, 61954 to realloc
Size     In Use       Free      Total nallocs nfrees
16       3738        870       4608   161720370   161716632
32      18039      17289      35328   1698256206   1698238167
64       6128       3088       9216   559872733   559866605
128       6438      42138      48576   255263288   255256850
256       8682      19510      28192   286944797   286936115
512       1507       2221       3728   357926514   357925007
1024       2947       9813      12760   101064888   101061941
2048       1086        198       1284    5063873    5062787
4096         26        138        164    4854969    4854943
8192        820         11        831   19562870   19562050
16384         23         10         33       5676       5653
32768          6          1          7        101         95
65536         12          0         12         12          0
524288          1          0          1          1          0
Total Used     Total Free    Total Used + Free
all sizes   18797440   23596160   42393600

MemOwners:

.server_config server_2 -v “help memowners”

Usage:
memowners [dump | showmap | set … ]

Description:
memowners [dump] – prints memory owner description table
memowners showmap – prints a memory usage map
memowners memfrag [chunksize=#] – counts free chunks of given size
memowners set priority=# tag=# – changes dump priority for a given tag
memowners set priority=# label=’string’ – changes dump priority for a given label
The priority value can be set to 0 (lowest) to 7 (highest).

Sample Output (truncated):

1408979513: KERNEL: 6: Memory_Owner dump.
nTotal Frames 1703936 Registered = 75,  maxOwners = 128
1408979513: KERNEL: 6:   0 (   0 frames) No owner, Dump priority 6
1408979513: KERNEL: 6:   1 (3386 frames) Free list, Dump priority 0
1408979513: KERNEL: 6:   2 (40244 frames) malloc heap, Dump priority 6
1408979513: KERNEL: 6:   3 (6656 frames) physMemOwner, Dump priority 7
1408979513: KERNEL: 6:   4 (36091 frames) Reserved Mem based on E820, Dump priority 0
1408979513: KERNEL: 6:   5 (96248 frames) Address gap based on E820, Dump priority 0
1408979513: KERNEL: 6:   6 (   0 frames) Rmode isr vectors, Dump priority 7

Advertisements

Celerra Health Check with CLI Commands

Here are the first commands I’ll type when I suspect there is a problem with the Celerra, or if I want to do a simple health check.

1. <watch> /nas/sbin/getreason.  This will quickly give you the current status of each data mover. 5=up, 0=down/rebooting.  Typing watch before the command will run the command with continuous updates so you can monitor a datamover if you are purposely rebooting it.

10 – slot_0 primary control station
5 – slot_2 contacted
5 – slot_3 contacted

2. nas_server -list.¬† This lists all of the datamovers and their current state.¬† It’s a good way to quickly tell which datamovers are active and which are standby.

1=nas, 2=unused, 3=unused, 4=standby, 5=unused, 6=rdf

id      type  acl  slot groupID  state  name
1        1    0     2                         0    server_2
2        4    0     3                        0    server_3

3. server_sysstat.  This will give you a quick overview of memory and CPU utilization.

server_2 :
threads runnable = 6
threads blocked  = 4001
threads I/J/Z    = 1
memory  free(kB) = 2382807
cpu     idle_%   = 70

4. nas_checkup.   This runs a system health check.

Check Version:  5.6.51.3
Check Command:  /nas/bin/nas_checkup
Check Log    :  /nas/log/checkup-run.110608-143203.log

————————————-Checks————————————-
Control Station: Checking if file system usage is under limit………….. Pass
Control Station: Checking if NAS Storage API is installed correctly…….. Pass

5. server_log server_2.  This shows the current alert log.  Alert logs are also stored in /nas/log/webui.

6. vi /nas/jserver/logs/system_log.   This is the java system log.

7. vi /var/log/messages.  This displays system messages.

Useful Celerra / VNX File Commands

vnx1.jpg

Here is a list of VNX OE for File and Celerra commands I keep at my desk for reference.  I have another post that references some additional undocumented commands here.

NAS Commands:

nas_disk   -list    Lists the disk table
nas_checkup     Runs a system health check.
nas_pool   -size -all    Lists available space on each defined storage pool
nas_replicate¬† -info ‚Äďall | grep <fs>¬†¬†Info about each filesystem‚Äôs replication status, grep to view just one.
nas_replicate  -list    A list of all current replications
nas_server  -list     Lists all datamovers. 1=primary,4=standby,6=rdf (remote data facility)
<watch> /nas/sbin/getreason   Shows current status of each datamover. 5=up, 0=down or rebooting
nas_fs      Creates, deletes, extends, modifies, and lists filesystems.
nas_config     Control station configuration (requires root login)
nas_version     View current nas revision
nas_ckpt_schedule    Manage  checkpoint schedule
nas_storage -list¬†¬† List the attached backend storage systems (with ID’s)
nas_storage -failback id=<x>¬†¬†¬† Fail back failed over SP’s or disks
nas_server  -vdm <vdm_name> -setstate loaded      Loads a VDM
nas_server  -vdm <vdm_name> -setstate mounted    Unloads a VDM
/nas/sbin/t2reset pwron -s   This command will power on a data mover that has been shut down.  This was user submitted in the comments on this post.

Several nas_<x> commands can be run with an additional database query option for reporting purposes.  Please view my blog post about it here for more information.

Server commands:

server_cpu server_<x> -r now   Reboots a datamover
server_ping <IP>    ping any IP from the control station
server_ifconfig server_2 ‚Äďall¬†¬†¬†View all configured interfaces
server_route server_2 {-list,flush,add,delete}   Routing table commands
server_mount     Mount a filesystem
server_export     Export a filesystem
server_stats     Provides realtime stats for a datamover, many different options.
server_sysconfig    Modifies hardware config of the data movers.
server_devconfig    Configures devices on the data movers.
server_sysstat     Shows current Memory, CPU, and thread utilization
server_log server_2    Shows current log
vi /nas/jserver/logs/system_log   Java System log
vi /var/log/messages    System Messages
server_ifconfig server_2 <interface_name> up  Bring up a specific interface
server_ifconfig server_2 <interface_name> down Take a specific interface down
server_date     Sets system time and NTP server settings
server_date <server_X> timesvc start ntp <time_server_IP_address>  Starts NTP on a data mover
server_date <server_X> timesvc stats ntp    To view the status of NTP.
server_date <server_X> timesvc update ntp    Forces an update of NTP
server_file     FTP equivalent command
server_dns     Configure DNS
server_cifssupport    Support services for CIFS users

To create a single checkpoint:
nas_ckpt_schedule -create <ckpt_fs_name> -filesystem <fs_name> -recurrence once

To create a Read/Write copy of a single checkpoint:
fs_ckpt <ckpt_fs_name> -name <r/w_ckpt_fs_name> -Create -readonly n 

To export a Read/Write checkpoint copy to a CIFS Share:
server_export [vdm] -P cifs -name [filesystem]_ckpt1 -option netbios=[cifserver] [filesystem]_ckpt1_writeable1

To view HBA Statistics:
.server_config¬†server_2 -v “printstats fcp reset”¬† Toggles the service on/off
.server_config¬†server_2 -v “printstats fcp full” ¬†¬†¬† View the stats table (must wait a while¬†for some stats to collect before viewing)

To Join/Unjoin a CIFS Server from the domain:
server_cifs server_2 -Join compname=SERVERNAME,domain=DOMAIN.COM,admin=ADMINID
server_cifs server_2 -Unjoin compname=SERVERNAME,domain=DOMAIN.COM,admin=ADMINID

To view the current domain controllers visible on the data mover:
.server_config¬†server_2 -v “pdc dump”

To enable or disable a domain controller on the data mover:
.server_config server_2 -v “pdc enable=<ip_address>”¬† Enable a domain controller
.server_config server_2 -v “pdc disable=<ip_address>”¬† Disable a domain controller

To stop and start the CIFS service:
server_setup server_2 -P cifs -o stop   Stop CIFS Service
server_setup server_2 -P cifs -o start  Start CIFS Service

To stop, start or check the status of the iSCSI service:
server_iscsi server_2 -service -start     Start iSCSI service
server_iscsi server_2 -service -stop      Stop iSCSI service
server_iscsi server_2 -service -status  Check the status of the iSCSI service

To enable/disable NDMP Logging:
Turn it on:
.server_config¬† server_x¬† “logsys set¬† severity¬† NDMP=LOG_DBG2”
.server_config¬† server_x¬† “logsys set¬† severity¬† PAX=LOG_DBG2”
Turn it off:
.server_config¬† server_x¬† “logsys¬† set severity¬† NDMP=LOG_ERR”
.server_config¬† server_x¬† “logsys set severity¬†¬† PAX=LOG_ERR”

For gathering performance statistics:
server_netstat server_x -i               Interface statistics
server_sysconfig server_x -v         Lists virtual devices
server_sysconfig server_x -v -i vdevice_name  Informational stats on the virtual device
server_netstat server_x -s -a tcp  Retransmissions
server_nfsstat server_x                    NFS SRTs
server_nfsstat server_x -zero        Reset NFS stats

Filesystem specific commands:

fs_ckpt      Manage Checkpoints
fs_dhsm     Manage File Mover
fs_group     Manage filesystem groups

Complete List of¬† “nas_” ¬†Commands:

This is just for reference, you can easily pull up this list from a Celerra by typing nas_ and hitting the tab key.

nas_acl
nas_ckpt_schedule
nas_dbtable
nas_emailuser
nas_inventory
nas_pool
nas_slice
nas_task
nas_automountmap
nas_cmd
nas_devicegroup
nas_event
nas_license
nas_quotas
nas_stats
nas_version nas_cel
nas_copy
nas_disk
nas_fs
nas_logviewer
nas_replicate
nas_storage
nas_volume
nas_checkup
nas_cs
nas_diskmark
nas_fsck
nas_message
nas_server
nas_symm
nas_xml

Complete list of¬† “server_” ¬†Commands:

This is just for reference, you can easily pull up this list from a Celerra by typing server_ and hitting the tab key.

server_archive
server_cifssupport
server_file
server_log
server_name
server_ping6
server_sysconfig
server_vtlu
server_arp
server_cpu
server_ftp
server_mgr
server_netstat
server_rip
server_sysstat
server_cdms
server_date
server_http
server_mount
server_nfs
server_route
server_tftp
server_cepp
server_dbms
server_ifconfig
server_mountpoint
server_nfsstat
server_security
server_umount
server_certificate
server_devconfig
server_ip
server_mpfs
server_nis
server_setup
server_uptime
server_checkup
server_df
server_iscsi
server_mpfsstat
server_param
server_snmpd
server_usermapper
server_cifs
server_dns
server_kerberos
server_mt
server_pax
server_standby
server_version
server_cifsstat
server_export
server_ldap
server_muxconfig
server_ping
server_stats
server_viruschk

Complete list of¬† “fs_” Commands:

This is just for reference, you can easily pull up this list from a Celerra by typing fs_ and hitting the tab key.

fs_ckpt
fs_dedupe
fs_dhsm
fs_group
fs_rdf
fs_timefinder