Tag Archives: recovery

Data Domain CLI Command Reference Guide

Other CLI Reference Guides:
Isilon CLI  |  EMC ECS CLI  |  VNX NAS CLI | ViPR Controller CLINetApp Clustered ONTAP CLI  |  Brocade FOS CLI | EMC XTremIO CLI

This is a Data Domain CLI Command Reference Guide for the commands that are more commonly used.

If you’re looking to automate reports for your Data Domain, see my post Easy Reporting on Data Domain using the Autosupport Log.

# alerts notify-list create <group-name> Creates a notification list and subscribes to events belonging to the specified list of classes and severity levels.
# alerts notify-list add <group-name> Adds to a notification list and subscribes to events belonging to the specified list of classes and severity levels.
# alerts notify-list del <group-name> Deletes members from a notification list, a list of classes, a list of email addresses.
# alerts notify-list destroy <group-name> Destroys a notification list
# alerts notify-list reset Resets all notification lists to factory default
# alerts notify-list show Shows notification lists’ configuration
# alerts notify-list test Sends a test notification to alerts notify-list
# cifs share create share path {max-connections max connections | clients clients | users users | comment comment}
# cifs status Check CIFS Status
# cifs disable Disable CIFS Service
# cifs enable Enable CIFS Service
# nfs add path client-list [(option-list)] Add NFS clients to an Export
# nfs show active List clients active in the past 15 minutes and the mount path for each
# nfs show clients list NFS clients allowed to access the Data Domain system and the mount path and NFS options for each
# nfs show detailed-stats display NFS cache entries and status to facilitate troubleshooting
# NFS Status Display NFS Status
# NFS Enable Enable NFS Service
# NFS disable Disable NFS Service
DD Boost
# ddboost enable Enable DDBoost
# ddboost status show DDBoost status
# ddboost set user-name <user-name> Set DD Boost user
# ddboost access add clients <client-list> Add clients to DD Boost access list
# ddboost storage-unit create <storage-unit-name> Create storage-unit, setting quota limits
# ddboost storage-unit delete <storage-unit-name> Delete storage-unit
# ddboost storage-unit show [compression] [<storage-unit-name>] List the storage-units and images in a storage-unit:
# ddboost storage-unit create <storage-unit> user <user-name> Create a storage unit, assign tenant, and set quota and stream limits
# ddboost storage-unit delete <storage-unit> Delete a specified storage unit, its contents, and any DD Boost assocaitions
# ddboost storage-unit rename <storage-unit> <new-storage-unit> Rename a storage-unit
# ddboost storage-unit undelete <storage-unit> Recover a deleted storage unit
# ddboost option reset Reset DD Boost options
# ddboost option set distributed-segment-processing {enabled|disabled} Enable or disable distributed-segment-processing for DD Boost
# ddboost option set virtual-synthetics {enabled | disabled} Enable or disable virtual-synthetics for DD Boost
# ddboost option show Show DD Boost options
# ddboost option set fc {enabled | disabled} Enable or disable fibre-channel for DD Boost
# ddboost fc dfc-server-name set DDBoost Fibre-Channel set Server Name
# ddboost fc dfc-server-name show Show DDBoost Fibre-Channel Server Name
# ddboost fc status DDBoost Fibre Channel Status
# ddboost fc group show list [<group-spec>] [initiator<initiator-spec>] List configured DDBoost FC groups
# ddboost fc group create <group-name> Create a DDBoost FC group
# ddboost fc group add <group-name> initiator <initiator-spec> Add initiators to a DDBoost FC group
# ddboost fc group add <group-name> device-set Add DDBoost devices to a DDBoost FC group
Encryption and File system Locking
# filesys enable Enables the file system
# filesys disable Disables the file system
# filesys encryption enable Enables encryption. Enter a passphrase when prompted
# filesys encryption disable Disables encryption.
# filesys encryption show Checks the status of the encryption feature
# filesys encryption lock Locks the system by creating a new passphrase and destroying the cached copy of existing passphrase
# filesys encryption passphrase change Changes the passphrase for system encryption keys
# filesys encryption unlock Prepares the encrypted file system for use after it has arrived at its destination
# license add <license-code> [<license-code> …] Adds one or more licenses for features and storage capacity.
# license show [local] Displays license codes currently installed.
# license del <license-code> Deletes one or more licenses.
# license reset Removes all licenses and requires confirmation before deletion.
# net show settings Displays the interface’s network settings
# net show hardware Displays the interface’s hardware configuration
# net show config Displays the active network configuration
# net show domainname Displays the domain name associated with this device
# net show searchdomain Lists the domains that will be searched when only the host name is provided for a r command
# net show dns Lists the domain name servers used by this device.
# net show stats Provides a number of different networking statistics
# net show all Combines the output of several other net show CLI commands
Replication, Throttling, LBO, Encryption
# replication enable {<destination> | all} Enables replication
# replication disable {<destination> | all} Disables replication
# replication add source <source> destination <destination> Creates a replication pair
# replication break {<destination> | all} Removes the source or destination DD system from a replication pair
# replication initialize <destination> Initialize replication on the source (configure both source and destination first)
# replication modify <destination> {source-host | destination-host} <new-host-name> Modifies connection host, hostname
# replication modify <destination> connection-host <new-host-name> [port <port>] Modifies port
# replication add … low-bw-optim enabled Adds LBO
# replication modify … low-bw-optim enabled Modify LBO
# replication modify … low-bw-optim disabled Disable
# replication add … encryption enabled Add encryption over wire
# replication modify … encryption enabled Enable encryption over wire
# replication modify … encryption disabled Disable encryption over wire
# replication option set listen-port <port> Modify listening port  [context must be disabled before the connection port can be modified]
# replication option reset listen-port Reset listening port  [context must be disabled before the connection port can be modified]
# replication throttle add <sched-spec> <rate> Add a throttle schedule
# replication throttle add destination <host> <sched-spec> <rate> Add a destination specific throttle
# replication throttle del <sched-spec> Delete a throttle schedule
# replication throttle reset {current | override | schedule | all} Reset throttle configuration
# replication throttle set current <rate> Set a current override
# replication throttle set override <rate> Set a permanent override
# replication throttle show [KiB] Show throttle onfiguration
Retention Lock
# mtree retention-lock enable mtree_name Enables the retention-lock feature for the specified MTree
# mtree retention-lock disable mtree_name Disables the retention-lock feature for the specified MTree
# mtree retention-lock reset Resets the value of the retention period for the specified MTree to its default
# mtree retention-lock revert Reverts the retention lock for all files on a specified path
# mtree retention-lock set Sets the minimum or maximum retention period for the specified MTree
# mtree retention-lock show Shows the minimum or maximum retention period for the specified MTree
#mtree retention-lock status mtree_name Shows the retention-lock status for the specified MTree
#system sanitize abort Aborts the sanitization process
#system sanitize start Starts sanitization process immediately
#system sanitize status Shows current sanitization status
#system sanitize watch Monitors sanitization progress
SMT MTree stats
# mtree list List List the Mtrees on a Data Domain system
# mtree show stats Collect MTree real-time performance statistics
# mtree show performance Collect performance statistics for MTrees associated with a tenant-unit
# mtree show compression Collect compression statistics for MTrees associated with a tenant-unit
# quota capacity show List capacity quotas for MTrees and storage-units
# ddboost storage-unit modify Adjust or modify the quotas after the initial configuration
System Performance
# system show stats interval [interval in seconds] Shows system stats (Disk, IOs,…etc)
# system show performance [ {hr | min | sec} [ {hr | min | sec} ]] Show System Performance
# ndmpd enable Enable the NDMP daemon
# ndmpd show devicenames Verify that the NDMP daemon sees the devices created in the TapeServer access group
# ndmpd user add ndmp Add an NDMP user
# ndmpd option show all Check the options for the ndmpd daemon
# ndmpd option set authentication md5 Set the ndmpd service authentication to MD5
# ndmpd option show all Verify the serivce authentication

What is VPLEX?

vplexWe are looking at implementing a storage virtualization device and I started doing a bit of research on EMC’s product offering.  Below is a summary of some of the information I’ve gathered, including a description of what VPLEX does as well as some pros and cons of implementing it.  This is all info I’ve gathered by reading various blogs, looking at EMC documentation and talking to our local EMC reps.  I don’t have any first-hand experience with VPLEX yet.

What is VPLEX?

VPLEX at its core is a storage virtualization appliance. It sits between your arrays and hosts and virtualizes the presentation of storage arrays, including non-EMC arrays.  Instead of presenting storage to the host directly you present it to the VPLEX. You then configure that storage from within the VPLEX and then zone the VPLEX to the host.  Basically, you attach any storage to it, and like in-band virtualization devices, it virtualizes and abstracts them.

There are three VPLEX product offerings, Local, Metro, and Geo:

Local.  VPLEX Local manages multiple heterogeneous arrays from a single interface within a single data center location. VPLEX Local allows increased availability, simplified management, and improved utilization across multiple arrays.

Metro.  VPLEX Metro with AccessAnywhere enables active-active, block level access to data between two sites within synchronous distances.  Host application stability needs to be considered. It is recommended that depending on the application that consideration for Metro be =< 5ms latency. The combination of virtual storage with VPLEX Metro and virtual servers allows for the transparent movement of VM’s and storage across longer distances and improves utilization across heterogeneous arrays and multiple sites.

Geo.  VPLEX Geo with AccessAnywhere enables active-active, block level access to data between two sites within asynchronous distances. Geo improves the cost efficiency of resources and power.  It provides the same distributed device flexibility as Metro but extends the distance up to 50ms of network latency. 

Here are some links to VPLEX content from EMC, where you can learn more about the product:

What are some advantages of using VPLEX? 

1. Extra Cache and Increased IO.  VPLEX has a large cache (64GB per node) that sits in-between the host and the array. It offers additional read cache that can greatly improve read performance on databases because the additional cache is offloaded from the individual arrays.

2. Enhanced options for DR with RecoverPoint. The DR benefits are increased when integrating RecoverPoint with VPLEX Metro or Geo to replicate the data using real time replication. It includes a capacity based journal for very granular rollback capabilities (think of it as a DVR for the data center).  You can also use the native bandwidth reduction features (compression & deduplication) or disable them if you have WAN optimization devices installed like those from Riverbed.  If you want active/active read/write access to data across a large distance, VPLEX is your only option.  NetApp’s V-Series and HDS USPV can’t do it unless they are in the same data center. Here’s a few more advantages:

  • DVR-like recovery to any point in time
  • Dynamic synchronous and asynchronous replication
  • Customized recovery point opbjectives that support any-to-any storage arrays
  • WAN bandwidth reduction of up to 90% of changed data
  • Non-disruptive DR testing

4. Non disruptive data mobility & reduced maintenance costs. One of the biggest benefits of virtualizing storage is that you’ll never have to take downtime for a migration again. It can take months to migrate production systems and without virtualization downtime is almost always required. Also, migration is expensive, it takes a great deal of resources from multiple groups as well as the cost of keeping the older array on the floor during the process. Overlapping maintenance costs are expensive too.  By shortening the migration timeframe hardware maintenance costs will drop, saving money.   Maintenance can be a significant part of the storage TCO, especially if the arrays are older or are going to be used for a longer period of time.  Virtualization can be a great way to reduce those costs and improve the return on assets over time.

5. Flexibility based on application IO.  The ability to move and balance LUN I/O among multiple smaller arrays non-disruptively would allows you to balance workloads and increase your ability to respond to performance demands quickly.  Note that underlying LUNs can be aggregated or simply passed through the VPLEX.

6. Simplified Management and vendor neutrality.   Implementing VPLEX for all storage related provisioning tasks would reduce complexity with multiple vendor arrays.  It allows you to manage multiple heterogeneous arrays from a single interface.  It also makes zoning easier as all hosts would only need to be zoned to the VPLEX rather than every array on the floor, which makes it faster and easier to provision new storage to a new host.

7. Increased leverage among vendors.  This advantage would be true with any virtualization device.  When controller based storage virtualization is employed, there is more flexibility to pit vendors against each other to get the best hardware, software and maintenance costs.  Older arrays could be commoditized which could allow for increased leverage to negotiate the best rates.

8. Use older arrays for Archiving. Data could be seamlessly demoted or promoted to different arrays based on an array’s age, it’s performance levels and it’s related maintenance costs.  Older arrays could be retained for capacity and be demoted to a lower tier of service, and even with the increased maintenance costs it could still save money.

9. Scale.  You can scale it out and add more nodes for more performance when needed.  With a VPLEX Metro configuration, you could configure VPLEX with up to 16 nodes in the cluster between the two sites.

What are some possible disadvantages of VPLEX?

1. Licensing Costs. VPLEX is not cheap.  Also, it can be licensed per frame on VNX but must be licensed per TB on CX series.  Your large,older CX arrays will cost you a lot more to license.

2. It’s one more device to manage.   The VPLEX is an appliance, and it’s one more thing (or things) that has to be managed and paid for.

3. Added complexity to infrastructure.  Depending on the configuration, there could be multiple VPLEX appliances at every site, adding considerable complexity to the environment.

4. Managing mixed workloads in virtual enviornments.  When heavy workloads are all mixed together on the same array there is no way to isolate them, and the ability to migrate that workload non-disruptively to another array is one of the reasons to implement a VPLEX.  In practice, however, those VMs may end up being moved to another array with the same storage limitations as where they came from.  The VPLEX could be simply temporarily solving a problem by moving that problem to a different location.

5. Lack of advanced features. The VPLEX has no advanced storage features such as snapshots, deduplication, replication, or thin provisioning.  It relies on the underlying storage array for those type of features.  As an example, you may want to utilize block based deduplication with an HDS array by placing a Netapp V-series in front of it and using Netapp’s dedupe to enable it.  It is only possible to do that with a Netapp Vseries or HDS USP-V type device, the VPLEX can’t do that.

6. Write cache performance is not improved.  The VPLEX uses write-through caching while their competitor’s storage virtualization devices use write-back caching. When there is a write I/O in a VPLEX environment the I/O is cached on the VPLEX, however it is passed all the way back to the virtualized storage array before an ack is sent back to the host.  The Netapp V-Series and HDS USPV will store the I/O in their own cache and immediately return an ack to the host.  At that point the I/Os are flushed to the back end storage array using their respective write coalescing & cache flushing algorithms.  Because of the write-back behavior it is possible for a possible performance gain above and beyond the performance of the underlying storage arrays due to the caching on these controllers.  There is no performance gain for write I/O in VPLEX environments beyond the existing storage due to the write-through cache design.

Testing Disaster Recovery for VNX VDM’s and CIFS servers

After spending a few days working on a DR test recovery, I thought I’d describe the process along with a few roadblocks that I hit along the way.  We had some specific requirements that had to be met, so I thought I’d share my experiences.  Our host site has a VNX5500 and our DR site has an NS-960, and we have Celerra Replicator configured to replicate the VDM and all of the production filesystems from one site to the other.

Here were my business requirements for this test:

  1. Replicate the VDM, production CIFS server and production filesystems from the host site to DR site.
  2. Fail over (or bring up a copy of) the VDM from the host site to the DR site, mounting the replicated VDM at the DR site.
  3. Fail over (or bring up a copy of) the production CIFS server at the DR site.
  4. Create R/W checkpoints of all replicated filesystems at DR site to allow for appropriate user and application testing.
  5. Share the R/W checkpoints of the replicated filesystems on the CIFS server at the DR site rather than the original replicated filesystems, so original replicated data is not touched and does not need to be replicated again after the test.

I started off by setting up replication jobs for our VDM and all filesystems.  Once those were complete (after several weeks of data transfers) I was ready to test.

Step 1: Replicate VDM and production filesystems

This post isn’t meant to detail the process of actually setting up the initial replications, just how to get the replicated data working and accessible at your DR site.  Setting up replication is a well documented procedure which can be reviewed in EMC’s guide “Using Celerra Replicator (V2)”, P/N 300-009-989.  Once the VDMs and filesystems are replicated, you’re ready for the next step.

Step 2: Bring up the VDM at the DR site

The first step in my testing requirements is to bring up the VDM at the DR site.

Failed attempt 1:

I initially created a new replication session for the VDM as I didn’t want to use the actual production VDM, as this is a DR test and not an actual disaster.

After replicating a new copy of the VDM, I attempted to load it in the CLI with the command below.  This must be done from the CLI as there is no option to do this step in Unisphere.

nas_server –vdm <VDMNAME> -setstate loaded

It failed with this error:

Error 12066: root_fs <VDMNAME> is the source or destination object of a file system and cannot be unmounted or is the source or destination object of a VDM replication session and cannot be unloaded.

It was pretty obvious here that you need to stop the replication first before you can load the VDM.  So, as a next step, I stopped the replication with a simple right click/stop on the source side and tried again.

It failed with this error:

Error 4038: <interface_name_1> <interface_name_2> : interfaces not available on server_2

So, it looks like the interface names need to be the same.  I didn’t really want to change the interface names if I didn’t have to, so I tried a different approach next.

Failed attempt 2:

I thought this time I’d create a blank VDM on the destination side first and replicate the host VDM to it, thinking it wouldn’t keep the interface name requirement, and I still wouldn’t have to stop replication on the actual prod VDM, as I didn’t really want to use that one in a test.

I did just that. I created a blank VDM on the DR side, then started a new replication session from the host side and chose it as the destination, making sure to choose the overwrite option when I replicated to it.  The replication was successful.  I stopped the replication on the source side after it was complete, and then attempted to load the new replicated VDM on the DR side.

Voila! It worked:

nas_server –vdm <VDMNAME> -setstate loaded
            id          =          10
            name    =          vdm_replica
            acl        =          0
            type      =          vdm
            server   =          server_2
            rootfs    =          root_fs_vdm_replica
            I18N     =          UNICODE        
            Status   :
            Defined=          enabled
            Actual  =          loaded,ready

Now that it was loaded up, it was time to move on to the next step and create the R/W checkpoints of the filesystems. This is where the process failed again.

After clicking on the drop down box for “Choose Data Mover”, I got this error:

 No file systems exist

 Query file systems vdm_replica: All. File system not found. 

I’m not sure why this failed, but since the VDM couldn’t find the filesystems it was time to try another approach again.

Successful attempt:

After my first two failures, it looked pretty obvious that I’d need to change the interface names and use the original replicated VDM.  Making a copy of the VDM to a blank VDM didn’t work because it couldn’t see the filesystems, and using the original requires the interface names to be the same.  The lesson learned here is to make sure you have matching ports on your host and DR Celerras, and use the same interface names.  If I had done that, my first attempt would have been succesful.

If the original VDM has four CIFS servers (each with it’s own interface) and the DR Celerra only has one port configured on the network, you’d be out of luck.  You wouldn’t have enough interfaces to rename them all to match, and you’d never be able to load your VDM.  The VDM’s only look for the name to be the same, NOT the IP’s.  The IP’s can be different to match your DR network, and the IP’s that are already assigned to the DR site interfaces will NOT change when you load the VDM.

In my case, the host Celerra has two CIFS servers, each with it’s own interface.  One is for production, one is for backups.

Here are the steps that worked for me:

  1. Stop the replication of the VDM (You will see it change to a ‘stopped’ state in Unisphere).
  2. Change the interface names on the DR side (changing IP’s is not necessary) to match the host side.
  3. Load the VDM with the command  nas_server –vdm <VDMNAME> -setstate loaded
  4. You will see the VDM status change from ‘unloaded’ to ‘OK’.

Step 3:  Bring up the CIFS server at the DR site

After you’ve completed the previous step, the VDM will be loaded using the same exact same interfaces as production, and the CIFS servers will be automatically created as well.  If a CIFS server uses cge1-0 on server_2 on the host side, it will now be set up with the same name using cge1-0 on server_2 on the destination (DR) side.

This would be very useful in a real disaster, but for this test I wanted to create an alternate CIFS server with a different IP as the domain controller, DNS servers, and IP range used at our DR site is different.  You could choose to use the same CIFS server that was replicated with the VDM, but for our test I decided to bring up an entirely new CIFS server.  We use DFS for access all of our shares in production, so the name of the CIFS server won’t matter for our testing purposes.  We would just need to update DFS with the new name on the DR network.

Here are the steps I took to bring up the CIFS server for DR:

  1. Gather IP information from the DR team.  Will need a valid IP and subnet mask for the new CIFS server.
  2. Verify IP config on new DR network.
    1. Check that the default route matches the DR network
    2. Check that the DNS server entries match the DNS servers on the DR network
  3. Verify that the Domain controller in the DR network is up and available
  4. Modify the interface of your choice with the correct IP information for the CIFS server.
  5. Create the CIFS server and join it to the DR active directory domain.
    1. If you need to test an AD account, use this command:
    2. server_cifssupport <vdm_name> -cred -name -domain

That’s it for this step.  The CIFS server was successfully joined to the domain and I was able to ping it from one of our previously recovered windows servers on the DR network.

Step 4: Create Read/Write checkpoints of all replicated filesystems

One of my business requirements for this test was to allow read/write access to the replicated filesystems without having to actually change the production data.  The easy way to accomplish this is to create a single read/write checkpoint (snapshot) of each filesystem.  To do this, go to the checkpoint area in Unisphere, click create, and select the “Writeable Checkpoint” checkbox when you create the checkpoint.  You can also script the process and run it from the CLI on the control station.

First, create each checkpoint with this command:

nas_ckpt_schedule -create <ckpt_fs_name> -filesystem <fs_name> -recurrence once

Second, create a read/write copy of each checkpoint with this command:

fs_ckpt <ckpt_fs_name> -name <r/w_ckpt_fs_name>-Create -readonly n 

I would recommend running these no more than two a time and letting them finish.  I’ve had issues in the past running dozens of checkpoint jobs at once that hang and never complete, requiring a reboot of the data mover to correct.

Step 5: Share the replicated filesystems on the DR CIFS server

Once all of the R/W checkpoints are created, they can be shared on the DR CIFS server with the same share names as the original production share names. This allows all of our recovered application and file servers to connect to the same names, simplifying the configuration of the test environment.

You can use a CLI command to export each r/w copy to share them on your CIFS Server:

server_export [vdm] -P cifs -name [filesystem]_ckpt1 -option netbios=[cifserver] [filesystem]_ckpt1_writeable1

Step 6: Cleanup

That’s it!  We had a successful DR test.  Once the test was complete, I peformed the following cleanup steps:

  1. Remove CIFS server shares
  2. Remove CIFS server
  3. Change interfaces on DR celerra back to their original names and IP’s.
  4. Unload the replicated VDM with this command:
    1. nas_server –vdm <VDMNAME> -setstate mounted
    2. Restart the VDM replication from the source