Disk space reporting on sub folders on a VNX File CIFS shared file system

I recently had a comment on a different post asking how to report on the size of multiple folders on a single file system, so I thought I’d share the method I use to create reports and alerts on the space that sub folders on file systems consume. There is a way to navigate to all of the file systems from the control station, simply navigate to /nas/quota/slot_<x>. Slot_<x> refers to the data mover. The file systems on server_2 would be in the slot_2 folder.  Because we have access to those folders, we can simply run the standard unix ‘du’ command to get the amount of used disk space for each sub folder on any file system.

Running this command:

sudo du -h /nas/quota/slot_2/File_System/Sub_Folder

Will give an output that looks like this:

24K     /nas/quota/slot_2/File_System/Sub_Folder/1
0       /nas/quota/slot_2/File_System/Sub_Folder/2
16K     /nas/quota/slot_2/File_System/Sub_Folder

Each sub folder of the file system named “File_System” is listed with the space each sub folder uses. You’ll notice that I used the sudo command to run the du command. Unfortunately you need root access in order to run the du command on the /nas/quota directory, so you’ll have to add whatever account you log in to the celerra with to the /etc/sudoers file. If you don’t, you’ll get the error “Sorry, user nasadmin is not allowed to execute ‘/usr/bin/du -h -s /nas/quota/slot_2/File_System’ as root on “. I generally log in to the celerra as nasadmin, so I added that account to sudoers. To modify the file, su to root first and then (using vi) add the following to the very bottom of the file of /etc/sudoers (substitute nasadmin for the username you will be using):

nasadmin ALL=/usr/bin/du, /usr/bin/, /sbin/, /usr/sbin, /nas/bin, /nas/bin/, /nas/sbin, /nas/sbin/, /bin, /bin/
nasadmin ALL=/nas/bin/server_df

Once that is complete, you’ll have access to run the command.  You can now write a script that will report on specific folders for you. I have two scripts created that I’m going to share, one I use to send out a daily email report and the other will send an email only if the folder sizes have crossed a certain threshold of space utilization. I have them both scheduled as a cron job on the Celerra itself.  It should be easy to modify these scripts for anyone’s environment.

The following script will generate a report of folder and sub folder sizes for any given file system. In this example, I am reporting on three specific subfolders on a file system (Production, Development, and Test). I also use the grep and awk options at the beginning to pull out the percent full from the df command for the entire file system and include that in the subject line of the email.

#!/bin/bash
 NAS_DB="/nas"
 export NAS_DB
 TODAY=$(date)
 HOST=$(hostname)

PERCENT=`sudo df -h /nas/quota/slot_2/File_System | grep server_2 | awk '{print $5}'`

echo "File_System Folder Size Report Date: $TODAY Host:$HOST" > /home/nasadmin/report/fs_report.txt
 echo " " >> /home/nasadmin/report/fs_report.txt
 echo "Production" >> /home/nasadmin/report/fs_report.txt
 echo " " >> /home/nasadmin/report/fs_report.txt

sudo du -h -S /nas/quota/slot_2/File_System/Prod >> /home/nasadmin/report/fs_report.txt

echo " " >> /home/nasadmin/report/fs_report.txt
 echo "Development" >> /home/nasadmin/report/fs_report.txt
 echo " " >> /home/nasadmin/report/fs_report.txt

sudo du -h -S /nas/quota/slot_2/File_System/Dev >> /home/nasadmin/report/fs_report.txt

echo " " >> /home/nasadmin/report/fs_report.txt
 echo "Test" >> /home/nasadmin/report/fs_report.txt
 echo " " >> /home/nasadmin/report/fs_report.txt

sudo du -h -S /nas/quota/slot_2/File_System/Test >> /home/nasadmin/report/fs_report.txt

echo " " >> /home/nasadmin/report/fs_report.txt
 echo "% Remaining on File_System Filesystem" >> /home/nasadmin/report/fs_report.txt
 echo " " >> /home/nasadmin/report/fs_report.txt

sudo df -h /nas/quota/slot_2/File_System/ >> /home/nasadmin/report/fs_report.txt

echo $PERCENT

mail -s "Folder Size Report ($PERCENT In Use)" youremail@domain.com < /home/nasadmin/report/fs_report.txt

Below is what the output of that script looks like. It is included in the body of the email as plain text.

File_System Folder Size Report
Date: Fri Jun 28 08:00:02 CDT 2013
Host:<celerra_name>

Production

 24K     /nas/quota/slot_2/File_System/Production/1
 0       /nas/quota/slot_2/File_System/Production/2
 16K     /nas/quota/slot_2/File_System/Production

Development

 8.0K    /nas/quota/slot_2/File_System/Development/1
 108G    /nas/quota/slot_2/File_System/Development/2
 0          /nas/quota/slot_2/File_System/Development/3
 16K     /nas/quota/slot_2/File_System/Development

Test

 0       /nas/quota/slot_2/File_System/Test/1
 422G    /nas/quota/slot_2/File_System/Test/2
 0       /nas/quota/slot_2/File_System/Test/3
 16K     /nas/quota/slot_2/File_System/Test

% Remaining on File_System Filesystem

Filesystem Size Used Avail Use% Mounted on
 server_2:/ 2.5T 529G 1.9T 22% /nasmcd/quota/slot_2
 .

The following script will only send an email if the folder size has crossed a defined threshold. Simply change the path to the folder you want to report on and change the value of the ‘threshold’ variable in the script to whatever you want it to be, I have mine set at 95%.

#Get the % Disk Utilization from the server_df command
 percent=`sudo df -h /nas/quota/slot_2/File_System | grep server_2 | awk '{print $5}'`

#Strip the % Sign from the output value
 percentvalue=`echo $percent | awk -F"%" '{print $1}'`

#Set the critical threshold that will trigger an email
 threshold=95

#compare the threshold value to the reported value, send an email if needed
 if [ $percentvalue -eq 0 ] && [ $threshold -eq 0 ]
 then
  echo "Both are zero"
 elif [ $percentvalue -eq $threshold ]
 then
  echo "Both Values are equal"
 elif [ $percentvalue -gt $threshold ]
 then
 mail -s "File_System Critical Disk Space Alert: $percentvalue% utilization is above the threshold of $threshold%" youremail@domain.com < /home/nasadmin/report/fs_report.txt

else
  echo "$percentvalue is less than $threshold"
 fi
Advertisements

Upgrading EMC Ionix Control Center from Update Bundle 12 (UB12) to Update Bundle 14 (UB14)

Below are the steps I just took to upgrade our ECC environment from UB12 to UB14. There are quite a few steps and EMC has this process documented very well. Every install file and patch has it’s own readme file with detailed installation instructions along with additional info about special considerations depending on your environment. If you’re about to begin this upgrade, you can use these steps as a general guide but I highly recommend reading all of EMC’s documentation as well. There are general notes, warnings, and information in their documentation that did not apply to me but may apply to you. From beginning to end the upgrade process took me about seven hours to complete.

Prerequisites for the repository server:

    • Oracle 10g OCPU Update must be installed
    • Operating system locale settings must be set to English
    • Minimum 13GB of free disk space
    • Disable IOPath (ECC_IOPath.bat –i to check, ECC_IOPath.bat –d to disable)
    • You must back up the repository for ECC and StorageScope
    • ControlCenter Environment Checker must be run
    • All infrastructure services must be stopped
    • The following additional services must be stopped: AntiVirus, COM+, Server management services, vmware tools, Distributed Transaction Coordinator, Terminal Services, and WMI.

1. Download all of the required files. I’m running ECC on two 32 bit Windows servers (one repository server and one application server), so all of the download links I provided below are for windows.

a. Download the UB14 Update Bundle (32 bit is CC_4979.zip, 64 bit is CC_5013.zip):

https://download.emc.com/downloads/DL44408_ControlCenter_Update_Bundle_14_(64-bit)_software_download.zip

https://download.emc.com/downloads/DL44407_ControlCenter_Update_Bundle_14_(32-bit)_software_download.zip

b. Download the latest cumulative Patch (currently CC_5046.zip):

https://download.emc.com/downloads/DL47699_Cumulative_6.1_Update_Bundle_14_Storage_Agent_for_HDS_Patch_5046_Download.zip

c. Download the latest Ionix ControlCenter 6.1 Backup and Restore Utility:

https://download.emc.com/downloads/DL37276_Ionix_ControlCenter_6.1_Backup_and_Restore_Utility_.zip

d. Download the latest Ionix ControlCenter Environment Checker (Currently version 1.0.5.3):

https://download.emc.com/downloads/DL37274_Ionix_ControlCenter_Environment_Checker_1.0.5.3.exe

e. Download the latest Oracle 10g OCPU Patch here (CC_5030.zip):

https://download.emc.com/downloads/DL46584_ControlCenter_6.1_OCPU_(5030)_Download.zip

f. Download the latest Unisphere Host Client/Agent:

https://download.emc.com/downloads/DL30840_Unisphere__Client_(Windows)_1.2.26.1.0094.exe

g. Download the latest Navisphere CLI (Current release is 7.32.25.1.63):

https://download.emc.com/downloads/DL30859_Navisphere_CLI_(Windows_-_all_supported_32_&_64-bit_versions)_7.32.25.1.63.exe

h. I’d recommend reviewing the UB14 Read Me file to identify any specific warnings that may be relevant to your specific installation. The Read Me file for Update Bundle 14 can be downloaded here:

https://support.emc.com/docu44826_Ionix_ControlCenter_6.1_Update_Bundle_14_Read_Me.pdf?language=en_US

2. Log in to the repository server. Run the ECC Backup and Restore Utility. Because our ECC servers are virtualized, I also ran a snapshot from the VSphere console so I could revert back if needed. Below is an overview of the steps I took, review the readme file from EMC for more detail. Note that I shut down our ECC application server for the duration of the upgrade on the repository server.

a. Run the patch to install the files

b. Go to the ECC install directory, HF4938 folder, note timestamp on regutil610.exe

c. Check that the ECC/tools/util/regutil610.exe file has the same timestamp

d. Run ECC/HF498/backup.bat (that sets all ECC services to manual as well)

e. Reboot hosts, verify ECC services are not running

f. Manually back up the ECC folder to another location

3. Run Environment checker on the repository server.

a. Make sure that all checks have passed before proceeding with the upgrade.

4. Check ECC 6.1 compatibility matrix, make sure everything is up to date and compatible

a. Here’s a link to the compatibility matrix: https://support.emc.com/docu31657_Ionix-ControlCenter-6.1-Support-Matrix.pdf?language=en_US

b. Note that UB14 requires Solutions Enabler 7.5, 7.5.1 or 7.6.

c. Host Agent must be at 1.2.x and CLI must be at 7.32.x if you’re running the latest VNX hardware.

5. Run Oracle 10g OCPU Patch (CC_5030) on the repository server if required. This step took quite a bit of time to complete. I’d expect 60-90 minutes for this step.

a. Make sure all ECC application services are stopped.

b. Run %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_hotback.bat

c. Run %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ram_export_db.bat

d. Run %emcstsoh%\admin\emcsts_scripts\emcsts_coldback.bat

e. Run %emcstsoh%\admin\emcsts_scripts\emcsts_export_db.bat

f. Copy the contents of the %ECC_INSTALL_ROOT% to a backup location of your choice

g. Extract the SQL_PATCH_5030.zip to a local drive on the ECC Server.

h. Verify repository by running the following: %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_recomp_invalid.bat

i. Verify the storage scope repository by running the following: %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\emcsts_recomp_invalid.bat

j. If the prior two scripts ran successfully, proceed, if not call EMC.

k. Run the prereboot.bat script from the SQL_PATCH_5030 directory.

l. Reboot

m. Run the postreboot.bat script from the SQL_PATCH_5030 directory.

n. At the end of the postreboot script, it will ask if you want to revert all the services back to their original state. Respond with Y.

o. Verify successful installation by running %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\emcsts_hotfixstatus.bat

6. Update the Repository server with the latest Host agent and Navisphere CLI:

a. Unisphere Host Agent 1.2.25.1.0163: https://download.emc.com/downloads/DL30839_Unisphere_Host_Agent_(Windows_Â -_all_supported_32_&_64-bit_versions)__1.2.25.1.0163.exe

b. Navisphere CLI 7.32.25.1.63: https://download.emc.com/downloads/DL30859_Navisphere_CLI_(Windows_-_all_supported_32_&_64-bit_versions)_7.32.25.1.63.exe

c. A reboot is not required after installing the Agent and CLI. Make sure to preserve your security settings when asked in the installation.

7. Install a newer version of Solutions Enabler, if required. As mentioned earlier, UB14 requires version 7.5, 7.5.1 or 7.6.

a. Link to the MRLK Control Center supported version of SE (v7.5.0 – CC_5014): https://download.emc.com/downloads/DL44373_Solutions_Enabler_7.5_MRLK.zip

b. There is a readme pdf file included in the download with details on installation.

c. Stop the EMC storapid service.

d. Stop the EMC Control Center Server and Store Services.

e. Do NOT stop the OracleECCREP_HOMETNSListener or OracleServiceRAMBDB services.

f. Run the install.

g. Restart the services.

8. Prepare for UB14 Installation. Now all the prerequisite installs are complete on my system. The next step is to verify that the correct services are stopped and begin the UB14 upgrade. We have everything running on one repository server, your environment could be different.

a. On the repository server, stop the following services, set them to Manual, and reboot:

i. EMC ControlCenter API Server

ii. EMC ControlCenter Key Management Server

iii. EMC ControlCenter Master Agent

iv. EMC ControlCenter Repository

v. EMC ControlCenter Server

vi. EMC ControlCenter Store

vii. EMC ControlCenter STS MSA Web Server

viii. EMC ControlCenter Web Server

ix. EMC StorageScope Repository

x. EMC StorageScope Server

b. The following services must be stopped: AntiVirus, COM+, Server management services, vmware tools, Distributed Transaction Coordinator, and WMI.

c. Verify the following services are running before beginning the UB14 installation:

i. OracleECCREP_HOMETNSListener

ii. OracleServiceRAMBDB

iii. OracleServiceEMCSTSDB

9. Begin the UB14 Installation (CC_4079 for 32bit, CC_5013 for 64bit)

a. Execute the Patch61014383_4979_x86.exe patch (or the 64 bit patch if you’re on a 64bit server).

b. I recommend reading the UB14 readme file. If you encounter any errors, a few common install issues are listed.

c. This step takes a very long time, count on at least 3-4 hours. I also have StorageScope which added additional time.

d. After the patch install completes, change all the services back to their original ‘automatic’ state.

e. Reboot the repository server

10. Install the latest cumulative Update Bundle 14 patch next (The most current right now is CC_5046).

a. The current version of the UB14 patch can be downloaded here:

https://download.emc.com/downloads/DL47699_Cumulative_6.1_Update_Bundle_14_Storage_Agent_for_HDS_Patch_5046_Download.zip

b. Stop the EMC Control Center Store and Server Services

c. Stop the following Services:

i. EMC ControlCenter API Server

ii. EMC ControlCenter Key Management Server

iii. EMC ControlCenter Master Agent

iv. EMC ControlCenter Repository

v. EMC ControlCenter Server

vi. EMC ControlCenter Store

vii. EMC ControlCenter STS MSA Web Server

viii. EMC ControlCenter Web Server

ix. EMC StorageScope Repository

x. EMC StorageScope Server

d. Run Patch 61014615_5046_hds.exe

e. Restart all Services that were stopped earlier in steps B and C.

f. Apply patch to agents:

i. Start the ECC Console

ii. Right click on the repository server object

iii. Select Agents à Apply Patch

iv. If the task fails, restart the master agent from the console

11. Update The application server

a. Install the latest Unisphere Host Agent 1.2.25.1.0163: https://download.emc.com/downloads/DL30839_Unisphere_Host_Agent_(Windows_Â -_all_supported_32_&_64-bit_versions)__1.2.25.1.0163.exe

b. Install the latest Navisphere CLI 7.32.25.1.63: https://download.emc.com/downloads/DL30859_Navisphere_CLI_(Windows_-_all_supported_32_&_64-bit_versions)_7.32.25.1.63.exe

c. Update the ECC Console by navigating to https://<repository_server>:30002/webinstall

i. Click Installation

ii. Click Console Patch 6.1.0.14.383

d. Install Solutions Enabler on the application server (not the MRLK version)

i. Windows 32 bit download

https://download.emc.com/downloads/DL43505_se7500-WINDOWS-x86.exe.exe

ii. Windows 64 bit download:

https://download.emc.com/downloads/DL43504_se7500-WINDOWS-x64.exe.exe

iii. Stop existing ECC and Solutions Enabler services

iv. Launch Install

e. Apply Agent patches from within ECC Console

i. Start the ECC Console

ii. Right click on the application server object

iii. Select Agents à Apply Patch

iv. If the task fails, restart the master agent from the console

12. Verify that WLA Archive collection (performance data) is collecting properly.

a. Go to your WLAArchives folder

b. Go to the Clariion\<serial number>\interval subfolder

c. Sort the files by date. If new files are being written then it is working.

13. Verify that all agents are running and are at the correct patch level

a. Launch the Control Center console

b. Click on the small gear icon on the lower right side of the window, which will launch the agents view on the right side of the screen

c. Verify that all agents are running and are patched to 6.1.0.14.383. Any that require updates can be updated from this screen, right click, choose agents and install patch.

14. Remove backup directories (optional). The List is on page 16 & 17 of the UB14 readme file.

That’s it! You’re done.

Note: 24 hours after the completion of the upgrade I noticed that WLA archive data collection wasn’t working for some of the arrays. I deleted the arrays that were’nt working and rediscovered them, which resolved the problem. Deleting the arrays removes all historical data from StorageScope.

What’s new in EMC Control Center Update Bundle 14 (UB14)?

I’m in the process right now of upgrading our ECC installation from UB12 to UB14.  I’ll be making another post soon with the steps I took to complete the upgrade.  Below is a list of all the new features in UB14.

  • Added support for Oracle 11g.
  • Added Oracle 64-bit support in ControlCenter for 64-bit operating systems.
  • Added support for VMAX 1.5 and CKD devices.
  • Added support for Solutions Enabler 7.5.
  • Added support to discover and display new Federated Tiered Storage (FTS).
  • Added support for EMC CentraStar® 4.2.2.
  • Added support for Java 1.7 update 7.
  • Added support for HDS WLA by using JRE 1.7.
  • Added support for the availability of pool compression information in STS query builder while generating custom reports.
  • Added support for Mixed RAID types in VNX storage pool.
  • Added a new memory check during installation to verify if 4GB of RAM is available to install/upgrade UB14 in an all-in-one setup.
  • Added a new memory check during installation to verify if 3GB of RAM is available to install/upgrade UB14 in a distributed setup.
  • Added a new disk space check during installation to verify if minimum disk space is available for UB14 upgrade in both all-in-one and distributed setups and notify users if the requirement is not met.
  • Added support to export/import database to/from an external drive.
  • Added new free disk space availability alert.
  • Removed support for Symmetrix, SDM, and CLARiiON agents on HP-UX and AIX operating systems.
  • EMC Ionix ControlCenter 6.1 Update Bundle 14 Read Me 3
  • Removed support for Microsoft Internet Explorer version 6.0.
  • Added support to discover or rediscover Managed Objects through NAS, CLARiiON, Symmetrix, and Host Agents using DefaultHiddenDCP feature.
  • Added support to change the fully qualified domain name of non-infrastructure hosts or platforms.
  • Added support to remove virtual machines marked as Deleted on the console automatically using a scheduled job that runs every day.
  • Added support to reserve connections for WLA communications.
  • Added support to optimize the mechanism of sending MOLIST for Brocade switches.
  • Added support to optimize performance statistics collection for WLA.
  • Added support to optimize provider calls for switch and fabric discovery.
  • Added support to optimize logging and error handling in agent.
  • Added support to start processing opcode only after the agent INITX initialization.
  • Added virtual provisioning support for CLARiiON FLARE 28.
  • Added support to edit MessageReplyTimeout value in Gateway agent ini file.
  • Improved error messages in console for action.
  • Improved Store log to simplify root cause determination of issues.
  • Reduced Integration Gateway agent crashes.

Brocade Switch Type Matrix

I recently performed an inventory of all of our Brocade switches and stumbled upon this list of switch types that allows you to identify the Brocade model number.  Simply go to http:///SwitchInfo.html, do a search for “switchType” in the report, and compare that number to the table below to identify your model.

12

3900

2 Gb 32-port switch

16

3200

2 Gb 8-port value line switch

21

24000

2 Gb 128-port core fabric switch

26

3850

2 Gb 16-port switch with switch limit

27

3250

2 Gb 8-port switch with switch limit

29

4012

2 Gb 12-port Blade Server SAN I/O Module

34

200E

2 Gb 16-port switch with switch limit

37

4020

2 Gb 20-port Blade Server SAN I/O Module

43

4024

4 Gb 24-port Blade Server SAN I/O Module

44

4900

4 Gb 64-port switch

45

4016

2 Gb 16-port Blade Server SAN I/O Module

51

4018

2 Gb 16/18-port Blade Server SAN I/O Module

61

4424

2 Gb 24-port Blade Server SAN I/O Module

62

DCX

8 Gb 798-port core fabric backbone

64

5300

8 Gb 80-port switch

66

5100

8 Gb 40-port switch

67

Encryption Switch

8 Gb 16-port encryption switch

70

5410

8 Gb 12-port Blade Server SAN I/O Module

71

300

8 Gb 16-port switch

72

5480

8 Gb 24-port Blade Server SAN I/O Module

73

5470

8 Gb 20-port Blade Server SAN I/O Module

75

M5424

8 Gb 24-port Blade Server SAN I/O Module

77

DCX-4S

8 Gb 192-port core fabric backbone

83

7800

8 Gb 16-FC ports, 6 GbE ports extension switch

86

5450

8 Gb 26-port Blade Server SAN I/O Module

87

5460

8 Gb 26-port Blade Server SAN I/O Module

92

VA-40FC

8 Gb 40-port switch

109

6510

16 Gb 48-port switch

117

6547

16 Gb 48-port Blade Server SAN I/O Module

118

6505

16 Gb 24-port switch

120

DCX 8510-8

16 Gb 512-port core fabric backbone

121

DCX 8510-4

16 Gb 256-port core fabric backbone

124

5430

8 Gb 16-port Blade Server SAN I/O Module

125

5431

8 Gbit 16-port stackable switch module

129

6548

16 Gb 28-port Blade Server SAN I/O Module

130

M6505

16 Gbit 24-port Blade Server SAN I/O Module

133

6520

16 Gb 96-port switch

134

5432

8 Gb 24-port Blade Server SAN I/O Module

148

7840

16 Gb 24-FC ports, 16 10GbE ports, 2 40GbE ports extension switch