Tag Archives: ionix

Upgrading EMC Ionix Control Center from Update Bundle 12 (UB12) to Update Bundle 14 (UB14)

Below are the steps I just took to upgrade our ECC environment from UB12 to UB14. There are quite a few steps and EMC has this process documented very well. Every install file and patch has it’s own readme file with detailed installation instructions along with additional info about special considerations depending on your environment. If you’re about to begin this upgrade, you can use these steps as a general guide but I highly recommend reading all of EMC’s documentation as well. There are general notes, warnings, and information in their documentation that did not apply to me but may apply to you. From beginning to end the upgrade process took me about seven hours to complete.

Prerequisites for the repository server:

    • Oracle 10g OCPU Update must be installed
    • Operating system locale settings must be set to English
    • Minimum 13GB of free disk space
    • Disable IOPath (ECC_IOPath.bat –i to check, ECC_IOPath.bat –d to disable)
    • You must back up the repository for ECC and StorageScope
    • ControlCenter Environment Checker must be run
    • All infrastructure services must be stopped
    • The following additional services must be stopped: AntiVirus, COM+, Server management services, vmware tools, Distributed Transaction Coordinator, Terminal Services, and WMI.

1. Download all of the required files. I’m running ECC on two 32 bit Windows servers (one repository server and one application server), so all of the download links I provided below are for windows.

a. Download the UB14 Update Bundle (32 bit is CC_4979.zip, 64 bit is CC_5013.zip):



b. Download the latest cumulative Patch (currently CC_5046.zip):


c. Download the latest Ionix ControlCenter 6.1 Backup and Restore Utility:


d. Download the latest Ionix ControlCenter Environment Checker (Currently version


e. Download the latest Oracle 10g OCPU Patch here (CC_5030.zip):


f. Download the latest Unisphere Host Client/Agent:


g. Download the latest Navisphere CLI (Current release is


h. I’d recommend reviewing the UB14 Read Me file to identify any specific warnings that may be relevant to your specific installation. The Read Me file for Update Bundle 14 can be downloaded here:


2. Log in to the repository server. Run the ECC Backup and Restore Utility. Because our ECC servers are virtualized, I also ran a snapshot from the VSphere console so I could revert back if needed. Below is an overview of the steps I took, review the readme file from EMC for more detail. Note that I shut down our ECC application server for the duration of the upgrade on the repository server.

a. Run the patch to install the files

b. Go to the ECC install directory, HF4938 folder, note timestamp on regutil610.exe

c. Check that the ECC/tools/util/regutil610.exe file has the same timestamp

d. Run ECC/HF498/backup.bat (that sets all ECC services to manual as well)

e. Reboot hosts, verify ECC services are not running

f. Manually back up the ECC folder to another location

3. Run Environment checker on the repository server.

a. Make sure that all checks have passed before proceeding with the upgrade.

4. Check ECC 6.1 compatibility matrix, make sure everything is up to date and compatible

a. Here’s a link to the compatibility matrix: https://support.emc.com/docu31657_Ionix-ControlCenter-6.1-Support-Matrix.pdf?language=en_US

b. Note that UB14 requires Solutions Enabler 7.5, 7.5.1 or 7.6.

c. Host Agent must be at 1.2.x and CLI must be at 7.32.x if you’re running the latest VNX hardware.

5. Run Oracle 10g OCPU Patch (CC_5030) on the repository server if required. This step took quite a bit of time to complete. I’d expect 60-90 minutes for this step.

a. Make sure all ECC application services are stopped.

b. Run %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_hotback.bat

c. Run %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ram_export_db.bat

d. Run %emcstsoh%\admin\emcsts_scripts\emcsts_coldback.bat

e. Run %emcstsoh%\admin\emcsts_scripts\emcsts_export_db.bat

f. Copy the contents of the %ECC_INSTALL_ROOT% to a backup location of your choice

g. Extract the SQL_PATCH_5030.zip to a local drive on the ECC Server.

h. Verify repository by running the following: %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_recomp_invalid.bat

i. Verify the storage scope repository by running the following: %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\emcsts_recomp_invalid.bat

j. If the prior two scripts ran successfully, proceed, if not call EMC.

k. Run the prereboot.bat script from the SQL_PATCH_5030 directory.

l. Reboot

m. Run the postreboot.bat script from the SQL_PATCH_5030 directory.

n. At the end of the postreboot script, it will ask if you want to revert all the services back to their original state. Respond with Y.

o. Verify successful installation by running %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\emcsts_hotfixstatus.bat

6. Update the Repository server with the latest Host agent and Navisphere CLI:

a. Unisphere Host Agent https://download.emc.com/downloads/DL30839_Unisphere_Host_Agent_(Windows_Â -_all_supported_32_&_64-bit_versions)__1.

b. Navisphere CLI https://download.emc.com/downloads/DL30859_Navisphere_CLI_(Windows_-_all_supported_32_&_64-bit_versions)_7.

c. A reboot is not required after installing the Agent and CLI. Make sure to preserve your security settings when asked in the installation.

7. Install a newer version of Solutions Enabler, if required. As mentioned earlier, UB14 requires version 7.5, 7.5.1 or 7.6.

a. Link to the MRLK Control Center supported version of SE (v7.5.0 – CC_5014): https://download.emc.com/downloads/DL44373_Solutions_Enabler_7.5_MRLK.zip

b. There is a readme pdf file included in the download with details on installation.

c. Stop the EMC storapid service.

d. Stop the EMC Control Center Server and Store Services.

e. Do NOT stop the OracleECCREP_HOMETNSListener or OracleServiceRAMBDB services.

f. Run the install.

g. Restart the services.

8. Prepare for UB14 Installation. Now all the prerequisite installs are complete on my system. The next step is to verify that the correct services are stopped and begin the UB14 upgrade. We have everything running on one repository server, your environment could be different.

a. On the repository server, stop the following services, set them to Manual, and reboot:

i. EMC ControlCenter API Server

ii. EMC ControlCenter Key Management Server

iii. EMC ControlCenter Master Agent

iv. EMC ControlCenter Repository

v. EMC ControlCenter Server

vi. EMC ControlCenter Store

vii. EMC ControlCenter STS MSA Web Server

viii. EMC ControlCenter Web Server

ix. EMC StorageScope Repository

x. EMC StorageScope Server

b. The following services must be stopped: AntiVirus, COM+, Server management services, vmware tools, Distributed Transaction Coordinator, and WMI.

c. Verify the following services are running before beginning the UB14 installation:

i. OracleECCREP_HOMETNSListener

ii. OracleServiceRAMBDB

iii. OracleServiceEMCSTSDB

9. Begin the UB14 Installation (CC_4079 for 32bit, CC_5013 for 64bit)

a. Execute the Patch61014383_4979_x86.exe patch (or the 64 bit patch if you’re on a 64bit server).

b. I recommend reading the UB14 readme file. If you encounter any errors, a few common install issues are listed.

c. This step takes a very long time, count on at least 3-4 hours. I also have StorageScope which added additional time.

d. After the patch install completes, change all the services back to their original ‘automatic’ state.

e. Reboot the repository server

10. Install the latest cumulative Update Bundle 14 patch next (The most current right now is CC_5046).

a. The current version of the UB14 patch can be downloaded here:


b. Stop the EMC Control Center Store and Server Services

c. Stop the following Services:

i. EMC ControlCenter API Server

ii. EMC ControlCenter Key Management Server

iii. EMC ControlCenter Master Agent

iv. EMC ControlCenter Repository

v. EMC ControlCenter Server

vi. EMC ControlCenter Store

vii. EMC ControlCenter STS MSA Web Server

viii. EMC ControlCenter Web Server

ix. EMC StorageScope Repository

x. EMC StorageScope Server

d. Run Patch 61014615_5046_hds.exe

e. Restart all Services that were stopped earlier in steps B and C.

f. Apply patch to agents:

i. Start the ECC Console

ii. Right click on the repository server object

iii. Select Agents à Apply Patch

iv. If the task fails, restart the master agent from the console

11. Update The application server

a. Install the latest Unisphere Host Agent https://download.emc.com/downloads/DL30839_Unisphere_Host_Agent_(Windows_Â -_all_supported_32_&_64-bit_versions)__1.

b. Install the latest Navisphere CLI https://download.emc.com/downloads/DL30859_Navisphere_CLI_(Windows_-_all_supported_32_&_64-bit_versions)_7.

c. Update the ECC Console by navigating to https://<repository_server>:30002/webinstall

i. Click Installation

ii. Click Console Patch

d. Install Solutions Enabler on the application server (not the MRLK version)

i. Windows 32 bit download


ii. Windows 64 bit download:


iii. Stop existing ECC and Solutions Enabler services

iv. Launch Install

e. Apply Agent patches from within ECC Console

i. Start the ECC Console

ii. Right click on the application server object

iii. Select Agents à Apply Patch

iv. If the task fails, restart the master agent from the console

12. Verify that WLA Archive collection (performance data) is collecting properly.

a. Go to your WLAArchives folder

b. Go to the Clariion\<serial number>\interval subfolder

c. Sort the files by date. If new files are being written then it is working.

13. Verify that all agents are running and are at the correct patch level

a. Launch the Control Center console

b. Click on the small gear icon on the lower right side of the window, which will launch the agents view on the right side of the screen

c. Verify that all agents are running and are patched to Any that require updates can be updated from this screen, right click, choose agents and install patch.

14. Remove backup directories (optional). The List is on page 16 & 17 of the UB14 readme file.

That’s it! You’re done.

Note: 24 hours after the completion of the upgrade I noticed that WLA archive data collection wasn’t working for some of the arrays. I deleted the arrays that were’nt working and rediscovered them, which resolved the problem. Deleting the arrays removes all historical data from StorageScope.

How to troubleshoot EMC Control Center WLA Archive issues

We’re running EMC Control Center 6.1 UB12, and we use it primarly for it’s robust performance data collection and reporting capabilities.  Performance Manager is a great tool and I use it frequently.

Over the years I’ve had occasional issues with the WLA Archives not collecting performance data and I’ve had to open service requests to get it fixed.  Now that I’ve been doing this for a while, I’ve collected enough info to troubleshoot this issue and correct it without EMC’s assistance in most cases.

Check your ..\WLAArchives\Archives directory and look under the Clariion (or Celerra) folder, then the folder with your array’s serial number, then the interval folder.  This is where the “*.ttp” (text) and “*.btp” (binary) performance data files are stored for Performance Manager.  Sort by date.  If there isn’t a new file that’s been written in the last few hours data is not being collected.

Here are the basic items I generally review when data isn’t being collected for an array:

  1. Log in to every array in Unisphere, go to system properties, and on the ‘General’ tab make sure statistics logging is enabled.  I’ve found that if you don’t have an analyzer license on your array and start the 7 day data collection for a “naz” file, after the 7 days is up the stats logging option will be disabled.  You’ll have to go back in and re-enable it after the 7 day collection is complete.  If stats logging isn’t enabled on the array the WLA data collection will fail.
  2. If you recently changed the password on your clarion domain account, Make sure that naviseccli is updated properly for security access to all of your arrays (use the “addusersecurity” CLI option) and perform a rediscovery of all your arrays as well from within the ECC console.  There is no way from within the ECC console to update the password on an array, you must go through the discovery process again for all of them.
  3.  Verify the agents are running.  In the ECC console, click on the gears icon in the lower right hand corner.  It will create a window that shows the status of all the agents, including the WLA Archiver.  If WLA isn’t started, you can start it by right clicking on any array, choosing Agents, then start.  Check the WLAArchives  directories again (after waiting about an hour) and see if it’s collecting data again.

If those basic steps don’t work, checking the logs may point you in the right direction:

  1.  Review the Clariion agent logs for errors.  You’re not looking for anything specific here, just do a search for “error”, “unreachable” or for the specific IP’s of your arrays and see if there is anything obvious wrong. 

Here’s an example of an error I found in one case:

            MGL 14:10:18 C P I 2536   (29.94 MB) [MGLAgent::ProcessAlert] => Processing SP
            Unreachable alert. MO = APM00100600999, Context = Clariion, Category = SP
            Element = Unreachable

      2.   Review the WLA Agent logs.  Again, just search for errors and see if there is anything obvious that’s wrong. 


If the logs don’t show anything obvious, here are the steps I take to restart everything.  This has worked on several occasions for me.

  1. From the Control Center console, stop all agents on the ECC Agent server.  Do this by right clicking on the agent server (in the left pane), choose agents and stop.  Follow the prompts from there.
  2. Log in to the ECC Agent server console and stop the master agent.  You can do this in Computer Management | Services, stop the service titled “EMC ControlCenter Master Agent”.
  3. From the Control Center console, stop all agents on the Infrastructure server.  Do this by right clicking on the agent server (in the left pane), choose agents and stop.  Follow the prompts from there.
  4. Verify that all services have stopped properly.
  5. From the ECC Agent server console, go to C:\Windows\ECC\ and delete all .comfile and .lck files.
  6. Restart all agents on the Infrastructure server.
  7. Restart the Master Agent on the Agent server.
  8. Restart all other services on the Agent server.
  9. Verify that all services have restarted properly.
  10. Wait at least an hour and check to see if the WLA Archive files are being written.

If none of these steps resolve your problem and you don’t see any errors in the logs, it’s time to open an SR with EMC.  I’ve found the EMC staff  that supports ECC to be very knowledgeable and helpful.



Auto generating daily performance graphs with EMC Control Center / Performance Manager

This document describes the process I used to pull performance data using the ECC pmcli command line tool, parse the data to make it more usable with a graphing tool, and then use perl scripts to automatically generate graphs.

You must install Perl.  I use ActiveState Perl (Free Community Edition) (http://www.activestate.com/activeperl/downloads).

You must install Cygwin.  Link: http://www.cygwin.com/install.html. I generally choose all packages.

I use the follow CPAN Perl modules:

Step 1:

Once you have the software set up, the first step is to use the ECC command line utility to extract the interval performance data that you’re interested in graphing.  Below is a sample PMCLI command line script that could be used for this purpose.

:Get the current date

For /f “tokens=2-4 delims=/” %%a in (‘date /t’) do (set date=%%c%%a%%b)

:Export the interval file for today’s date.

D:\ECC\Client.610\PerformanceManager\pmcli.exe -export -out D:\archive\interval.csv -type interval -class clariion -date %date% -id APM00324532111

:Copy all the export data to my cygwin home directory for processing later.

copy /y e:\san712_interval.csv C:\cygwin\home\<userid>

You can schedule the command script above to run using windows task scheduler.  I run it at 11:46PM every night, as data is collected on our SAN in 15 minute intervals, and that gives me a file that reports all the way up to the end of one calendar day.

Note that there are 95 data collection points from 00:15 to 23:45 every day if you collect data at 15 minute intervals.  The storage processor data resides in the last two lines of the output file.

Here is what the output file looks like:

EMC ControlCenter Performance manager generated file from: <path>

Data Collected for DiskStats

Data Collected for DiskStats – 0_0_0

                                                             3/28/11 00:15       3/28/11 00:30      3/28/11  00:45      3/28/11 01:00 

Number of Arrivals with Non Zero Queue     12                         20                        23                      23 

% Utilization                                                30.2                     33.3                     40.4                  60.3

Response Time                                              1.8                        3.3                        5.4                     7.8

Read Throughput IO per sec                        80.6                    13.33                   90.4                    10.3

Great information in there, but the format of the data makes it very hard to do anything meaningful with the data in an excel chart.  If I want to chart only % utilization, that data is difficult to chart because there are so many counters around it that are also have data collected on them.   My next goal was to write a script to reformat the data in a much more usable format to automatically create a graph for one specific counter that I’m interested in (like daily utilization numbers), which could then be emailed daily or auto-uploaded to an internal website.

Step 2:

Once the PMCLI data is exported, the next step is to use cygwin bash scripts to parse the csv file and pull out only the performance data that is needed.  Each SAN will need a separate script for each type of performance data.  I have four scripts configured to run based on the data that I want to monitor.  The scripts are located in my cygwin home directory.

The scripts I use:

  • Iostats.sh (for total IO throughput)
  • Queuestats.sh (for disk queue length)
  • Resptime.sh (for disk response time in ms)
  • Utilstats.sh (for % utilization)

Here is a sample shell script for parsing the CSV export file (iostats.sh):


#This will pull only the timestamp line from the top of the CSV output file. I’ll paste it back in later.

grep -m 1 “/” interval.csv > timestamp.csv

#This will pull out only lines that begin with “total througput io per sec”.

grep -i “^Total Throughput IO per sec” interval.csv >> stats.csv

#This will pull out the disk/LUN title info for the first column.  I’ll add this back in later.

grep -i “Data Collected for DiskStats -” interval.csv > diskstats.csv

grep -i “Data Collected for LUNStats -” interval.csv > lunstats.csv

#This will create a column with the disk/LUN number .  I’ll paste it into the first column later.

cat diskstats.csv lunstats.csv > data.csv

#This adds the first column (disk/LUN) and combines it with the actual performance data columns.

paste data.csv stats.csv > combined.csv

#This combines the timestamp header at the top with the combined file from the previous step to create the final file we’ll use for the graph.  There is also a step to append the current date and copy the csv file to an archive directory.

cat timestamp.csv combined.csv > iostats.csv

cp iostats.csv /cygdrive/e/SAN/csv_archive/iostats_archive_$(date +%y%m%d).csv

#  This removes all the temporary files created earlier in the script.  They’re no longer needed.

rm timestamp.csv

rm stats.csv

rm diskstats.csv

rm lunstats.csv

rm data.csv

rm combined.csv

#This strips the last two lines of the CSV (Storage Processor data).  The resulting file is used for the “all disks” spreadsheet.  We don’t want the SP
data to skew the graph.  This CSV file is also copied to the archive directory.

sed ‘$d’ < iostats.csv > iostats2.csv

sed ‘$d’ < iostats2.csv > iostats_disk.csv

rm iostats2.csv

cp iostats_disk.csv /cygdrive/e/SAN/csv_archive/iostats_disk_archive_$(date +%y%m%d).csv

Note: The shell script above can be run in the windows task scheduler as long as you have cygwin installed.  Here’s the syntax:

c:\cygwin\bin\bash.exe -l -c “/home/<username>/iostats.sh”

After running the shell script above, the resulting CSV file contains only Total Throughput (IO per sec) data for each disk and lun.  It will contain data from 00:15 to 23:45 in 15 minute increments.  After the cygwin scripts have run we will have csv datasets that are ready to be exported to a graph.

The Disk and LUN stats are combined into the same CSV file.  It is entirely possible to rewrite the script to only have one or the other.  I put them both in there to make it easier to manually create a graph in excel for either disk or lun stats at a later time (if necessary).  The “all disks graph” does not look any different with both disk and lun stats in there, I tried it both ways and they overlap in a way that makes the extra data indistinguishable in the image.

The resulting data output after running the iostats.sh script is shown below.  I now have a nice, neat excel spreadsheet that lists the total throughput for each disk in the array for the entire day in 15 minute increments.   Having the data formatted in this way makes it super easy to create charts.  But I don’t want to have to do that manually every day, I want the charts to be created automatically.

                                                             3/28/11 00:15       3/28/11 00:30      3/28/11  00:45      3/28/11 01:00

Total Throughput IO per sec   – 0_0_0          12                             20                             23                           23 

Total Throughput IO per sec    – 0_0_1        30.12                        33.23                        40.4                         60.23

Total Throughput IO per sec    – 0_0_2         1.82                          3.3                           5.4                              7.8

Total Throughput IO per sec    -0_0_3         80.62                        13.33                        90.4                         10.3 

Step 3:

Now I want to automatically create the graphs every day using a Perl script.  After the CSV files are exported to a more usable format from the previous step, I Use the GD::Graph library from CPAN (http://search.cpan.org/~mverb/GDGraph-1.43/Graph.pm) to auto-generate the graphs.

Below is a sample Perl script that will autogenerate a great looking graph based on the CSV ouput file from the previous step.


#Declare the libraries that will be used.

use strict;

use Text::ParseWords;

use GD::Graph::lines;

use Data::Dumper;

#Specify the csv file that will be used to create the graph

my $file = ‘C:\cygwin\home\<username>\iostats_disk.csv’;

#my $file  = $ARGV[0];

my ($output_file) = ($file =~/(.*)\./);

#Create the arrays for the data and the legends

my @data;

my @legends;

#parse csv, generate an error if it fails

open(my $fh, ‘<‘, $file) or die “Can’t read csv file ‘$file’ [$!]\n”;

my $countlines = 0;

while (my $line = <$fh>) {

chomp $line;

my @fields = Text::ParseWords::parse_line(‘,’, 0, $line);

#There are 95 fields generated to correspond to the 95 data collection points in each
of the output files.

my @field =

push @data, \@field;

if($countlines >= 1){

push @legends, @fields[0];




#The data and legend arrays will read 820 lines of the CSV file.  This number will change based on the number of disks in the SAN, and will be different depending on the SAN being reported on.  The legend info will read the first column of the spreadsheet and create a color box that corresponds to the graph line.  For the purpose of this graph, I won’t be using it because 820+ legend entries look like a mess on the screen.

splice @data, 1, -820;

splice @legends, 0, -820;

#Set Graphing Options

my $mygraph = GD::Graph::lines->new(1024, 768);

# There are many graph options that can be changed using the GD::Graph library.  Check the website (and google) for lots of examples.


title => ‘SP IO Utilization (00:15 – 23:45)’,

y_label => ‘IOs Per Second’,

y_tick_number => 4,

values_vertical => 6,

show_values => 0,

x_label_skip => 3,

) or warn $mygraph->error;

#As I said earlier, because of the large number of legend entries for this type of graph, I change the legend to simply read “All disks”.  If you want the legend to actually put the correct entries and colors, use this line instead:  $mygraph->set_legend(@legends);

$mygraph->set_legend(‘All Disks’);

#Plot the data

my $myimage = $mygraph->plot(\@data) or die $mygraph->error;

# Export the graph as a gif image.  The images are currently moved to the IIS folder (c:\inetpub\wwwroot) with one of the scripts.  The could also be emailed using a sendmail utility.

my $format = $mygraph->export_format;

open(IMG,”>$output_file.$format”) or die $!;

binmode IMG;

print IMG $myimage->gif;

close IMG;

After this script runs the resulting image file will be saved in the cygwin home directory (It saves it in the same directory that the CSV file is located in).  One of the nightly scripts I run will copy the image to our interal IIS server’s image directory, and sendmail will email the graph to the SAN Admin team.

That’s it!  You now have lots of pretty graphs with which you can impress your management team. 🙂

Here is a sample graph that was generated with the Perl script: