Tag Archives: steps

Upgrading EMC Ionix Control Center from Update Bundle 12 (UB12) to Update Bundle 14 (UB14)

Below are the steps I just took to upgrade our ECC environment from UB12 to UB14. There are quite a few steps and EMC has this process documented very well. Every install file and patch has it’s own readme file with detailed installation instructions along with additional info about special considerations depending on your environment. If you’re about to begin this upgrade, you can use these steps as a general guide but I highly recommend reading all of EMC’s documentation as well. There are general notes, warnings, and information in their documentation that did not apply to me but may apply to you. From beginning to end the upgrade process took me about seven hours to complete.

Prerequisites for the repository server:

    • Oracle 10g OCPU Update must be installed
    • Operating system locale settings must be set to English
    • Minimum 13GB of free disk space
    • Disable IOPath (ECC_IOPath.bat –i to check, ECC_IOPath.bat –d to disable)
    • You must back up the repository for ECC and StorageScope
    • ControlCenter Environment Checker must be run
    • All infrastructure services must be stopped
    • The following additional services must be stopped: AntiVirus, COM+, Server management services, vmware tools, Distributed Transaction Coordinator, Terminal Services, and WMI.

1. Download all of the required files. I’m running ECC on two 32 bit Windows servers (one repository server and one application server), so all of the download links I provided below are for windows.

a. Download the UB14 Update Bundle (32 bit is CC_4979.zip, 64 bit is CC_5013.zip):

https://download.emc.com/downloads/DL44408_ControlCenter_Update_Bundle_14_(64-bit)_software_download.zip

https://download.emc.com/downloads/DL44407_ControlCenter_Update_Bundle_14_(32-bit)_software_download.zip

b. Download the latest cumulative Patch (currently CC_5046.zip):

https://download.emc.com/downloads/DL47699_Cumulative_6.1_Update_Bundle_14_Storage_Agent_for_HDS_Patch_5046_Download.zip

c. Download the latest Ionix ControlCenter 6.1 Backup and Restore Utility:

https://download.emc.com/downloads/DL37276_Ionix_ControlCenter_6.1_Backup_and_Restore_Utility_.zip

d. Download the latest Ionix ControlCenter Environment Checker (Currently version 1.0.5.3):

https://download.emc.com/downloads/DL37274_Ionix_ControlCenter_Environment_Checker_1.0.5.3.exe

e. Download the latest Oracle 10g OCPU Patch here (CC_5030.zip):

https://download.emc.com/downloads/DL46584_ControlCenter_6.1_OCPU_(5030)_Download.zip

f. Download the latest Unisphere Host Client/Agent:

https://download.emc.com/downloads/DL30840_Unisphere__Client_(Windows)_1.2.26.1.0094.exe

g. Download the latest Navisphere CLI (Current release is 7.32.25.1.63):

https://download.emc.com/downloads/DL30859_Navisphere_CLI_(Windows_-_all_supported_32_&_64-bit_versions)_7.32.25.1.63.exe

h. I’d recommend reviewing the UB14 Read Me file to identify any specific warnings that may be relevant to your specific installation. The Read Me file for Update Bundle 14 can be downloaded here:

https://support.emc.com/docu44826_Ionix_ControlCenter_6.1_Update_Bundle_14_Read_Me.pdf?language=en_US

2. Log in to the repository server. Run the ECC Backup and Restore Utility. Because our ECC servers are virtualized, I also ran a snapshot from the VSphere console so I could revert back if needed. Below is an overview of the steps I took, review the readme file from EMC for more detail. Note that I shut down our ECC application server for the duration of the upgrade on the repository server.

a. Run the patch to install the files

b. Go to the ECC install directory, HF4938 folder, note timestamp on regutil610.exe

c. Check that the ECC/tools/util/regutil610.exe file has the same timestamp

d. Run ECC/HF498/backup.bat (that sets all ECC services to manual as well)

e. Reboot hosts, verify ECC services are not running

f. Manually back up the ECC folder to another location

3. Run Environment checker on the repository server.

a. Make sure that all checks have passed before proceeding with the upgrade.

4. Check ECC 6.1 compatibility matrix, make sure everything is up to date and compatible

a. Here’s a link to the compatibility matrix: https://support.emc.com/docu31657_Ionix-ControlCenter-6.1-Support-Matrix.pdf?language=en_US

b. Note that UB14 requires Solutions Enabler 7.5, 7.5.1 or 7.6.

c. Host Agent must be at 1.2.x and CLI must be at 7.32.x if you’re running the latest VNX hardware.

5. Run Oracle 10g OCPU Patch (CC_5030) on the repository server if required. This step took quite a bit of time to complete. I’d expect 60-90 minutes for this step.

a. Make sure all ECC application services are stopped.

b. Run %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_hotback.bat

c. Run %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ram_export_db.bat

d. Run %emcstsoh%\admin\emcsts_scripts\emcsts_coldback.bat

e. Run %emcstsoh%\admin\emcsts_scripts\emcsts_export_db.bat

f. Copy the contents of the %ECC_INSTALL_ROOT% to a backup location of your choice

g. Extract the SQL_PATCH_5030.zip to a local drive on the ECC Server.

h. Verify repository by running the following: %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_recomp_invalid.bat

i. Verify the storage scope repository by running the following: %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\emcsts_recomp_invalid.bat

j. If the prior two scripts ran successfully, proceed, if not call EMC.

k. Run the prereboot.bat script from the SQL_PATCH_5030 directory.

l. Reboot

m. Run the postreboot.bat script from the SQL_PATCH_5030 directory.

n. At the end of the postreboot script, it will ask if you want to revert all the services back to their original state. Respond with Y.

o. Verify successful installation by running %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\emcsts_hotfixstatus.bat

6. Update the Repository server with the latest Host agent and Navisphere CLI:

a. Unisphere Host Agent 1.2.25.1.0163: https://download.emc.com/downloads/DL30839_Unisphere_Host_Agent_(Windows_Â -_all_supported_32_&_64-bit_versions)__1.2.25.1.0163.exe

b. Navisphere CLI 7.32.25.1.63: https://download.emc.com/downloads/DL30859_Navisphere_CLI_(Windows_-_all_supported_32_&_64-bit_versions)_7.32.25.1.63.exe

c. A reboot is not required after installing the Agent and CLI. Make sure to preserve your security settings when asked in the installation.

7. Install a newer version of Solutions Enabler, if required. As mentioned earlier, UB14 requires version 7.5, 7.5.1 or 7.6.

a. Link to the MRLK Control Center supported version of SE (v7.5.0 – CC_5014): https://download.emc.com/downloads/DL44373_Solutions_Enabler_7.5_MRLK.zip

b. There is a readme pdf file included in the download with details on installation.

c. Stop the EMC storapid service.

d. Stop the EMC Control Center Server and Store Services.

e. Do NOT stop the OracleECCREP_HOMETNSListener or OracleServiceRAMBDB services.

f. Run the install.

g. Restart the services.

8. Prepare for UB14 Installation. Now all the prerequisite installs are complete on my system. The next step is to verify that the correct services are stopped and begin the UB14 upgrade. We have everything running on one repository server, your environment could be different.

a. On the repository server, stop the following services, set them to Manual, and reboot:

i. EMC ControlCenter API Server

ii. EMC ControlCenter Key Management Server

iii. EMC ControlCenter Master Agent

iv. EMC ControlCenter Repository

v. EMC ControlCenter Server

vi. EMC ControlCenter Store

vii. EMC ControlCenter STS MSA Web Server

viii. EMC ControlCenter Web Server

ix. EMC StorageScope Repository

x. EMC StorageScope Server

b. The following services must be stopped: AntiVirus, COM+, Server management services, vmware tools, Distributed Transaction Coordinator, and WMI.

c. Verify the following services are running before beginning the UB14 installation:

i. OracleECCREP_HOMETNSListener

ii. OracleServiceRAMBDB

iii. OracleServiceEMCSTSDB

9. Begin the UB14 Installation (CC_4079 for 32bit, CC_5013 for 64bit)

a. Execute the Patch61014383_4979_x86.exe patch (or the 64 bit patch if you’re on a 64bit server).

b. I recommend reading the UB14 readme file. If you encounter any errors, a few common install issues are listed.

c. This step takes a very long time, count on at least 3-4 hours. I also have StorageScope which added additional time.

d. After the patch install completes, change all the services back to their original ‘automatic’ state.

e. Reboot the repository server

10. Install the latest cumulative Update Bundle 14 patch next (The most current right now is CC_5046).

a. The current version of the UB14 patch can be downloaded here:

https://download.emc.com/downloads/DL47699_Cumulative_6.1_Update_Bundle_14_Storage_Agent_for_HDS_Patch_5046_Download.zip

b. Stop the EMC Control Center Store and Server Services

c. Stop the following Services:

i. EMC ControlCenter API Server

ii. EMC ControlCenter Key Management Server

iii. EMC ControlCenter Master Agent

iv. EMC ControlCenter Repository

v. EMC ControlCenter Server

vi. EMC ControlCenter Store

vii. EMC ControlCenter STS MSA Web Server

viii. EMC ControlCenter Web Server

ix. EMC StorageScope Repository

x. EMC StorageScope Server

d. Run Patch 61014615_5046_hds.exe

e. Restart all Services that were stopped earlier in steps B and C.

f. Apply patch to agents:

i. Start the ECC Console

ii. Right click on the repository server object

iii. Select Agents à Apply Patch

iv. If the task fails, restart the master agent from the console

11. Update The application server

a. Install the latest Unisphere Host Agent 1.2.25.1.0163: https://download.emc.com/downloads/DL30839_Unisphere_Host_Agent_(Windows_Â -_all_supported_32_&_64-bit_versions)__1.2.25.1.0163.exe

b. Install the latest Navisphere CLI 7.32.25.1.63: https://download.emc.com/downloads/DL30859_Navisphere_CLI_(Windows_-_all_supported_32_&_64-bit_versions)_7.32.25.1.63.exe

c. Update the ECC Console by navigating to https://<repository_server>:30002/webinstall

i. Click Installation

ii. Click Console Patch 6.1.0.14.383

d. Install Solutions Enabler on the application server (not the MRLK version)

i. Windows 32 bit download

https://download.emc.com/downloads/DL43505_se7500-WINDOWS-x86.exe.exe

ii. Windows 64 bit download:

https://download.emc.com/downloads/DL43504_se7500-WINDOWS-x64.exe.exe

iii. Stop existing ECC and Solutions Enabler services

iv. Launch Install

e. Apply Agent patches from within ECC Console

i. Start the ECC Console

ii. Right click on the application server object

iii. Select Agents à Apply Patch

iv. If the task fails, restart the master agent from the console

12. Verify that WLA Archive collection (performance data) is collecting properly.

a. Go to your WLAArchives folder

b. Go to the Clariion\<serial number>\interval subfolder

c. Sort the files by date. If new files are being written then it is working.

13. Verify that all agents are running and are at the correct patch level

a. Launch the Control Center console

b. Click on the small gear icon on the lower right side of the window, which will launch the agents view on the right side of the screen

c. Verify that all agents are running and are patched to 6.1.0.14.383. Any that require updates can be updated from this screen, right click, choose agents and install patch.

14. Remove backup directories (optional). The List is on page 16 & 17 of the UB14 readme file.

That’s it! You’re done.

Note: 24 hours after the completion of the upgrade I noticed that WLA archive data collection wasn’t working for some of the arrays. I deleted the arrays that were’nt working and rediscovered them, which resolved the problem. Deleting the arrays removes all historical data from StorageScope.

Advertisements

Adding/Removing modules from a datamover

I recently had an issue where a brand new datamover installed by EMC would not allow me to make it a standby for the existing datamovers.  It turns out that the hardware (specifically the number of FC and ethernet interfaces) must match PRECISELY, the number of ports and the slots the modules are installed in have to match across all datamovers.

The new datamover that was installed had an extra 4 port ethernet module installed in it.  Below is the procedure I used to remove the module, including all the commands to take it down, reconfigure it, and bring it back up successfully.  Removing the extra module solved the problem, it matched the config of the others and allowed me to configure it as a standby.

First, log in to the CLI on the control station with root priviliges.  Next, just run the commands below in order.

Turn off connecthome and emails to avoid false alarms.
 /nas/sbin/nas_connecthome -service stop
 /nas/bin/nas_emailuser -modify -enabled no
 /nas/bin/nas_emailuser -info

Copy and paste this to save, it will list the current datamover config.
 nas_server -i -a

Run this to shut the datamover down.  Run getreason to verify when it’s down.
 server_cpu server_<x> -halt now
 /nasmcd/sbin/getreason

Remove/replace the module now.

Power the datamover back on.
 /nasmcd/sbin/t2reset pwron -s <slot number>

Watch getreason for status
 /nasmcd/sbin/getreason
(Wait for it to reboot and say ‘Hardware Misconfigured’)

Once it is in a ‘misconfigured’ state, run setup_slot to configure it:
 /nasmcd/sbin/setup_slot -i 4

Run this command to view the current hardware config, verify that your change was made:
 server_sysconfig server_4 -p

Restart connecthome and email services.
 /nas/sbin/nas_connecthome -service start -clear
 /nas/sbin/nas_connecthome -i
 /nas/bin/nas_emailuser -modify -enabled yes
 /nas/bin/nas_emailuser -info

That’s it!  your datamover has been updated and reconfigured.

A guide for troubleshooting CIFS issues on the Celerra

In my experience, every CIFS issue you may have will fall into 8 basic areas, the first five being the most common.   Check all of these things and I can almost guarantee you will resolve your problem. 🙂

1. CIFS Service.  Check and make sure the CIFS Service is running:  server_cifs server_2 -protocol CIFS -option start

2. DNS.  Check and make sure that your DNS server entries on the Celerra are correct, that you’re configured to point to at least two, and that they are up and running with the DNS Service running.

3. NTP.  Make sure your NTP server entry is correct on the Celerra, and that the IP is reachable on the network and is actively providing NTP services.

4. User Mapping.

5. Default Gateway.  Double check your default gateway in the Celerra’s routing table.  Get the network team involved if you’re not sure.

6. Interfaces.  Make sure the interfaces are physically connected and properly configured.

7. Speed/Duplex.  Make sure the speed and duplex settings on the Celerra match those of the switch port that the interfaces are plugged in to.

8. VLAN.  Double check your VLAN settings on the interfaces, make sure it matches what is configured on the connected switch.