White Paper

EMC SOLUTIONS GROUP

Abstract
This white paper describes how a 12 TB Oracle data warehouse was transported
from a data center in Massachusetts, U.S, to Cork, Ireland. The infrastructure
was re-created at the Cork site on EMC

Symmetrix

VMAX storage and the

database was restored and recovered. EMC Data Domain

Encryption was used

to secure the data during transport.
July 2011
REMOTE SITE RECOVERY OF ORACLE
ENTERPRISE DATA WAREHOUSE USING
EMC DATA DOMAIN
A Detailed Review

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
2



Copyright 2011 EMC Corporation. All Rights Reserved.
EMC believes the information in this publication is accurate as of its
publication date. The information is subject to change without notice.
The information in this publication is provided as is. EMC Corporation makes
no representations or warranties of any kind with respect to the information in
this publication, and specifically disclaims implied warranties of
merchantability or fitness for a particular purpose.
Use, copying, and distribution of any EMC software described in this
publication requires an applicable software license.
For the most up-to-date listing of EMC product names, see EMC Corporation
Trademarks on EMC.com.
All trademarks used herein are the property of their respective owners.
Part Number H8238.1

3 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Table of contents
Executive summary ............................................................................................................... 5
Business case .................................................................................................................................. 5
Solution overview ............................................................................................................................ 5
Key results ....................................................................................................................................... 6
Introduction .......................................................................................................................... 7
Purpose ........................................................................................................................................... 7
Scope .............................................................................................................................................. 7
Audience.......................................................................................................................................... 7
Terminology ..................................................................................................................................... 7
Technology overview ............................................................................................................. 8
Overview .......................................................................................................................................... 8
EMC Data Domain DD660 ................................................................................................................. 8
EMC Data Domain Encryption ........................................................................................................... 9
EMC Data Domain file system encryption lock .................................................................................. 9
EMC Symmetrix VMAX ...................................................................................................................... 9
EMC Solutions Enabler ..................................................................................................................... 9
EMC Symmetrix TimeFinder .............................................................................................................. 9
EMC PowerPath .............................................................................................................................. 10
Oracle Database ............................................................................................................................ 10
Solution overview ............................................................................................................... 11
Use case requirements ................................................................................................................... 11
The challenge................................................................................................................................. 11
Nondisruptive to production ...................................................................................................... 11
Distance and latency ................................................................................................................. 11
Database size ............................................................................................................................ 12
The solution ................................................................................................................................... 12
Project timeline .............................................................................................................................. 14
Discovery primary site ...................................................................................................... 15
Overview ........................................................................................................................................ 15
Security .......................................................................................................................................... 15
Business and technical .................................................................................................................. 16
Primary data center configuration information................................................................................ 16
Remote site infrastructure .............................................................................................................. 17
Architecture ............................................................................................................................... 17
Hardware resources ................................................................................................................... 18
Software resources .................................................................................................................... 18

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
4
Backup appliance ...................................................................................................................... 18
Oracle database backup primary site ................................................................................ 19
Overview ........................................................................................................................................ 19
DD660 setup .................................................................................................................................. 19
Network settings ....................................................................................................................... 19
Encryption ................................................................................................................................. 20
RMAN backup ................................................................................................................................ 20
File system lock .............................................................................................................................. 20
Data compression results ............................................................................................................... 21
Oracle server deployment and storage provisioning remote site ........................................ 23
Overview ........................................................................................................................................ 23
Deploying the remote site server .................................................................................................... 23
Configuring the remote site storage ............................................................................................... 23
Primary data warehouse configuration information .................................................................... 23
Provisioning storage at the remote site ...................................................................................... 24
Setting up the Oracle database environment.................................................................................. 25
Oracle database recovery remote site ............................................................................... 27
Overview ........................................................................................................................................ 27
DD660 setup .................................................................................................................................. 27
Network settings ....................................................................................................................... 27
Unlock file system ..................................................................................................................... 28
RMAN restore ................................................................................................................................. 29
Verifying the integrity of the database ............................................................................................ 32
Data Domain consolidation remote site ............................................................................ 34
Overview ........................................................................................................................................ 34
Replication process ........................................................................................................................ 34
Conclusion ......................................................................................................................... 37
Summary ....................................................................................................................................... 37
Findings ......................................................................................................................................... 37
References .......................................................................................................................... 38
White papers ................................................................................................................................. 38
Product documentation .................................................................................................................. 38
Other documentation ..................................................................................................................... 38

5 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Executive summary

EMC offers an extensive series of solutions for quick, efficient, and nondisruptive
backup of large Oracle data warehouse environments. Increasingly, customers are
looking to repurpose or reuse these backups to facilitate operations such as:
Migration of Oracle data warehouses to new data center locations
Seeding of Oracle data warehouse environments at remote disaster recovery
locations
Deployment of remote database copies for test and development purposes
These scenarios present challenges in relation to determining the most appropriate
mechanism for transferring the data from the primary site to the remote site, and
developing validated procedures for re-creating the production environment and
recovering the database at the remote site.
The main considerations in relation to data transfer to the remote site include:
Security and corporate governance compliance
Distance and latency
Database size
Cost effectiveness
For one-off migration scenarios, where the data warehouse is many terabytes in size
and the distance between data centers is great, it is impractical to transfer the data
across the WAN. Consequently, customers are looking for an offline transport solution
that is economical and secure.

At its headquarters in Hopkinton, Massachusetts, EMC IT maintains a 12 TB Oracle
data warehouse, deployed on an Oracle Real Application Clusters (RAC) 11g
environment. To demonstrate the benefits and implementation of the solution
documented in this white paper, this real-world production database was cloned and
backed up, shipped by air from the primary data center in Hopkinton, Massachusetts
to the EMC Solutions Center in Cork, Ireland, and then restored and recovered for test
and development use.
The remote data center infrastructure consisted of a two-node Oracle RAC 11g cluster
that accessed the Oracle database on an EMC

Symmetrix

VMAX storage system.

This reflects the infrastructure at the primary data center.
The production database in Hopkinton was cloned using EMC TimeFinder

and then
backed up to an EMC Data Domain

DD660 appliance using Oracle Recovery Manager

(RMAN). Data Domain Encryption and file system encryption locking were used to
ensure security and corporate governance compliance during transport.
At the remote site, the DD660 was added to the environment and the file system was
unlocked. The database was then restored and recovered to a test system using
Oracle RMAN.
Business case
Solution overview

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
6
After verifying that the database had been successfully restored, Data Domain
Replicator was used to move the database image from the DD660 to a shared Data
Domain DD880 appliance in the data center in Cork, so that the DD660 could be
repurposed for another project.

The key benefits demonstrated by the solution include:
Nondisruptive backup
Creating a TimeFinder clone has no impact on production. Mounting the clone
to a separate backup server provides a fully-usable, independent, point-in-time
copy that can be backed up to the Data Domain appliance with no disruption to
the production environment.
Reduced backup footprint and shipping costs
Data Domain deduplication reduces the footprint of the backup database
significantly. In this solution, a 70 percent reduction in capacity requirements
enabled use of the midrange DD660 appliance, which minimized shipping
costs.
Seamless Oracle RMAN integration
Data Domain systems integrate seamlessly into Oracle architectures,
supporting existing backup strategies, including Oracle RMAN.
Data security and corporate governance compliance
Data Domain Encryption ensures that all data is physically stored in an
encrypted manner and cannot be accessed without first decrypting it. Data
Domain file system encryption lock adds a further level of protection that
enables shipping on commercial air transport while satisfying corporate
governance compliance requirements.
Validated and documented end-to-end procedures
Database integrity testing at the end of the project demonstrated successful
recovery of the database at the remote data center.
This white paper documents the procedures for achieving these results, from
the initial discovery phase necessary for designing the infrastructure for the
remote site, to a complete restore and recovery of a working data warehouse.
These procedures are applicable to any solution involving recovery of an Oracle
database.

Key results

7 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Introduction

This white paper documents a case study that involved cloning and backing up a real-
world 12 TB database, shipping it by air from the primary data center in Hopkinton,
Massachusetts to the EMC Solutions Center in Cork, Ireland, and then restoring and
recovering the database for test and development use.

The white paper covers the following:
An overview of the technologies involved in the solution
The discovery process carried out at the production site to develop a build plan
for the remote site
Re-creation of the data warehouse at the remote site
Backup and securing of the DD660 for transfer by commercial air transport
Restore and recovery of the database at the remote site
Validation of the recovered database
Replication of the database from the DD660 to a DD880 appliance

This white paper is intended for Oracle, storage, server, and backup administrators
who want to understand how to migrate an Oracle database to a remote site and the
build requirements at the remote site to enable recovery of the database. It is
assumed that the reader is familiar with the following EMC and Oracle products:
EMC Symmetrix VMAX
EMC Data Domain
Oracle Database 11g R1 Enterprise Edition with Oracle Clusterware

Table 1 defines terms used in this white paper.
Table 1. Terminology
Term Definition
AES Advanced Encryption Standard
ASM Automatic Storage Management
DD Data Domain
DD OS Data Domain operating system
RAC Real Application Clusters
RMAN Recovery Manager
SISL Stream-Informed Segment Layout architecture
Purpose
Scope
Audience
Terminology

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
8
Technology overview

The case study uses the following hardware and software components:
EMC Data Domain DD660
EMC Data Domain Encryption
EMC Data Domain file system encryption lock
EMC Symmetrix VMAX
EMC Symmetrix TimeFinder
EMC PowerPath

Oracle Database 11g R1 Enterprise Edition with Oracle Clusterware
Oracle RMAN

For the solution, an EMC Data Domain DD660 appliance was used to back up the
source database and transport it to the recovery site in Cork.
Data Domain deduplication storage systems dramatically reduce the amount of disk
storage needed to retain and protect enterprise data, requiring a storage footprint
that is 10 to 30 times smaller, on average, than that required by the original data set.
The DD660 is a midrange appliance that offers up to 2 TB/hour of aggregate inline
deduplication throughput, and up to 700 GB/hour for a single stream. The model
used in the case study provides 8 TB of storage capacity, with much higher data
storage capacity enabled by deduplication.
Like all Data Domain systems, the DD660 derives its performance advantages from
the Data Domain Stream-Informed Segment Layout (SISL) scaling architecture. SISL
identifies 99 percent of duplicate segments in RAM, inline, ensuring that only unique
data is stored to disk. The storage of unique data is invisible to backup software,
which sees the entire virtual file system. The integrity and recoverability of backup
data are secured by the Data Domain Operating System (DD OS) and Data
Invulnerability Architecture, which ensures that data is stored and recoverable with
continuous write verification, fault-detection, and self-healing mechanisms.
When data enters the Data Domain system, using any of the supported protocols
(NFS, CIFS, VTL, OST, and Distributed Segmented Processing), the stream is
segmented, fingerprinted, deduplicated (Global Compression), grouped into multi-
segment compression regions, and then locally compressed. When confidentiality is
required, deduplicated and compressed data can be encrypted inline and the file
system can then be locked.
Data Domain systems integrate seamlessly with Oracle RMAN, with no changes to
scripts, backup processes, or system architecture.


Overview
EMC Data Domain
DD660

9 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review


Data Domain Encryption protects data if the Data Domain system is stolen or lost
during transit, and eliminates accidental exposure during replacement of failed
drives. DD Encryption is designed such that if an intruder circumvents other network
security controls and gains access to encrypted data, the data is unreadable and
unusable to that person without the proper cryptographic keys.
The inline encryption process integrates seamlessly with the inline deduplication
process, encrypting data before it is written to disk. Similar to the advantages of
inline deduplication, inline encryption requires minimal resources to provide fast,
reliable, and secure backup and recovery.

File system encryption lock is specifically designed to lock an encrypted Data Domain
appliance and its external storage prior to transporting the appliance between
locations.
File system encryption lock requires an encryption passphrase. This is a human-
readable key that is used to generate a machine-usable AES256 encryption key that
encrypts the Data Domain system encryption key. Locking the file system involves
changing the passphrase. This changes the encryption of the Data Domain system
key and removes the passphrase from the system. Locking the system allows a Data
Domain system to be transported with the encryption keys on the system. The keys
are not recoverable until the file system is unlocked using the passphrase (which is
not stored on the appliance).

An EMC Symmetrix VMAX provides the storage platform at both the primary and
remote data centers.
Symmetrix VMAX storage arrays provide high performance and scalability for
demanding enterprise storage environments. Built on the strategy of simple,
intelligent, modular storage, the Symmetrix VMAX incorporates a highly scalable
Virtual Matrix Architecture that enables it to grow seamlessly and cost-effectively
from an entry-level configuration into the worlds largest storage system. The VMAX
supports Flash drives, Fibre Channel drives, and SATA drives within a single array, as
well as an extensive range of RAID types.

EMC Solutions Enabler is the software that provides a host with the Symmetrix
Command Line Interface (SYMCLI), which is a comprehensive command set for
managing your Symmetrix storage environment. SYMCLI is used in single command
line entries and scripts to monitor device configuration and status, and perform
control operations on devices and data objects within a managed storage system.

EMC Symmetrix TimeFinder software provides powerful replication capabilities for
creating local point-in-time copies that are independent of the hosts and operating
systems, applications, and databases. These copies are immediately available for
read and write access.
For the solution, a TimeFinder copy of the Oracle database was used to offload
backup overhead from the production environment to another server.
EMC Data Domain
Encryption
EMC Data Domain
file system
encryption lock
EMC Symmetrix
VMAX
EMC Solutions
Enabler
EMC Symmetrix
TimeFinder

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
10

EMC PowerPath

is server-resident software that enhances performance and

application availability by supporting multiple I/O paths to logical devices and
intelligently distributing I/O requests across all available paths. PowerPath also
provides automatic failover in the event of a hardware failure by automatically
detecting the path failure and redirecting I/O to another path.
For the solution, PowerPath provides intelligent path management and automatic
path failover at the remote site.

The solution involves an Oracle Database 11g R1 data warehouse and uses the
features of Oracle ASMLib, Oracle Database 11g, and Oracle Clusterware 11g such as
RAC, Automatic Storage Management (ASM), and RMAN.

EMC PowerPath
Oracle Database

11 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Solution overview

This solution is based on a real-world use case and is essentially a case study. The
requirements of the use case were to take a production database and transfer it to a
lab environment for test and development purposes.
The EMC IT Production DB resides in EMC IT corporate headquarters in Hopkinton,
Massachusetts, as shown in Figure 1. The test and development lab is located in the
EMC Solutions Center in Cork, Ireland. The database is approximately 12 TB.

Figure 1. The challenge
An economical and secure mechanism was required to transport the data, and the
process also had to have zero impact to the EMC IT production system.

Nondisruptive to production
As the project involved a real-world, production system, it was critical that the backup
procedure did not impact database operations. This factor dictated a nondisruptive
backup solution.
Distance and latency
The distance between the data centers in Hopkinton and Cork is approximately 4,740
kilometers (2,946 miles). The minimum latency for the round-trip is 47.4 ms, not
including other delays introduced by encryption or the network itself. In fact, the
average measured ping time between the data centers is 90 ms.
For example, a dedicated WAN link, with 47.4 ms round-trip latency, 1 Gb/s
bandwidth, and a TCP window size of 65,536 bytes, would restrict the maximum
throughput to 1.3 MB/s and take approximately 112 days to move 12 TB of data.
As this was a one-off project, it was not envisaged that sufficient dedicated
bandwidth would be made available on the corporate WAN for the data migration.
These factors dictated an offline method for transporting the data.
Use case
requirements
The challenge

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
12
Database size
The size of the databaseapproximately 12 TB was a significant challenge and
another factor that made it impractical to transport the data across the WAN. Figure 2
indicates just how big 12 TB is in real terms.

Figure 2. What 12 TB looks like

To facilitate transporting the data in an economical and secure fashion, a Data
Domain DD660 appliance was chosen. The reasons for this are:
Small form factor
The DD660 used has a 2U form factor and weighs 30 kg (66 lb)perfectly
suited for low shipping costs.
Data compression
The DD660 has 8 TB of data storage space. However, Data Domain systems
store only unique data, using Global Compression to pool redundant data from
each backup image. This enables much higher data storage rates than the raw
capacity of the device.
Figure 3 shows the difference in size between the original database, the database
backed up by RMAN, and the deduplicated, encrypted database on the DD660.

Figure 3. Original and deduplicated database sizes
The solution

13 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Data integrity
The Data Domain Data Invulnerability Architecture protects against data loss
from hardware and software failures.
Encryption of data at rest
Data Domain Encryption protects data if the Data Domain system is stolen or
lost during transit.
Encrypted file system lock
Data Domain file system encryption lock protects an encryption-enabled Data
Domain system (and its external storage devices) while in transit. The encryption
keys are not recoverable without a passphrase, which is not stored with the
system.
To ensure that the production environment was not impacted by the project, it was
decided to create a TimeFinder clone of the database and mount this to a backup
server. This would provide a fully-usable, independent, point-in-time copy of the
production database that could be backed up to the DD660 nondisruptively.
Figure 4 illustrates the solution designed for the project and outlines the procedures
involved.

Figure 4. The solution

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
14

The initial planning and feasibility phase of this project took place over several weeks
prior to the official kickoff of the project. After the official kickoff, the entire project
took just 22 working days to complete, from commencing discovery to verification of
the restored database at the remote site. Figure 5 shows the project schedule.
Predictably, discovery was the lengthiest phase (2 weeks) and was iterative in nature.
It progressed in parallel with, and was informed by, the build process at the remote
site. The comprehensive information gathering and planning during this phase was
key to ensuring successful completion of the project. The result was a complete and
successful restore and recovery of the database at the remote site in only 28 hours.

Figure 5. Project timeline

Project timeline

15 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Discovery primary site

This section describes the discovery process carried out at the start of the project to
determine the feasibility of the project and to gather infrastructure and configuration
information from the production site. This information was used to determine the
transport mechanism between the two data centers and to create a build plan for the
remote site. This phase of the project involved the steps shown in Figure 6.

Figure 6. Discovery process
The stakeholders involved in the project were:
Primary site
Stakeholders: EMC IT; EMC IT Security and Compliance
Location: EMC IT headquarters, Hopkinton, Massachusetts, U.S.
Database: EMC IT Production DB
Remote site
Stakeholders: EMC Global Solutions (Oracle Engineering)
Location: EMC Solutions Center, Cork, Ireland (CSC)
Database: CSC test and development database

The security of the data to be transferred between the data centers was of critical
importance to the feasibility of the project. EMC IT and EMC IT Security considered
data security issues in terms of:
Corporate governance and compliancethat is, ensuring data security both in
transit and at the remote data center so as to meet legislative requirements and
EMC and industry standards.
Identifying any potential personal/sensitive data for masking/cleaning.
Choosing a backup/copy mechanism that would be nondisruptive to the
production database.

Overview
Security

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
16
This feasibility phase concluded that a copy of the EMC IT Production DB would be
supplied to EMC Global Solutions under the following conditions:
The data would be stored in a secure EMC data center at the remote site.
Access to the data would be limited to specified EMC personnel.
No data would be made public or available to external parties.
A Data Domain DD660 would be used to transport the data, with Data Domain
Encryption and Data Domain file system encryption lock enabled.

To enable recovery and repurposing of the data warehouse, the infrastructure at the
primary site would be re-created at the remote site. In preparation for this, the
following business and technical issues were considered:
Mapping and configuration of the underlying infrastructureservers, storage,
and IP and storage networks
Identifying hardware and software versions for the:
Server operating system
Oracle database
Symmetrix VMAX storage system
Data Domain system
Identifying the groups and individuals within EMC IT who could supply
information and carry out the necessary tasks in the production environment
A project plan was then agreed to that would minimize the impact on EMC IT staff and
would move the data to Cork with minimal delay.

To facilitate the project, EMC IT made several important documents available to the
EMC Global Solutions team in Cork. These documents detailed the design and
architecture of the EMC IT data warehouse.
The Detailed Solutions Design document for the data warehouse was key to
understanding the production environment. This document contains the following
information:
High-level architecture
Hardware inventory and configuration information for the database and backup
servers and cluster, including CPU and memory, internal drives, and HBA and
network cards and ports
List of required Shared Infrastructure Services, including: SAN, storage array,
and IP networksprivate (Cluster Interconnect), data center, and backup
Storage rationale and design, hardware and software, and configuration
Securityuser and group access and authentication
Data center hardware provisioningenvironment and cabling details
Business and
technical
Primary data center
configuration
information

17 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

In addition, the following live information was supplied:
A copy of the current symapi_db.bin file from the production Symmetrix array
An extract of the Oracle data dictionary views including V$ASM_DISKGROUP
and V$ASM_DISK
A listing of the Oracle ASM disks
A mapping of the PowerPath devices and paths to the Symmetrix LUNs
Multiple sample Automatic Workload Repository (AWR) reports
A listing of the latest operating system and Oracle Clusterware and database
patch levels
Discovery was an iterative process, and continued in parallel with, and in response
to, the remote site build.

Architecture
From the information gathered during the discovery phase, the infrastructure for the
remote site was designed as shown in Figure 7.

Figure 7. Remote site architecture
Remote site
infrastructure

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
18
Hardware resources
Table 2 details the hardware environment for the remote site.
Table 2. Hardware resources
Purpose Quantity Resources
Storage array

1 Symmetrix VMAX with:
2 x VMAX 64 GB engines
8 Gb FC connectivity
450 GB 15k FC drives
Oracle RAC database servers 2 Linux server with:
Quad-core CPU
96 GB RAM
FC switches 2 8 GB/s FC switches
Ethernet switches 2 1 Gigabit Ethernet switches

Software resources
Table 3 details the software resources for the remote site.
Table 3. Software resources
Software Purpose
Red Hat Enterprise Linux 5.5 Server OS for database and application servers
Oracle Database 11g R1 (11.1.0.7) Database and cluster software
EMC Enginuity 5875 Symmetrix VMAX operating environment

Backup appliance
Table 4 details the configuration of the Data Domain appliance used for backing up
and transporting the database.
Table 4. Data Domain appliance configuration
Appliance Details
EMC Data Domain DD600 12 x 1 TB SATA drives
1 Gb IP connectivity
DD OS 4.9


19 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Oracle database backup primary site

The process for configuring the DD660 and backing up the production database at
the primary data center involved the steps shown in Figure 8. When the process was
complete, the appliance was secure and ready to ship to Corkthat is, the data was
encrypted and the file system locked.
Local site
backup
Create clone copy
of production
database
Backup
complete
Configure DD660
network settings &
add to network
Enable DD660 file
system encryption
RMAN backup to
DD660
Lock DD660 file
system
Test database
restore
DD660 secure & ready to ship
Data encrypted
File system locked

Figure 8. Oracle database backup process

Network settings
The DD660 used for this solution had two 1 Gb network interfaces. To improve
network performance and resiliency, these were aggregated and used in parallel.
Figure 9 shows the link aggregation setup.

Figure 9. DD660 link aggregation setup
To enable backup of the database, the appliance was then provided with an IP
address in the local network.

Overview
DD660 setup

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
20
Encryption
Data Domain Encryption was enabled prior to enabling the file system on the DD660.
Data was encrypted using the 128-bit Advanced Encryption Standard (AES) algorithm
(aes_128_cbc), as shown in Figure 10. This option strikes a reasonable balance
between security and performance for the use case.

Figure 10. DD660 encryption

On the production site, a fresh clone copy of the storage was created using
TimeFinder. An Oracle RMAN backup of this copy was then taken, with the database
mounted, but not opened, on a backup server. A recovery catalog was used to record
RMAN metadata.
With Data Domain Encryption enabled, the data being backed up to the DD660 was
deduplicated and then encrypted inline, before it was written to disk.
The following were backed up to the DD660:
A full RMAN backup of the Oracle database
The database control file
A schema export of the RMAN recovery catalog
The Oracle team on the production site then validated the backup by restoring the
control file and database to the backup server. Table 5 shows the duration of the
backup and restore operations.
Table 5. Backup and restore times
Process Timings
Database backup to DD660 28 hours and 38 minutes
Database restore (for verification) 30 hours and 2 minutes

The filesys encryption lock command was used to lock the Data Domain file system.
This required a passphrase change, as shown in Figure 11, automatically re-encrypts
the system encryption key, and removes the new passphrase from the DD system.
The new passphrase was communicated separately to the remote site.
RMAN backup
File system lock

21 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review


Figure 11. Data Domain file system encryption lock

Data Domain deduplication eliminated redundant data segments during the backup
process. This allowed the Data Domain appliance to store and manage much more
data than would be possible with a traditional storage device.
Figure 12 shows a DD OS view that summarizes data storage and compression
statistics for the use case and illustrates the value of Data Domain deduplication
when transporting an Oracle database. Table 6 also summarizes these statistics.

Figure 12. Use case compression summary
The total physical space available on the appliance is 7.5 TB, the total uncompressed
data written by RMAN to the appliance was 10.6 TB, and the space required to store
the compressed data was 3.2 TB. This demonstrates a 69.7 percent reduction in the
space required to store the database.
Table 6. Use case compression summary
Item Value
Total physical space available 7,528.6 GiB (7.5 TB)
Total data written to appliance (uncompressed) 10,613.6 GiB (10.6 TB)
Data stored on appliance (compressed) 3,217.0 GiB (3.2 TB)
Deduplication factor 3.3
Percent reductionstorage space saving 69.7%

Data compression
results

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
22
Figure 13 shows the allocated, used, and free space information for the source
database. This closely matches the statistics gathered by the DD660 and shown in
Figure 12.

Figure 13. SQL output showing database allocated and free space information
The database had 11.6 TB of storage assigned to its data files, not including temp
files, redo logs, control files and archive logs. Of this, 10.13 TB was allocated to
tables and indexes within the data files, with 1,504 GB of free space available.

23 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Oracle server deployment and storage provisioning remote site

This section describes preparation of the hardware and software infrastructure at the
Solutions Center lab in Cork, as outlined in Figure 14.

Figure 14. Remote site preparation process

To simplify the restore process, the Oracle Solutions Engineering team decided to
restore the database as a single instance before migrating it to Oracle RAC.
The database server was configured with Red Hat Enterprise Linux 5.5 and prepared
for a standard Oracle ASM/Database install. Server configuration involved the
following steps:
1. Install and configure EMC PowerPath.
2. Install and configure ASMLIB.

Primary data warehouse configuration information
During the discovery phase of the project, EMC IT provided the Oracle Solutions
Engineering team in Cork with the configuration database (symapi_db.bin) for the IT
data warehouse storage array.
At the remote site, the configuration information for the storage array was extracted
by querying symapi_db.bin in offline mode using standard Solutions Enabler display
commands. The procedure involved the following steps:
1. Copy symapi_db.bin to a host with Solutions Enabler installed.

Overview
Deploying the
remote site server
Configuring the
remote site storage

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
24
2. Use the following Solutions Enabler commands to set up Solutions Enabler to
query the database file:
export SYMCLI_OFFLINE=1
export SYMCLI_DB_FILE=/DWproject/symapi_db.bin
export SYMCLI_SID=XXX
3. Use the following command to extract the configuration details for all devices
in the database storage group (GDWORAPRD), as shown in Figure 15:
symsg show SG_GDWORAPRD

Figure 15. Extracting device configuration information from a storage group
With this information it was possible to understand how the production storage was
provisioned. In conjunction with the other information sources supplied by EMC IT
during the discovery phase, the Cork team was able to determine how much storage
was required on the target Symmetrix VMAX, the Oracle ASM disk groups required,
and the number of LUNs in each disk group.
Provisioning storage at the remote site
Provisioning storage at the remote site involved the following steps:
1. Create the devices on the target Symmetrix VMAX array, as listed in Table 7.
Table 7. Storage assigned to Symmetrix VMAX array at remote site
ASM disk group Number of devices Device size (MB)
GDWP_TEMP 32 32,263
GDWP_ARCH 16 64,528
GDWP_REDO01 16 8,066
GDWP_REDO02 16 8,066
GDWP_T0_DATA01 16 129,056
GDWP_T1_DATA01 80 129,056
GDWP_T5_DATA01 40 129,056
Total Devices 216

2. Configure each device with a device identifier label that corresponds to the
name of the ASM disk group to which the device will belong. This label can be
read from the host using the Inquiry utility, which makes device identification
from the host much easier.




25 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Figure 16 shows these labels being created with Solutions Enabler.

Figure 16. Setting device identifiers with Solutions Enabler
3. Create a masking view for the devices so that they are visible to the RAC
cluster.
At this point, the storage is ready to be incorporated into the Oracle configuration at
the remote site.

Setting up the Oracle database environment at the remote site involved the following
steps:
1. Partition and assign the PowerPath devices for use with Oracle ASM.
Figure 17 shows the mapping of the Oracle ASM disks to the Symmetrix
assigned device_name.

Figure 17. Oracle ASM disk mapping to Symmetrix devices
2. Install Oracle Clusterware and the database binaries.

Setting up the
Oracle database
environment

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
26
3. Configure the ASM instance and create the ASM disk groups listed in Table 8.
Table 8. ASM disk groups and sizing
Disk group
Allocation unit
(MB)
Size
(GB)
GDWP_ARCH 1 1,008
GDWP_REDO01 1 126
GDWP_REDO02 1 126
GDWP_T0_DATA01 16 2,016
GDWP_T1_DATA01 16 10,081
GDWP_T5_DATA01 16 5,041
GDWP_TEMP 1 1,008

For example, Figure 18 lists the 16 Oracle ASM disks assigned to disk group
GDWP_ARCH.

Figure 18. Oracle ASM disks assigned to disk group GDWP_ARCH

27 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Oracle database recovery remote site

This section describes the restore and recovery of the Oracle database at the remote
site, as outlined in Figure 19.

Figure 19. Database restore and recovery process

Network settings
When the appliance arrived at the remote data center, the IP address had to be
changed to enable connectivity to the remote network. Link aggregation was already
in place, so this was simply a case of changing the IP address of the virtual interface,
as shown in Figure 20.
Overview
DD660 setup

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
28

Figure 20. Reset the IP address
Unlock file system
The Data Domain file system was locked during transit and could not be enabled
without the passphrase, as indicated in Figure 21. The passphrase was sent to Cork
separately from the DD660 appliance.

Figure 21. Data Domain file system enable refusedpassphrase required
The filesys encryption unlock command allows the user to enter the passphrase to
unlock the file system, as shown in Figure 22.

Figure 22. Unlocking and enabling the Data Domain file system



29 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review


With the Data Domain file system unlocked, the database was then restored from the
DD660 using RMAN. This process involved the following steps:
1. Mount the DD660 on the /backup directory, as shown in Figure 23.

Figure 23. Mounting the DD660 on the remote site server
2. Create the Oracle database and build the RMAN recovery catalog from the
export file held on the DD660. Figure 24 shows a summary listing of the
database and control file backups from the catalog.

Figure 24. Summary of database and control file backups
3. Use RMAN to connect to the target database and the recovery catalog.
4. Start up the Oracle instance without mounting the database (nomount mode),
as shown in Figure 25.
RMAN restore

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
30

Figure 25. Starting up the Oracle instance in nomount mode
5. Restore the backup control file to the control_file location specified in the
SPFILE, as shown in Figure 26.

Figure 26. Restoring the backup database control file
6. Allocate the channels and restore the database, as shown in Figure 27.

31 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review


Figure 27. Database restore
7. Mount and open the database, as shown in Figure 28.

Figure 28. Mounting and opening the database
In this use case, further recovery was not required as the database had been
backed up in a consistent state, a backup control file was used, and no
further archive logs needed to be applied.
8. Re-create and size the Oracle temp files for the temporary tablespaces TEMP
and TEMP1. The SQL commands used to do this are shown in Figure 29.

Figure 29. Re-creating the Oracle temp files
This completed the successful restore and recovery of the database and provided the
first step in validating the integrity of the database.
The total time taken for the restore and recovery operations was 27 hours, 8 minutes.

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
32

Figure 30 shows a breakdown of the restored database objects and files and their
mappings to the Oracle ASM disk groups.

Figure 30. Mapping of Oracle database objects to Oracle ASM disk groups
As part of the discovery process, EMC IT supplied the Cork team with Automatic
Workload Repository (AWR) reports from the production database. A number of key
SQL queries in the reports were identified to verify the integrity of the restored
database. At the remote site, each of these queries was extracted from the AWR
reports and run against the restored database.

Verifying the
integrity of the
database

33 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Figure 31 demonstrates this process for one of the queries. The first item is an extract
from the SQL ordered by Gets section of an AWR report. The second item is an extract
from the AWR report for SQL ID: 0v7knzwvnc9u7. The third item shows the Oracle
TKProf output of the trace file gathered for this SQL query when it was run against the
restored database.
Figure 31. Testing the integrity of the restored database




SQL
query ID
SQL
statement

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
34
Data Domain consolidation remote site

The Solutions Center in Cork uses a shared Data Domain DD880 for consolidated
backup on many projects within the lab. To ensure consistency, and to maintain a
copy of the EMC IT Production DB in its original state, Data Domain Replicator was
used to copy the database image to the DD880 at the end of the project, as shown in
Figure 32. The DD660 was then repurposed to another project.

Figure 32. Database consolidation to shared DD880
Data Domain Replicator software provides automated, fast, and reliable replication of
data for disaster recovery (DR), remote office data protection, and multiple site tape
consolidation. Replicator duplicates the compressed, deduplicated data over a
network (including WANs), greatly reducing the demands on the network.
Once replication has been configured between a source and a destination, any new
data written to the source can be automatically replicated to the destinationfor the
use case, a one-off copy was made to the DD880 but continued replication was not
required.

Setting up replication between the DD660 and DD880 involves simply creating a
replication pair by specifying the replica source and destination. This can be done
using the DD Enterprise Manager, as shown in Figure 33, or using the DD OS CLI.
Overview
Replication
process
EMC Data Domain DD880
Cork Solutions Center shared
backup target
Cork Solutions Center
DD Replicator
EMC Data Domain
DD660

35 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review


Figure 33. Creating a replication pair
The Replication view in DD Enterprise Manager provides configuration, status,
performance, and topology information for the replication pair. Figure 34 shows this
information for the use case replication process at completion.

Figure 34. Data Domain Enterprise ManagerReplication view

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
36
Figure 35 shows the timeline for the replication process, which took 23 hours to
complete.

Figure 35. Data Domain replication timeline





37 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review

Conclusion

This white paper captures and describes the process used to transport a 12 TB Oracle
data warehouse between two EMC data centers and to restore and recover the
database at the target site. The data centers were located on different continents and
over 4,000 kilometers apart.
Data security was a critical requirement. If the transport mechanism was not secure,
then it would not meet EMC corporate governance compliance and the move could
not happen. In addition, as the process involved a real-world data warehouse, the
solution had to be nondisruptive to EMC core production activities.
The large size of the database and the long distance involved meant that:
Transfer over a shared WAN link was neither practical nor efficient
Creating a dedicated WAN link was neither practical nor economical
In comparison, backing up the data to a Data Domain DD660 appliance, and
transporting this by commercial carrier to the target site, proved to be a secure,
practical, economical, and efficient solution.
At the primary site, EMC TimeFinder enabled offloading of the backup process to a
non-production server, and using an NFS mount allowed simple RMAN backup and
restore on the DD660.

The key findings of this solution include:
Nondisruptive backup
EMC TimeFinder and seamless Data Domain integration with Oracle RMAN
ensured zero impact to the production environment.
Security and corporate governance compliance
Data Domain data integrity, Data Domain Encryption, and Data Domain file
system encryption locking secured the system, enabling it to be shipped by
standard commercial transport.
Simple restore and recovery
Using an NFS mount allowed simple RMAN backup and restore from disk. Data
Domain technology ensured balanced backup and restore speeds.
The 12 TB data warehouse was available for use within 3 days of arrival in Cork.
Economic transport
Data Domain deduplication reduced the database storage and transport
requirements from 12 TB to 3.3 TBa small form factor DD660 (2U, 60 kg)
minimized shipping costs.
Summary

Findings

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
38
References

For additional information, see the white papers listed below.
EMC IT's Migration to the Open, Expandable Oracle BI Grid Applied
Technology
EMC Backup and Recovery for Oracle Database 11g Data Warehouse A
Detailed Review

For additional information, see the product documents listed below.
EMC DD OS 4.9 Administration Guide
EMC Solutions Enabler Symmetrix CLI 7.2 Command Reference

For additional information, see the documents listed below.
Oracle Clusterware Administration and Deployment Guide 11g Release 1 (11.1)
Oracle Real Application Clusters Administration and Deployment Guide 11g
Release 1 (11.1)
Oracle Clusterware Installation Guide 11g Release 1 (11.1) for Linux
Oracle Database Installation Guide 11g Release 1 (11.1) for Linux
Oracle Real Application Clusters Installation Guide 11g Release 1 (11.1) for
Linux and UNIX
Oracle Database Backup and Recovery Reference 11g Release 1 (11.1)
Oracle Database Backup and Recovery User's Guide 11g Release 1 (11.1)

White papers
Product
documentation
Other
documentation