2.
12 Re-Imaging the Oracle Exadata Database Server 2-102
2.12.1 Contact Oracle Support Services 2-103
2.12.2 Download Latest Release of Cluster Verification Utility 2-103
2.12.3 Remove the Database Server from the Cluster 2-103
2.12.4 Image the Database Server 2-104
2.12.5 Configure the Re-imaged Database Server 2-104
2.12.6 Prepare the Re-imaged Database Server for the Cluster 2-105
2.12.7 Apply Oracle Exadata System Software Patch Bundles to the Replacement Database Server 2-107
2.12.8 Clone Oracle Grid Infrastructure to the Replacement Database Server 2-108
2.12.9 Clone Oracle Database Homes to the Replacement Database Server 2-111
2.13 Changing Existing Elastic Configurations for Database Servers 2-113
2.13.1 Adding a New Database Server to the Cluster 2-113
2.13.2 Moving an Existing Database Server to a Different Cluster 2-114
2.12 Re-Imaging the Oracle Exadata Database Server
The re-image procedure is necessary when a database server needs to be brought to an initial
state for various reasons.
Some examples scenarios for re-imaging the database server are:
• You want to install a new server and need to use an earlier release than is in the image
already installed on the server.
• You need to replace a damaged database server with a new database server.
• Your database server had multiple disk failures causing local disk storage failure and you
do not have a database server backup.
• You want to repurpose the server to a new rack.
During the re-imaging procedure, the other database servers on Oracle Exadata are available.
When the new server is added to the cluster, the software is copied from an existing database
server to the new server. It is your responsibility to restore scripting, CRON jobs, maintenance actions,
and non-Oracle software.
Note : The procedures in this section assume the database is Oracle Database 11g Release
2 (11.2) or later.
Starting with Oracle Exadata System Software release 19.1.0, Secure Eraser is automatically
started during re-imaging if the hardware supports Secure Eraser. This significantly simplifies
the re-imaging procedure while maintaining performance. Now, when re-purposing a rack, you
only have to image the rack and the secure data erasure is taken care of transparently as part
of the process.
The following tasks describes how to re-image an Oracle Exadata Database Server running
Oracle Linux:
• Contact Oracle Support Services
If a failed server is being replaced, open a support request with Oracle Support Services.
• Download Latest Release of Cluster Verification Utility
The latest release of the cluster verification utility (cluvfy) is available from My Oracle Support.
• Remove the Database Server from the Cluster
If you are repurposing a deployed server or reimaging a failed server, you must remove the
server from the associated cluster before you reimage it. If you are reimaging the server for
a different reason, skip this task and proceed with the reimaging task next.
• Image the Database Server
After the database server has been installed or replaced, you can image the new database
server.
• Configure the Re-imaged Database Server
The re-imaged database server does not have any host names, IP addresses, DNS or
NTP settings. The steps in this task describe how to configure the re-imaged database
server.
• Prepare the Re-imaged Database Server for the Cluster
This task describes how to ensure the changes made during initial installation are done to
the re-imaged, bare metal database server.
• Apply Oracle Exadata System Software Patch Bundles to the Replacement Database
Server.
Oracle periodically releases Oracle Exadata System Software patch bundles for Oracle
Exadata.
• Clone Oracle Grid Infrastructure to the Replacement Database Server
This procedure describes how to clone Oracle Grid Infrastructure to the replacement
database server.
• Clone Oracle Database Homes to the Replacement Database Server
The following procedure describes how to clone the Oracle Database homes to the
replacement server.
2.12.1 Contact Oracle Support Services
If a failed server is being replaced, open a support request with Oracle Support Services.
The support engineer will identify the failed server, and send a replacement. The support engineer will
ask for the output from the imagehistory server. The output provides a link to the command run from a
running database computeImageMaker file that was used to image the original database server, and
provides a means to restore the system to the same level.
2.12.2 Download Latest Release of Cluster Verification Utility
The latest release of the cluster verification utility ( cluvfy ) is available from My Oracle Support. See My
Oracle Support note 316817.1 for download instructions and other information. Related Topics • My
Oracle Support note 316817.1: Cluster Verification Utility (CLUVFY) FAQ
2.12.3 Remove the Database Server from the Cluster
If you are repurposing a deployed server or reimaging a failed server, you must remove the
server from the associated cluster before you reimage it. If you are reimaging the server for a
different reason, skip this task and proceed with the reimaging task next.
To remove an Exadata database server from an existing Oracle Grid Infrastructure cluster,
following the procedure for Deleting a Cluster Node on Linux and UNIX Systems in the Oracle
Clusterware Administration and Deployment Guide.
Note: If the server being removed has failed or is inaccessible, you can skip any steps that require
running commands on that server.
2.12.4 Image the Database Server
After the database server has been installed or replaced, you can image the new database server. You
can use installation media on a USB thumb drive, or a touchless option using PXE or ISO attached to the
ILOM. See Imaging a New System in Oracle Exadata Database Machine Installation and Configuration
Guide for the details.
2.12.5 Configure the Re-imaged Database Server
The re-imaged database server does not have any host names, IP addresses, DNS or NTP
settings. The steps in this task describe how to configure the re-imaged database server.
You need the following information prior to configuring the re-imaged database server:
• Name servers
• Time zone, such as Americas/Chicago
• NTP servers
• IP address information for the management network
• IP address information for the client access network
• IP address information for the RDMA Network Fabric
• Canonical host name
• Default gateway
The information should be the same on all database servers in Oracle Exadata. The IP
addresses can be obtained from DNS. In addition, a document with the information should
have been provided when Oracle Exadata was installed.
The following procedure describes how to configure the re-imaged database server:
1. Power on the replacement database server. When the system boots, it automatically runs
the Configure Oracle Exadata routine, and prompts for information.
2. Enter the information when prompted, and confirm the settings. The start up process will
continue.
Note:
• If the database server does not use all network interfaces, then the configuration process stops, and
warns that some network interfaces are disconnected. It prompts whether to retry the discovery
process. Respond with yes or no , as appropriate for the environment.
• If bonding is used for the client access network, then it is set in the default active passive mode at this
time.
2.12.6 Prepare the Re-imaged Database Server for the Cluster
This task describes how to ensure the changes made during initial installation are done to the re-imaged,
bare metal database server.
1. Copy or merge the contents of the following files using files on a working database server
as reference:
a. Copy the contents of the /etc/security/limits.conf file.
b. Merge the contents of the /etc/hosts files.
c. Copy the /etc/oracle/cell/network-config/cellinit.ora file.
d. Update the /etc/oracle/cell/network-config/cellinit.ora file with the
IP_ADDRESS of the ifcfg-bondib0 interface (in case of active/passive bonding) or
ib0 and ib1 interfaces (in case of active/active bonding) of the replacement server.
e. Copy the /etc/oracle/cell/network-config/cellip.ora file.
The content of the cellip.ora file should be the same on all database servers.
f. Configure additional network requirements, such as 10 GbE.
g. Copy the modprobe configuration.
The contents of the configuration file should be the same on all database servers.
• Oracle Linux 5 or 6: The file is located at /etc/modprobe.conf.
• Oracle Linux 7 or later: The file is located at /etc/modprobe.d/exadata.conf.
h. Copy the /etc/sysctl.conf file.
The contents of the file should be the same on all database servers.
i. Update the cellroute.ora.
Make a copy of the /etc/oracle/cell/network-config/cellroute.ora file. Modify
the contents on the replacement server to use the local InfiniBand interfaces on the
new node.
j. Restart the database server so the network changes take effect.
2. Set up the users for the software owners on the replacement database server by adding
groups.
If you are using role-separated management, then the users are usually oracle and grid.
If you use a single software owner, then the user is usually oracle. The group information
is available on a working database server.
a. Obtain the current group information from a working database server.
# id oracle
uid=1000(oracle) gid=1001(oinstall)
groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)
b. Use the groupadd command to add the group information to the replacement database
# groupadd -g 1001 oinstall
# groupadd -g 1002 dba
# groupadd -g 1003 oper
# groupadd -g 1004 asmdba
c. Obtain the current user information from a working database server.
# id oracle uid=1000(oracle) gid=1001(oinstall) \
groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)
d. Add the user information to the replacement database server.
# useradd -u 1000 -g 1001 -G 1001,1002,1003,1004 -m -d /home/oracle -s \
/bin/bash oracle
e. Create the Oracle Base and Grid home directories, such as
and /u01/app/12.2.0.1/grid
# mkdir -p /u01/app/oracle
# mkdir -p /u01/app/12.2.0.1/grid
# chown -R oracle:oinstall /u01/app
f.Change the ownership on the cellip.ora and cellinit.ora files.
The ownership is usually oracle:oinstall.
# chown -R oracle:oinstall /etc/oracle/cell/network-config
g. Secure the restored database server.
# chmod u+x /opt/oracle.SupportTools/harden_passwords_reset_root_ssh
# /opt/oracle.SupportTools/harden_passwords_reset_root_ssh
The database server restarts. Log in as the root user when prompted by the system.
You are prompted for a new password. Set the password to match the of the other database servers.
h. Set the password for the Oracle software owner.
The owner is usually oracle.
# passwd oracle
3. Set up SSH for the oracle account.
a. Log in to the oracle account. account on the replacement database server.
# su - oracle
b. Create the dcli root password group file on the replacement database server listing the servers in the
Oracle cluster.
c. Run the following command on the replacement database server.
$ dcli -g dbs_group -l oracle -k
d. Exit and log in again as the oracle user
$ exit
# su - oracle
e. Verify SSH equivalency.
$ dcli -g dbs_group -l oracle date
4. Set up or copy any custom login scripts from a working database server to the replacement database
server.
In the following command, replacement_server is the name of the new server, such as dm01db01 .
$ scp .bash* oracle@replacement_server:.
2.12.7 Apply Oracle Exadata System Software Patch Bundles to the
Replacement Database Server
Oracle periodically releases Oracle Exadata System Software patch bundles for Oracle
Exadata.
If a patch bundle has been applied to the working database servers that was later than the
release of the computeImageMaker file, then the patch bundle must be applied to the
replacement Oracle Exadata Database Server. Determine if a patch bundle has been applied
as follows:
• Prior to Oracle Exadata System Software release 11.2.1.2.3, the database servers did not
maintain version history information. To determine the release number, log in to Oracle
Exadata Storage Server, and run the following command:
imageinfo –ver
If the command shows a different release than the release used by the computeImageMaker
file, then Oracle Exadata System Software patch has been applied to Oracle Exadata
Database Machine and must be applied to the replacement Oracle Exadata Database
Server.
• Starting with Oracle Exadata System Software release 11.2.1.2.3, the imagehistory
command exists on the Oracle Exadata Database Server. Compare information on the
replacement Oracle Exadata Database Server to information on a working Oracle Exadata
Database Server. If the working database has a later release, then apply the Oracle
Exadata Storage Server patch bundle to the replacement Oracle Exadata Database
Server.
2.12.8 Clone Oracle Grid Infrastructure to the Replacement Database Server
This procedure describes how to clone Oracle Grid Infrastructure to the replacement database
server.
In the following commands, working_server is a working database server, and
replacement_server is the replacement database server. The commands in this procedure are
run from a working database server as the Grid home owner. When the root user is needed to
run a command, it will be called out.
1. Verify the hardware and operating system installation using the cluster verification utility
(cluvfy).
$ cluvfy stage -post hwos -n replacement_server,working_server -verbose
The phrase Post-check for hardware and operating system setup was successful
should appear at the end of the report. If the cluster verification utility fails to validate the
storage on the replacement server, you can ignore those messages.
2. Verify peer compatibility.
$ cluvfy comp peer -refnode working_server -n replacement_server \
-orainv oinstall -osdba dba | grep -B 3 -A 2 mismatched
The following is an example of the output:
Compatibility check: Available memory [reference node: dm01db02]
Node Name Status Ref. node status Comment
------------ ----------------------- ----------------------- ---------
dm01db01 31.02GB (3.2527572E7KB) 29.26GB (3.0681252E7KB) mismatched
Available memory check failed
Compatibility check: Free disk space for "/tmp" [reference node: dm01db02]
Node Name Status Ref. node status Comment
------------ ----------------------- ---------------------- ---------
dm01db01 55.52GB (5.8217472E7KB) 51.82GB (5.4340608E7KB) mismatched
Free disk space check failed
If the only failed components are related to the physical memory, swap space and disk
space, then it is safe to continue.
3. Perform the requisite checks for adding the server.
a. Ensure the GRID_HOME/network/admin/samples directory has permissions set to
750.
b. Validate the addition of the database server.
Run the following command as the oracle user. The command prompts for the
password of the root user.
$ cluvfy stage -pre nodeadd -n replacement_server -fixup -method root
verbose
Enter "ROOT" password:
If the only failed component is related to swap space, then it is safe to continue.
If the command returns an error, then set the following environment variable and rerun
the command:
$ export IGNORE_PREADDNODE_CHECKS=Y
4. Add the replacement database server to the cluster.
If you are using Oracle Grid Infrastructure release 12.1 or higher, include the
CLUSTER_NEW_NODE_ROLES attribute, as shown in the following example.
$ cd GRID_HOME/addnode
$ ./addnode.sh -silent "CLUSTER_NEW_NODES={replacement_server}" \
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={replacement_server-vip}" \
"CLUSTER_NEW_NODE_ROLES={hub}"
The second command causes Oracle Universal Installer to copy the Oracle Clusterware
software to the replacement database server. A message similar to the following is
displayed:
WARNING: A new inventory has been created on one or more nodes in this
session.
However, it has not yet been registered as the central inventory of this
system. To register the new inventory please run the script at
'/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes
'dm01db01'. If you do not register the inventory, you may not be able to
Update or patch the products you installed.
The following configuration scripts need to be executed as the "root" user
in each cluster node:
/u01/app/oraInventory/orainstRoot.sh #On nodes dm01db01
/u01/app/12.1.0.2/grid/root.sh #On nodes dm01db01
5. Run the configuration scripts.
As the root user, first disable HAIP, then run the orainstRoot.sh and root.sh scripts
on the replacement database server using the commands shown in the following example.
# export HAIP_UNSUPPORTED=true
# /u01/app/oraInventory/orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
# GRID_HOME/root.sh
Note: Check GRID_HOME/install/ log files for the output of root.sh script.
If you are running Oracle Grid Infrastructure release 11.2, then the output file created by
the script reports that the listener resource on the replaced database server failed to start.
This is the expected output.
/u01/app/11.2.0/grid/bin/srvctl start listener -n dm01db01 \
...Failed
/u01/app/11.2.0/grid/perl/bin/perl \
-I/u01/app/11.2.0/grid/perl/lib \
-I/u01/app/11.2.0/grid/crs/install \
/u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
After the scripts are run, the following message is displayed:
The Cluster Node Addition of /u01/app/12.1.0.2/grid was successful.
Please check '/tmp/silentInstall.log' for more details.
6. Check the cluster.
$ GRID_HOME/bin/crsctl check cluster -all
**************************************************************
node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
7. If you are running Oracle Grid Infrastructure release 11.2, then re-enable the listener
resource.
Run the following commands on the replacement database server.
# GRID_HOME/grid/bin/srvctl enable listener -l LISTENER \
-n replacement_server
# GRID_HOME/grid/bin/srvctl start listener -l LISTENER \
-n replacement_server
8. Start the disk groups on the replacement server.
a. Check disk group status.
In the following example, notice that disk groups are offline on the replacement server.
$ crsctl stat res –t
-------------------------------------------------------------------------------
Name Target State Server State details
-------------------------------------------------------------------------------
Local Resources
-------------------------------------------------------------------------------
ora.DATAC1.dg
ONLINE ONLINE node1 STABLE
OFFLINE OFFLINE node2 STABLE
ora.DBFS_DG.dg
ONLINE ONLINE node1 STABLE
ONLINE ONLINE node2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE node1 STABLE
ONLINE ONLINE node2 STABLE
ora.RECOC1.dg
ONLINE ONLINE node1 STABLE
OFFLINE OFFLINE node2 STABLE
b. For each offline disk group, run the START DISKGROUP
command for each disk group that is offline from either the original server or the replacement
server.
$ srvctl start diskgroup -diskgroup dgname
Related Topics
• Oracle Real Application Clusters Administration and Deployment Guide
2.12.9 Clone Oracle Database Homes to the Replacement Database Server
The following procedure describes how to clone the Oracle Database homes to the
replacement server.
Run the commands from a working database server as the needed to run a command, it will be
called out.
1. Add the Oracle Database ORACLE_HOME to the replacement database server using the
following commands:
$ cd /u01/app/oracle/product/12.1.0.2/dbhome_1/addnode
$ ./addnode.sh -silent "CLUSTER_NEW_NODES={replacement_server}"
The second command causes Oracle Universal Installer to copy the Oracle Database
software to the replacement database server.
WARNING: The following configuration scripts need to be executed as the
"root"
user in each cluster node.
/u01/app/oracle/product/12.1.0.2/dbhome_1/root.sh #On nodes dm01db01
To execute the configuration scripts:
Open a terminal window.
Log in as root.
Run the scripts on each cluster node.
After the scripts are finished, the following messages appear:
The Cluster Node Addition of /u01/app/oracle/product/12.1.0.2/dbhome_1 was
successful.
Please check '/tmp/silentInstall.log' for more details.
2. Run the following script on the replacement database server:
# /u01/app/oracle/product/12.1.0.2/dbhome_1/root.sh
Check the /u01/app/orcale/product/12.1.0.2/dbhome_1/install/
root_replacement_server.com_date.log file for the output of the script.
3. Run the Oracle Database Configuration Assistant (DBCA) in interactive mode to add
database instances to the target nodes.
a. Start up DBCA.
$ cd /u01/app/oracle/product/12.1.0.2/dbhome_1/bin
$ ./dbca
b. On the Database Operation screen, select Instance Management. Click Next.
c. On the Instance Operation screen, select Add an instance. Click Next.
d. On the Database List screen, select the cluster database to which you want to add an
instance.
e. The List Instance screen displays the current instances. Click Next to add a new
instance.
f. The Add Instance screen displays the default name and the newly added node to the
cluster. Accept the defaults and click Next.
g. On the Summary screen, verify the plan and click Finish.
h. On the Progress screen, watch for 100% completion.
i.On the Finish screen, acknowledge the confirmation that the new instance was
successfully added.
Verify that the instance has been added:
$ srvctl config database -db dbm01
Verify the administrative privileges on the target node:
$ cd /u01/app/oracle/product/12.1.0.2/dbhome_1/bin
$ ./cluvfy comp admprv -o db_config -d /u01/app/oracle/product/12.1.0.2/
dbhome_1 -n new_node
4. Ensure the instance parameters are set for the replaced database instance. The following
is an example for the CLUSTER_INTERCONNECTS parameter.
SQL> SHOW PARAMETER cluster_interconnects
NAME TYPE VALUE
------------------------------ -------- -------------------------
cluster_interconnects string
SQL> ALTER SYSTEM SET cluster_interconnects='192.168.73.90' SCOPE=spfile
SID='dbm1';
5. Validate the configuration files as follows:
• The Oracle_home/dbs/initSID.ora file points to the SPFILE in the Oracle ASM
shared storage.
• The password file that is copied in the Oracle_home/dbs directory has been
changed to orapwSID.
6. Check that any services that incorporated this instance before and ensure the services are
updated to include this replacement instance.
7. If this procedure was performed on Oracle Exadata Eighth Rack, then perform the
procedure described in Configuring Oracle Exadata Database Machine Eighth Rack Oracle
Linux Database Server After Recovery.
Related Topics
• Oracle Real Application Clusters Administration and Deployment Guide
