Practice 2

Create Oracle 12c R1 Two-Node RAC Database

Practice Overview
In this practice you will create an Oracle 12c R1 two-node RAC database on the virtual machines that
you created in the previous practice. To accomplish this target, you will perform the following:

• Carry out OS preparation steps

• Install Grid Infrastructure software

• Create ASM Disk Groups

• Install Oracle Database software

• Create the Oracle RAC database

• Learn how to make a copy of the virtual machines for relocation

• Learn how to startup and shutdown Oracle RAC database

• Get the Enterprise Manager Database Express working in the VM

Practice Assumptions
The practice assumes that you have the virtual machines srv1 and srv2 up and running.
Practice Environment Architecture
The following diagram shows the Oracle RAC database architecture that you will create in this
practice:
Practice Environment Preparation Procedure

A. Set the OS environment variables in the Oracle software user owner

profiles

1. In a Putty session, login to srv1 and srv2 as oracle user

In srv1 and srv2, set the OS environment variables in the oracle user profile:
mv ~/.bash_profile ~/.bash_profile_bk
vi ~/.bash_profile

# .bash_profile
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

ORACLE_SID=rac1; export ORACLE_SID

ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/12.1.0/db_1; export ORACLE_HOME
ORACLE_TERM=xterm; export ORACLE_TERM
NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"; export NLS_DATE_FORMAT
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
PATH=.:${PATH}:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/local/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
THREADS_FLAG=native; export THREADS_FLAG
export TEMP=/tmp
export TMPDIR=/tmp
export EDITOR=vi
umask 022

2. In the .bash_profile file of the oracle account in srv2, change the value assigned to
ORACLE_SID from rac1 to rac2.
3. In the terminal sessions connected to srv1 and srv2, switch to grid user.

4. In srv1 and srv2, set the OS environment variables in the grid user profile:
mv ~/.bash_profile ~/.bash_profile_bk
vi ~/.bash_profile

# .bash_profile
# OS User: grid

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_BASE=/u01/app/grid; export ORACLE_BASE
# it must not be under the ORACLE_BASE
ORACLE_HOME=/u01/app/12.1.0/grid; export ORACLE_HOME
ORACLE_TERM=xterm; export ORACLE_TERM
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
PATH=.:${PATH}:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/local/bin
export PATH
export TEMP=/tmp
export TMPDIR=/tmp
umask 022

5. In .bash_profile file of the grid account in srv2, change the value assigned to ORACLE_SID from
+ASM1 to +ASM2.
B. Set the resource limits for the Oracle software installation owners

6. In srv1 and srv2, switch user to root and set the resource limits for the software installation
owner users. It is set for oracle user. You need to set it the same for grid user.
# take backup of existing file:
mv /etc/security/limits.d/oracle-rdbms-server-12cR1-preinstall.conf
/etc/security/limits.d/oracle-rdbms-server-12cR1-preinstall.conf.bak

# create the file and paste the code below in it:

vi /etc/security/limits.d/oracle-rdbms-server-12cR1-preinstall.conf

# oracle-rdbms-server-12cR1-preinstall setting for nofile soft limit is 1024

oracle soft nofile 1024
grid soft nofile 1024

# oracle-rdbms-server-12cR1-preinstall setting for nofile hard limit is 65536

oracle hard nofile 65536
grid hard nofile 65536

# oracle-rdbms-server-12cR1-preinstall setting for nproc soft limit is 16384

# refer orabug15971421 for more info.
oracle soft nproc 16384
grid soft nproc 16384

# oracle-rdbms-server-12cR1-preinstall setting for nproc hard limit is 16384

oracle hard nproc 16384
grid hard nproc 16384

# oracle-rdbms-server-12cR1-preinstall setting for stack soft limit is 10240KB

oracle soft stack 10240
grid soft stack 10240

# oracle-rdbms-server-12cR1-preinstall setting for stack hard limit is 32768KB

oracle hard stack 32768
grid hard stack 32768

# oracle-rdbms-server-11gR2-preinstall setting for memlock hard limit is maximum

of {128GB (x86_64) / 3GB (x86) or 90 % of RAM}
oracle hard memlock 134217728
grid hard memlock 134217728

# oracle-rdbms-server-11gR2-preinstall setting for memlock soft limit is maximum

of {128GB (x86_64) / 3GB (x86) or 90% of RAM}
oracle soft memlock 134217728
grid soft memlock 134217728

7. In srv1 and srv2, set the number of processes a user can create to 2048
vi /etc/security/limits.d/90-nproc.conf

* soft nproc 2048

C. Install Grid Infrastructure Software

8. In the Oracle VirtualBox window, login as grid to srv1. Open a terminal window, change the
current window to the GI installation files and start the installer.
# the following source command is needed only if you were already logged on when
# you edited the profile file.
source .bash_profile

cd /media/sf_staging/grid
./runInstaller

9. Respond to the Installer utility windows as follows:

Window Response

Installation Option Install and Configure Oracle Grid Infrastructure for a Cluster

Cluster Type Configure a Standard Cluster

Installation Type Advanced Installation

Product Language English

Grid Plug and Play Cluster Name: rac

SCAN Name: srv-scan
SCAN Port: 1521
unmark Configure GNS

Cluster Node click on Add button

Information
Public Hostname: srv2.localdomain
Virtual Hostname: srv2-vip.localdomain
click on OK button

click on SSH Connectivity button

enter the OS Password (oracle)
click on Setup button
click on Test button

Note: If the Setup was successful but the Test failed, you may need
to restart the nodes and try again.

Click on Next button

Network Interface eth0 : select Public

Usage eth1 : select Private
eth2 : select Do Not Use

Storage Option Use Standard ASM for Storage

Create ASM Disk click on Change Discovery Path button
Group enter Disk Discovery Path: /dev/oracleasm/disks*
click on OK button

Disk Group Name: CRS

Redundancy: External
Allocation Unit Size: 1MB
Mark DISK1

click on Next button

ASM Password select "Use same password for these accounts"

Specify Password: oracle
Confirm Password: oracle

Failure Isolation select "Do not use Intelligent Platform Management Interface (IPMI)"

Management Options click on Next button

Operating System OSASM Group asmadmin

Groups OSDBA for ASM Group asmdba
OSOPER for ASM Group blank

Installation Location Oracle Base /u01/app/grid

Software Location /u01/app/12.1.0/grid
Note: those values taken from the OS variables

Click on Next button

Create Inventory Inventory Directory /u01/app/oraInventory

Root script execution Mark "Automatically run configuration scripts"

Enter the root password: 111111

Prerequisite Checks The verification takes some time.

Following warnings could be ignored:
- Physical Memory
- Swap Size
- Device Checks for ASM
- Task resolve.conf Integrity

Note: If you receive other warnings, check their details. Resolve the
issue and click on "Check Again" button.

select Ignore All option

select on Next button

Summary click on Install button

 At about %80 You will receive the following message:
progress

click on Yes button

While the script is running, the progress bar does not pass the 84%
value. Do not consider this a hanging status. It takes some time and
it will eventually finish.

In the end of the installation, you will receive the following error:
"Oracle Cluster Verification Utility failed"

The log file has details of the error causes. The following causes can
be safely ignored:
- Sufficient physical memory is not available on node ...
- Sufficient swap size is not available on node ...
- Group of device "/dev/oracleasm/disks/DISK1" did not match the
expected group
- Attempt to get udev information from node ...

10. In srv1, check the status of the running clusterware resources. The state of all the resources
should be ONLINE.
crsctl status resource -t

11. Ensure that all the cluster services are up and running in all the cluster nodes.
crsctl check cluster -all

Note: in real life scenario, you are always advised to apply the latest patch set on the grid software
home straight away after installing it. You will learn how to apply a PSU on Oracle Grid Infrastructure
later in the course.
D. Mount the ASM Disk Groups

In this section of the practice, you will mount the DATA and FRA diskgroups in ASM.

12. In the Oracle VirtualBox window, make sure you are logged on as grid to srv1.

13. Open a terminal windows and start asmca utility.

asmca

14. Click on Create button

15. In the Disk Group Name field enter DATA, for Redundancy select "External", and mark DISK2,
then click on OK button. You should see after a few seconds a diskgroup creation success
message.
16. Similarly, create FRA diskgroup.

17. Eventually, the asmca window should look like the following screenshot. All the diskgroups must
be mounted and see by the two nodes.

Click on Exit button.

E. Install Oracle Database Software

18. In the Oracle VirtualBox window, logout from srv1 and login as oracle.

19. Create the sqlnet.ora file and add the following code in it.
mkdir -p $ORACLE_HOME/network/admin
vi $ORACLE_HOME/network/admin/sqlnet.ora

DIAG_ADR_ENABLED=ON
NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)

20. Change the directory to the installation files directory and start the installer.
cd /media/sf_staging/database/
./runInstaller

21. Respond to the Installer utility windows as follows:

Window Response

Configure Security Uncheck the "I wish to receive security updates..."

Update
press Next button then Yes button

Installation Option select "Install database software only" option

Grid Installation select "Oracle Real Application Clusters database installation"

Options

Nodes Selection Make sure both nodes are selected.

Press on SSH Connectivity
enter oracle password
click Setup button.
After the SSH connectivity setup is finished,
click on Test button to test it.
click on Next button.

Product Language Make sure English is selected

click on Next button.

Database Edition select Enterprise Edition option

Installation Location Oracle Base /u01/app/oracle

Software Location /u01/app/oracle/product/12.1.0/db_1
Note: those values taken from the OS variables

Click on Next button

 You will receive the following message, click on Yes button:

Operating System select dba group for all the options except OSOPER keep it blank
Groups

Prerequisite Checks The following warnings could be ignored:

- Swap Size
- Task resolv.conf Integrity
- Single Client Access Name (SCAN)

Note: If you receive other warnings, check their details. Resolve the
issue and click on "Check Again" button.

Click on Ignore All option

Summary click on Install button

Execute Execute the root script when prompted on both nodes.

Configuration scripts
Execute the script in the second node only after it finishes execution
in the first node.
click on Close button

Note: as is the case with Oracle grid software, it is recommended to apply the latest patch sets on the
Oracle software straight away after installing it. This is much faster than applying them after creating
the database.
F. Create the Oracle RAC Database

22. In the Oracle VirtualBox window, make sure you are logged on as oracle.

23. Start the dbca utility

dbca

Note: observe that you do not create a listener in oracle home. The listener starts up from the grid
home and controlled by the clusterware.

24. Respond to the dbca windows as follows:

Window Response

Database Operation select Create Database option

Creation Mode select Advanced Mode option

Database Template Make sure the Database Type is Oracle Real Application Clusters
(RAC) database
Change the Configuration Type to Admin Managed
Make sure that the General Purpose or Transaction Processing
option is selected.

Database Global Database Name: rac.localdomain

Identification
SID Prefix: rac
Make sure that the "Create As Container Database" checkbox is not
marked.

Database Placement Move the node srv2 from the Available list to the Selected list

Management Options Unmark the "Run Cluster Verification Utility (CVU) Checks
Periodically" checkbox
Mark the "Configure Enterprise Manager (EM) Database
Express" checkbox

Database Credentials set the password for the users

Storage Locations Database files Storage Type: ASM

Use Common Location for All Database Files: +DATA
Mark the "Use Oracle-Managed Files" checkbox
Recovery files Storage Type: ASM
Mark the Specify Fast Recovery Area
Fast Recovery Area: +FRA
Fas Recovery Size: 12 GB
Make sure that the Enable Archiving checkbox is Unmarked

Database Options Mark the Sample Schemas checkbox

Initialization Rise the Memory Slider to nearly 50% then select the Custom
Parameters Settings. The SGA size and the PGA size will automatically be filled.
SGA Size: 1,500 M Bytes
PGA Size: 512 M Bytes

click on the Sizing tab

Processes: 500

click on Character Sets tab

select "Use Unicode (AL32UTF8)" option

click on Next button

Creation Options Make sure "Create Database" option is selected.

Prerequisite Checks Ignore the following warnings

- Single Client Access Name SCAN

click on "Ignore All" option

click on "Next" button

Summary click on "Finish" button

Database click on "Close" button

Configuration
Assistant
G. First examination on the Oracle RAC database

In this section of the practice, you will perform some initial checking on the create RAC database.

25. Log in as oracle to srv1 on a Putty session.

26. Make sure that the SCAN hostname replies to ping command.
ping -c 3 srv-scan

27. Issue the following commands and examine their output.

srvctl status database -d rac
srvctl config database -d rac

28. Identify the database instance names that are currently running on srv1 from the OS command
shell.
ps -ef | grep -i pmon

29. Login as sysdba to the database and examine the contents of v$active_instances
sqlplus / as sysdba
col inst_name format a50
SELECT INST_NUMBER , INST_NAME FROM V$ACTIVE_INSTANCES;

30. Make sure the tnsnames.ora file has been configured for connecting to rac database. This has
automatically been done by the dbca utility.
cat $TNS_ADMIN/tnsnames.ora

Note: Applications should connect to the RAC database using dynamic services. Services will be
covered in a later practice.

31. In the following sub-steps, you will examine how the client sessions are distributed among the
database instances. You will create multiple SQL*Plus sessions and check to which instance every
session is connected to.
a. Connect to rac database as system using the password authentication.
sqlplus system/oracle@rac

b. Retrieve the instance name that the current SQL*Plus session is connected to.
SELECT INSTANCE_NAME FROM V$INSTANCE ;

c. Issue the following command in SQL*Plus. This command will open a new OS sub-shell.
The SQL*Plus session is still active. When you exit from the OS sub-shell, you will get back
to the SQL*Plus session.
!
d. Open another SQL*Plus session using the same credential.
sqlplus system/oracle@rac

e. Check the instance name that current SQL*Plus session is connected to.
f. Repeat the previous steps twice, and check the current instance name in each time. You will
observe that the sessions are distributed in a round-robin fashion.
g. Exit from all the SQL*Plus sessions created above.
H. Shutdown the RAC database

In this section of the practice, you will learn the clean way of shutting down the Oracle RAC database
and the Clusterware technology stack.

32. Make sure you are logged on as oracle to srv1 in a Putty session then issue the following
command to display help information about using srvctl stop command.
This is a handy way to remember the format of using the command when you forget it.
srvctl stop database -h

33. Use the srvctl to stop rac database. This command shuts down the instances in all the cluster
nodes.
-d can be replaced with “-database”, and “-o” is a shortcut to “stop_options”
srvctl stop database -d rac -o immediate
srvctl status database -d rac

34. Log in as root to srv1 and issue the following command. This command shuts down the
Clusterware technology stack.
Observe that the full path of the crsctl should be used. This is because that path of this utility is
not included in the PATH variable defined for the root user. When you use the crsctl utility with
grid user, you do not have to use the full utility pathname because it is included in the PATH
value for the grid user.
Some clusterware commands should be executed by root and some of them should be run by
grid.
/u01/app/12.1.0/grid/bin/crsctl stop crs

35. Log in as root to srv2 and issue the same command as in the previous step to stop the
Clusterware technology stack.

36. Shutdown all the virtual machines.

I. Making a Copy of the Virtual Machines for relocation

If you want to make a copy of the existing virtual machines to use them on another machine or move
them to another disk drive, the traditional method of copying the virtual machine folders will not work.
This is because the disk UID of the shared disks is registered in Oracle VirtualBox. If you just copy the
disk file, Oracle VirtualBox will return an error complaining that the same disk UID is being used.

In the following steps, you will learn the steps that you should follow how to make a copy of the
virtual machines.

Note: you do not have to perform this procedure. Refer to it only if you wish to relocate the system.

oifi90 If your target is to make a backup copy of the existing environment, you do not have to
perform this procedure. Shutting down the machines and copying their parent folder is fair enough.

37. In the VirtualBox Manager open click on File menu | Virtual Media Manager. Then select the
first shared disk used by the virtual machines (DISK1).

38. Click on the "Release" icon and then confirm in the pop-up window. Note that this disk now
shows as "Not attached".

39. Perform the same previous two steps on the remaining two shared disks: DISK2 and DISK3.

40. Select DISK1 and then click on "Copy" to start the Disk Copying Wizard.

41. Accept the Virtual disk to copy and click on "Next".

42. Accept the VDI Virtual disk file type and click on "Next".

43. Select "Fixed size" and click on "Next".

44. On the next screen you can set the location and name of the new file. After that, click on "Next".

45. Click on "Copy" button to start copying the disk.

46. Perform the same steps to copy the remaining disks.

47. Close the Media Manager

48. Export srv1 and srv2 virtual machines: in the VirtualBox Manager select the VM, then click on
File | Export Appliance.
a. Select the virtual machines, click on Next
b. Set the location where you want to export the appliances. Click on Next
c. Accept the appliance settings, click on Export

49. You can now copy the exported file and the copy of the shared disks to the new location.

50. Re-attach the shared disks in srv1: in Oracle VirtualBox, select the virtual machine, click on
Settings, select Storage option from the right pane, click on the Sata Controller, then click on
Add Hardisk. Click on Choose Existing Disk option.

51. Perform the same pervious step on srv2.

52. Start srv1 and srv2

J. Startup the RAC database

By default, when you boot up a cluster node, the Clusterware stack itself automatically starts up and it
starts up the database service to its pervious open state. If the database was up when you shut down
the node, the Clusterware starts up the database when you reboot the server. If the database was
down when you shut down the node, the Clusterware will NOT start up the database when you reboot
the server.

Note: You will learn the details of the auto-restart setting in the next lecture.

53. Login as grid to srv1 in a Putty session

54. Check the status of the crs resources.
crsctl status resource -t

55. Check the AUTO_START attribute value of the rac database service.
This attribute controls whether the resource should be automatically started when the machine is
rebooted or not. Observe that this is different from the auto-restart mechanism in the single-
instance database.
crsctl status res ora.rac.db -f | grep AUTO_START

The restore value means that the resource will be started to the same state that it was in when
the server stopped. Oracle Clusterware attempts to restart the resource if the value of TARGET
was ONLINE before the server stopped.

56. Make sure the autostart service is enabled (must be run as root).
su -
/u01/app/12.1.0/grid/bin/crsctl config has

57. Switch user to oracle then start up the database.

Note: You will learn the details of starting up a database in the next lecture.
su - oracle
srvctl start database -d rac
srvctl status database -d rac
K. Getting the Enterprise Manager Database Express Working in the VM

In this section of the practice, you will open the Oracle Enterprise Manager Database Express. Because
the pre-installed Mozilla Firefox in the virtual appliance does not have the Adobe Flash plug-in installed
in it by default, you need to install the plug-in in the browser.

58. In the VirtualBox window of srv1, login as oracle user.

59. Open the Mozilla Firefox browser and open the EM Database Express page using the following URL
address:

https://srv-scan.localdomain:5500/em

60. You will receive a warning message about trusting the URL source, as follows:
61. Click on the "Add Exception" button. You will see the following message popping up:

62. Click on the "Confirm Security Exception" button.

63. You will see the following message in the browser:

64. If your VM appliance has access to the Internet, go to next step. If you VM appliance does not
have a connection to the Internet, you can install the Flash plug-in manually by performing the
following steps.

a. Download the file "flash_player_npapi_linux.x86_64.tar.gz" from the lecture downloadable

resources.
b. Copy the file flash_player_npapi_linux.x86_64.tar.gz to the directory /home/oracle/source
c. Connect via Putty to the machine srv1 as root
d. Extract the file
tar -xvzf flash_player_npapi_linux.x86_64.tar.gz

e. Copy the extracted so file to the plug-ins directory

cp /home/oracle/source/libflashplayer.so /usr/lib64/mozilla/plugins

f. Change the permissions of the file as follows:

chmod 775 libflashplayer.so /usr/lib64/mozilla/plugins/libflashplayer.so
65. In the following steps you will download and install the Flash plug-in in the FireFox browser in
srv1 machine. Those steps assume that the machine has a connection to the Internet:

a. In the message that appeared to you, click on "GetFlash" link

g. The page that follows appears in the browser. Select ".rpm for Linux" option from the drop
list then click on Download button.

h. The following message appears to you. Click on OK button.

 i. The following message appears to you. Click on Install button.

j. The following message appears to you. Enter the root password then click on Authenticate
button

66. After the Flash plugin got installed, close the browser and open it again.

67. Enter the EM URL. You should be able to see the EM Database Express page as follows. Enter the
sys username, its password, mark the "as sysdba" option then click on Login button.
68. You should see the EM Database Express opening and displaying a page similar to the following:
Summary

In this practice, you performed the following:

• Carry out OS preparation steps

• Install Grid Infrastructure software

• Create ASM Disk Groups

• Install Oracle Database Software

• Create the Oracle RAC database

• Learn how to make a copy of the virtual machines for relocation

• Learn how to startup and shutdown Oracle RAC database

• Get the Enterprise Manager Database Express working in the VM machine