Wednesday, July 17, 2013

Post 49 of series - Install Oracle RAC 12c on Vmware with Windows 7 64-bit as host OS and Linux CentOS or RHEL 6 as guest OS (Part 2, Setup Base Brick Machine for RAC 12c install (bbm) and (san_rac_bbm) )



In their dreams at night, people wander around as long as they sleep; just so, they are under the power of the snake Maya, as long as their hearts are filled with ego and duality. Through the Guru's Teachings, they come to understand and see that this world is just a dream.




Now we will create a Base machine, I call it Base Brick Machine as I did in 11gR2 RAC install.

This machine will be used to make the SAN machine and the RAC machines. This note can also be used to install Linux on Windows 7 with Virtual machines.

Open Vmware and click File->New Virtual Machine and you will have following screen:




 Click next and you will see the following:






Click next and you see the following:
Make sure you select “I will install the operating system later”.  Otherwise when we start the machine it will start the default install of linux. I want to do custom install so select this option.




Click next and select “Linux” and Version as “Red Hat Enterprise Linux 6 64-bit”



 Click next and enter name as “bbm” (i.e. base brick machine)
Location as : “C:\VM\rac12c\bbm”




Click next and I will not change anything here and will keep one processor




Click next and I will give 2GB o this machine so that the Linux install is quicker. Later on we will reduce its size.


Click next and select “Use bridged network card”

                                                             


Click next and you will have following and make sure you have “LSI Logic (Recommended)” selected



Click next and select “Create a new virtual disk”


Click next and make sure “SCSI (Recommended)” is selected


Click next and select Maximum disk size as “100 GB” and make sure “Split virtual disk into multiple files” is selected.


Click next and make sure  the path provided is “C:\VM\rac12c\bbm\bbm.vmdk” 


Click next  and then click finish.


You will see the following machine in the Vmware Workstation Console.



Let’s do some modifications to it. Click “Edit virtual machine settings” and you will have the following screen.  




Let’s remove “Floppy, USB Controller, Sound Card and Printer and add some more network cards to the machine.  Select “Floppy” and then click remove.


Select “USB Controller” and click remove and in the same way select “Sound Card” and click remove and also select “Printer” and click remove. 


Finally the machine looks like following:



Now let’s add network cards.  Click “Add”. You will get the following screen and select  “Network Adapter” . 


Click next and select “Host-only” and make sure “Connect at power on” is selected.



Click finish. Add 2  more network cards like this and in the end you will have something like following. Now select “CD/DVD (IDE)” and select “Use ISO image file” and then select its location on your hard drive 



Click “OK”
Now power on the machine and by clicking green play button :





You will see the following screen on the vmware machine and to input commands on the machine just click on the screen inside the vmware workstation. If you want to come out of it press “ALT+CTL” together. We need to select the very first option i.e. “Install or upgrade an existing system” , since its alreadt selected just hit enter.



Then you will see the following screen. Now the machine is being prepared for Linux Install.



 No need to test the media before installation. So with arrow key more to “Skip” and hit enter.



On the following screen just click “Next”




Click Next again.


Click Next again.



Make sure you select “Basic Storage Devices” and then click “Next”.



 Just click “Yes, discard any data”. Don’t worry it will not delete any data from Windows. It will just cleanup stuff on this particular virtual machine and where we have nothing by now.


Then on the following screen name the machine. I am naming it “bbm.rac.meditate.com”
BBM is Base Brick Machine.
“rac.meditate.com”    AS   “Meditation is better than medication to cure all diseases in the world”




On the same screen click  “Configure Network” and the pop up opens. All network interfaces names will be “System eth1”,  “System eth2”, “System eth3”, “System eth0”
We need to fix the ip’s and their names. Select one interface card and the click edit.




Change name “Connection name” to “eth0” and make sure “Connect automatically” is checked and then click tab “IPv4 Settings” and now let’s set the ip’s. Click “Add” and then enter under address “192.168.1.115” and under netmask “255.255.255.0” and under Gateway “192.168.1.254” (My router’s ip address is “192.168.1.254”, if your router address is different use that one, it can be “192.168.1.1” 
In “DNS servers” also enter same router’s ip and is “192.168.1.254” 


Then apply and then click eth1 and edit and make sure “Connect automatically” is checked and then click tab “IPv4 Settings” and now let’s set the ip’s. Click “Add” and then enter under address “10.10.1.115” and under netmask “255.255.255.0” . No need to supply Gateway and DNS server.



As we did in eth1 do the same in eth2 but keep address “10.10.2.115” and under netmask “255.255.255.0” . No need to supply Gateway and DNS server.


As we did in eth1 and eth2 do the same in eth3 but keep address “10.10.3.115” and under netmask “255.255.255.0” . No need to supply Gateway and DNS server.


Click apply and you will see the following.


Click close and you will see the following, click next.




In the following screen select your time zone as I am in Pacific Time Zone in San Diego, I am selecting "America/Los Angeles" . Uncheck "system clock uses UTC option"




On the following screen supply your root password.




 If you get following pop-up just click "Use Anyway"


On the following screen make sure you select "Use All Space" option and make sure you select "Review and modify partition Layout" and click next.



Then you will see the following option and now we will change the size of /home and swap and / mount points.


Click /home and click edit and make its size as 10240 MB i.e. 10GB and click OK.


Then select  swap and edit.


Make swap 5GB, i.e. 5120 MB


Now select / mount point and click edit.


Make its size 86520MB , i.e. just leave 16 or 32 MB unallocated and click OK.


Finally you will see the following and click next.




 Now you will see the following just click "Format". Its safe as it will just format the Vmware machine mountpoints info and will not at all touch anything on your windows machine.






On next screen, click "Write changes to Disk"



Then you will see the following screen.



 On next screen make sure "Install boot loader on "/dev/sda" is selected and radio button is clicked for "CentOS /dev/mapper/vg_bbm-lv_root" is also selected and click next.


Make sure you select "Basic Server" option and "CentOS" is selected and "Customise Now" radio button is also selected and click next.



I am select ing the following packages only. Nothing from applications.


From "Base Systems" just "Base", "Compatibility Libraries", "Console Internet tools", "Debugging Tools", "Directory Client", "Hardware Monitoring Utilities", "Jave Platform", "Large Systems Performance", "Network File system client", "Performance Tools", "Perl Support".






Nothing from Databases

Nothing from "Desktops"




 Select "Additional Development" and "Development Tools" from "Development"



 Nothing from "High Availability"


 Nothing from languages , just keep default



 Noting from "Load Balancer"


Nothing from "Resillent Storage"




 Nothing from "Scaleable filesystem Support"


From servers install "CIFS file Server","FTP Server","NFS file server","Network Infrastructure Server",""Network Storage Server","Server Platform","System Administration Tools"



 Nothing from "System Management"



Nothing from "Virtualization"


Nothing from "web services"



Then Click next and it will take some time and Linux Install will start and wait till it asks for reboot.
 You will see the following screen showing progress of install.




Now click "Reboot"


Now machine will reboot



Then login as root on the console and execute command "shutdown -h now" to shutdown the machine.




You will see the following now.



Now take the backup of the machine and we will make the SAN machine where all our luns will be hosted.
To take the backup, just copy folder "bbm" under directory C:\VM\rac12c in same directory and name it as "san12c".




After copying bbm folder under directory "C:\VM\rac_12c" rename new folder as "san_rac_bbm"
Open "Vmware VIrtual Machine Configuration File" bbm under directoy "C:\VM\rac_12c\san_rac_bbm" in your editor and
Update the following file.

Change following:
scsi0:0.fileName = "C:\VM\rac_12c\bbm\bbm.vmdk"
to
scsi0:0.fileName = "C:\VM\rac_12c\san_rac_bbm\bbm.vmdk"

Now open this machine in vmware workstation as follows:


File -> Open and browse to the following file:
C:\VM\rac_12c\san_rac_bbm\bbm

And click open and you will see the following screen:



Click “Edit Virtual Machine Settings” and you will see the following screen and here update the virtual machine name to “san_rac_bbm”  





Click Ok.




Now start the machine by clicking “Power on this virtual machine” and then on the following screen click “I copied it” and the machine will start now.




Once started on the following screen login as root:






Once logged in as root vi file “/etc/sysconfig/network” to change the hostname of the machince to “san_rac_bbm”.
In this file update line:
HOSTNAME=bbm.rac.meditate.com
To
HOSTNAME=san_rac_bbm.rac.meditate.com
Save the file.

Now we will fix the networking stuff of machine so that we can connect to the machine with putty.
To do that execute the following command as root:
rm –f /etc/udev/rules.d/70-persistent-net.rules

Now vi files:
/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network-scripts/ifcfg-eth1

/etc/sysconfig/network-scripts/ifcfg-eth2
/etc/sysconfig/network-scripts/ifcfg-eth3

and remove the existing HWADDR line in all the above mentioned files. Noew restart the machine with command “reboot”

Now putty should connect, here my settings for putty:

Download the putty.exe from following link:


Now create a directory under “C:” drive on windows machine “putty_logs” to store the putty_logs.

Open it by double clicking it and do the following settings:









Login as root and try ping to www.oracle.com and you will see its connecting. If it not pinging, you have missed some step so you will need to look at it.

As follows:

login as: root
root@192.168.1.115's password:
Last login: Wed Jul 17 19:18:07 2013 from 192.168.1.70
[root@san_rac_bbm ~]#
[root@san_rac_bbm ~]#
[root@san_rac_bbm ~]#
[root@san_rac_bbm ~]# ping ww.oracle.com
ping: unknown host ww.oracle.com
[root@san_rac_bbm ~]# ping www.oracle.com
PING e7075.b.akamaiedge.net (23.43.182.140) 56(84) bytes of data.
64 bytes from a23-43-182-140.deploy.static.akamaitechnologies.com (23.43.182.140): icmp_seq=1 ttl=54 time=25.7 ms
64 bytes from a23-43-182-140.deploy.static.akamaitechnologies.com (23.43.182.140): icmp_seq=2 ttl=54 time=25.0 ms
64 bytes from a23-43-182-140.deploy.static.akamaitechnologies.com (23.43.182.140): icmp_seq=3 ttl=54 time=24.8 ms
^C
--- e7075.b.akamaiedge.net ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2366ms
rtt min/avg/max/mdev = 24.801/25.195/25.703/0.418 ms
[root@san_rac_bbm ~]#

Now we will install package “oracle-rdbms-server-12cR1-preinstall” on this machine, it will create oracle user and will groups and will modify the kernel parameters required for install. But to install it we will have add http://public-yum.oracle.com to our yum library to add it to yum library execue the following commands as root on putty window.

You will get something like this:

[root@san_rac_bbm ~]# cd /etc/yum.repos.d
[root@san_rac_bbm yum.repos.d]#
[root@san_rac_bbm yum.repos.d]#
[root@san_rac_bbm yum.repos.d]# wget http://public-yum.oracle.com/public-yum-ol6.repo
--2013-07-17 19:39:00--  http://public-yum.oracle.com/public-yum-ol6.repo
Resolving public-yum.oracle.com... 137.254.56.43
Connecting to public-yum.oracle.com|137.254.56.43|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2887 (2.8K) [text/plain]
Saving to: âpublic-yum-ol6.repoâ

100%[====================================================================================================================================================>] 2,887       --.-K/s   in 0.007s

2013-07-17 19:39:00 (429 KB/s) - âpublic-yum-ol6.repoâ

[root@san_rac_bbm yum.repos.d]#
[root@san_rac_bbm yum.repos.d]#

Now we will execute yum to install this package as root (Note that I am just doing copy paste in the putty window, if you are not using putty window then you will have to type the commands by hand in the vmware console):

yum install oracle-rdbms-server-12cR1-preinstall -y


You will have the following output:

[root@san_rac_bbm yum.repos.d]# yum install oracle-rdbms-server-12cR1-preinstall -y
Loaded plugins: fastestmirror, security
base                                                                                                                                                                   | 3.7 kB     00:00
base/primary_db                                                                                                                                                        | 4.4 MB     00:03
extras                                                                                                                                                                 | 3.4 kB     00:00
extras/primary_db                                                                                                                                                      |  18 kB     00:00
ol6_UEK_latest                                                                                                                                                         | 1.2 kB     00:00
ol6_UEK_latest/primary                                                                                                                                                 | 8.0 MB     00:21
ol6_UEK_latest                                                                                                                                                                        183/183
ol6_latest                                                                                                                                                             | 1.4 kB     00:00
ol6_latest/primary                                                                                                                                                     |  30 MB     00:41
ol6_latest                                                                                                                                                                        21835/21835
updates                                                                                                                                                                | 3.4 kB     00:00
updates/primary_db                                                                                                                                                     | 3.8 MB     00:04
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package oracle-rdbms-server-12cR1-preinstall.x86_64 0:1.0-8.el6 will be installed
--> Processing Dependency: xorg-x11-utils for package: oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64
--> Processing Dependency: kernel-uek for package: oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64
--> Processing Dependency: ksh for package: oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64
--> Processing Dependency: libaio-devel for package: oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64
--> Processing Dependency: xorg-x11-xauth for package: oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64
--> Running transaction check
---> Package kernel-uek.x86_64 0:2.6.39-400.109.1.el6uek will be installed
--> Processing Dependency: kernel-uek-firmware = 2.6.39-400.109.1.el6uek for package: kernel-uek-2.6.39-400.109.1.el6uek.x86_64
---> Package ksh.x86_64 0:20100621-19.el6_4.4 will be installed
---> Package libaio-devel.x86_64 0:0.3.107-10.el6 will be installed
---> Package xorg-x11-utils.x86_64 0:7.5-6.el6 will be installed
--> Processing Dependency: libdmx.so.1()(64bit) for package: xorg-x11-utils-7.5-6.el6.x86_64
--> Processing Dependency: libXxf86misc.so.1()(64bit) for package: xorg-x11-utils-7.5-6.el6.x86_64
--> Processing Dependency: libXxf86dga.so.1()(64bit) for package: xorg-x11-utils-7.5-6.el6.x86_64
---> Package xorg-x11-xauth.x86_64 1:1.0.2-7.1.el6 will be installed
--> Processing Dependency: libXmuu.so.1()(64bit) for package: 1:xorg-x11-xauth-1.0.2-7.1.el6.x86_64
--> Running transaction check
---> Package kernel-uek-firmware.noarch 0:2.6.39-400.109.1.el6uek will be installed
---> Package libXmu.x86_64 0:1.1.1-2.el6 will be installed
---> Package libXxf86dga.x86_64 0:1.1.3-2.el6 will be installed
---> Package libXxf86misc.x86_64 0:1.0.3-4.el6 will be installed
---> Package libdmx.x86_64 0:1.1.2-2.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================
 Package                                                       Arch                            Version                                          Repository                               Size
==============================================================================================================================================================================================
Installing:
 oracle-rdbms-server-12cR1-preinstall                          x86_64                          1.0-8.el6                                        ol6_latest                               15 k
Installing for dependencies:
 kernel-uek                                                    x86_64                          2.6.39-400.109.1.el6uek                          ol6_UEK_latest                           27 M
 kernel-uek-firmware                                           noarch                          2.6.39-400.109.1.el6uek                          ol6_UEK_latest                          3.6 M
 ksh                                                           x86_64                          20100621-19.el6_4.4                              ol6_latest                              686 k
 libXmu                                                        x86_64                          1.1.1-2.el6                                      base                                     66 k
 libXxf86dga                                                   x86_64                          1.1.3-2.el6                                      base                                     25 k
 libXxf86misc                                                  x86_64                          1.0.3-4.el6                                      base                                     18 k
 libaio-devel                                                  x86_64                          0.3.107-10.el6                                   base                                     13 k
 libdmx                                                        x86_64                          1.1.2-2.el6                                      base                                     21 k
 xorg-x11-utils                                                x86_64                          7.5-6.el6                                        base                                     94 k
 xorg-x11-xauth                                                x86_64                          1:1.0.2-7.1.el6                                  base                                     35 k

Transaction Summary
==============================================================================================================================================================================================
Install      11 Package(s)

Total download size: 32 M
Installed size: 109 M
Downloading Packages:
(1/11): kernel-uek-2.6.39-400.109.1.el6uek.x86_64.rpm                                                                                                                  |  27 MB     00:47
(2/11): kernel-uek-firmware-2.6.39-400.109.1.el6uek.noarch.rpm                                                                                                         | 3.6 MB     00:03
(3/11): ksh-20100621-19.el6_4.4.x86_64.rpm                                                                                                                             | 686 kB     00:00
(4/11): libXmu-1.1.1-2.el6.x86_64.rpm                                                                                                                                  |  66 kB     00:00
(5/11): libXxf86dga-1.1.3-2.el6.x86_64.rpm                                                                                                                             |  25 kB     00:00
(6/11): libXxf86misc-1.0.3-4.el6.x86_64.rpm                                                                                                                            |  18 kB     00:00
(7/11): libaio-devel-0.3.107-10.el6.x86_64.rpm                                                                                                                         |  13 kB     00:00
(8/11): libdmx-1.1.2-2.el6.x86_64.rpm                                                                                                                                  |  21 kB     00:00
(9/11): oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64.rpm                                                                                                      |  15 kB     00:00
(10/11): xorg-x11-utils-7.5-6.el6.x86_64.rpm                                                                                                                           |  94 kB     00:00
(11/11): xorg-x11-xauth-1.0.2-7.1.el6.x86_64.rpm                                                                                                                       |  35 kB     00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                         606 kB/s |  32 MB     00:53
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
Importing GPG key 0xC105B9DE:
 Userid : CentOS-6 Key (CentOS 6 Official Signing Key)
 Package: centos-release-6-4.el6.centos.10.x86_64 (@anaconda-CentOS-201303020151.x86_64/6.4)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Retrieving key from http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6
Importing GPG key 0xEC551F03:
 Userid: "Oracle OSS group (Open Source Software group) "
 From  : http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : libXmu-1.1.1-2.el6.x86_64                                                                                                                                                 1/11
  Installing : 1:xorg-x11-xauth-1.0.2-7.1.el6.x86_64                                                                                                                                     2/11
  Installing : libdmx-1.1.2-2.el6.x86_64                                                                                                                                                 3/11
  Installing : ksh-20100621-19.el6_4.4.x86_64                                                                                                                                            4/11
  Installing : libXxf86dga-1.1.3-2.el6.x86_64                                                                                                                                            5/11
  Installing : libXxf86misc-1.0.3-4.el6.x86_64                                                                                                                                           6/11
  Installing : xorg-x11-utils-7.5-6.el6.x86_64                                                                                                                                           7/11
  Installing : kernel-uek-firmware-2.6.39-400.109.1.el6uek.noarch                                                                                                                        8/11
  Installing : kernel-uek-2.6.39-400.109.1.el6uek.x86_64                                                                                                                                 9/11
  Installing : libaio-devel-0.3.107-10.el6.x86_64                                                                                                                                       10/11
  Installing : oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64                                                                                                                    11/11
  Verifying  : xorg-x11-utils-7.5-6.el6.x86_64                                                                                                                                           1/11
  Verifying  : kernel-uek-2.6.39-400.109.1.el6uek.x86_64                                                                                                                                 2/11
  Verifying  : libaio-devel-0.3.107-10.el6.x86_64                                                                                                                                        3/11
  Verifying  : oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64                                                                                                                     4/11
  Verifying  : kernel-uek-firmware-2.6.39-400.109.1.el6uek.noarch                                                                                                                        5/11
  Verifying  : libXxf86misc-1.0.3-4.el6.x86_64                                                                                                                                           6/11
  Verifying  : libXxf86dga-1.1.3-2.el6.x86_64                                                                                                                                            7/11
  Verifying  : 1:xorg-x11-xauth-1.0.2-7.1.el6.x86_64                                                                                                                                     8/11
  Verifying  : ksh-20100621-19.el6_4.4.x86_64                                                                                                                                            9/11
  Verifying  : libdmx-1.1.2-2.el6.x86_64                                                                                                                                                10/11
  Verifying  : libXmu-1.1.1-2.el6.x86_64                                                                                                                                                11/11

Installed:
  oracle-rdbms-server-12cR1-preinstall.x86_64 0:1.0-8.el6

Dependency Installed:
  kernel-uek.x86_64 0:2.6.39-400.109.1.el6uek       kernel-uek-firmware.noarch 0:2.6.39-400.109.1.el6uek       ksh.x86_64 0:20100621-19.el6_4.4           libXmu.x86_64 0:1.1.1-2.el6
  libXxf86dga.x86_64 0:1.1.3-2.el6                  libXxf86misc.x86_64 0:1.0.3-4.el6                          libaio-devel.x86_64 0:0.3.107-10.el6       libdmx.x86_64 0:1.1.2-2.el6
  xorg-x11-utils.x86_64 0:7.5-6.el6                 xorg-x11-xauth.x86_64 1:1.0.2-7.1.el6

Complete!
[root@san_rac_bbm yum.repos.d]#


Now unix user “oracle”  has been created and initial set-up is also done.

Now let’s, do a full upgrade using the following command as root:

yum update  


Now let’s set the password of user oracle with the following command as root:

passwd oracle

You will have the following update:

[root@san_rac_bbm yum.repos.d]# passwd oracle
Changing password for user oracle.
New password:
BAD PASSWORD: it is based on a dictionary word
BAD PASSWORD: is too simple
Retype new password:
passwd: all authentication tokens updated successfully.
[root@san_rac_bbm yum.repos.d]#


Now, I will update the /etc/hosts file, add the following lines in the end of the file /etc/hosts:

# iSCSI SAN Server
10.10.1.116 san12c.rac.meditate.com san12c

# SCAN IP and Name
192.168.1.112   rac20-scan.rac.meditate.com rac20-scan
192.168.1.113   rac20-scan.rac.meditate.com rac20-scan
192.168.1.114   rac20-scan.rac.meditate.com rac20-scan

# This node primary address
192.168.1.117 rac20.rac.meditate.com rac20
# This node SAN address
10.10.1.117 rac20-san.rac.meditate.com rac20-san
# This node private interconnect address
10.10.3.117 rac20-priv.rac.meditate.com rac20-priv
# This node VIP address
192.168.1.122 rac20-vip.rac.meditate.com rac20-vip

# Repeat for second node in cluster
192.168.1.118 rac21.rac.meditate.com rac21
10.10.1.118 rac21-san.rac.meditate.com rac21-san
10.10.3.118 rac21-priv.rac.meditate.com rac21-priv
192.168.1.123 rac21-vip.rac.meditate.com rac21-vip

# Repeat for third node in cluster
192.168.1.119 rac22.rac.meditate.com rac22
10.10.1.119 rac22-san.rac.meditate.com rac22-san
10.10.3.119 rac22-priv.rac.meditate.com rac22-priv
192.168.1.124 rac22-vip.rac.meditate.com rac22-vip




Now the new /etc/hosts file looks like following :

 [root@san_rac_bbm yum.repos.d]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
# iSCSI SAN Server
10.10.1.116 san12c.rac.meditate.com san12c

# SCAN IP and Name
192.168.1.112   rac20-scan.rac.meditate.com rac20-scan
192.168.1.113   rac20-scan.rac.meditate.com rac20-scan
192.168.1.114   rac20-scan.rac.meditate.com rac20-scan

# This node primary address
192.168.1.117 rac20.rac.meditate.com rac20
# This node SAN address
10.10.1.117 rac20-san.rac.meditate.com rac20-san
# This node private interconnect address
10.10.3.117 rac20-priv.rac.meditate.com rac20-priv
# This node VIP address
192.168.1.122 rac20-vip.rac.meditate.com rac20-vip

# Repeat for second node in cluster
192.168.1.118 rac21.rac.meditate.com rac21
10.10.1.118 rac21-san.rac.meditate.com rac21-san
10.10.3.118 rac21-priv.rac.meditate.com rac21-priv
192.168.1.123 rac21-vip.rac.meditate.com rac21-vip

# Repeat for third node in cluster
192.168.1.119 rac22.rac.meditate.com rac22
10.10.1.119 rac22-san.rac.meditate.com rac22-san
10.10.3.119 rac22-priv.rac.meditate.com rac22-priv
192.168.1.124 rac22-vip.rac.meditate.com rac22-vip

[root@san_rac_bbm yum.repos.d]#


Now change file "/etc/security/limits.d/90-nproc.conf" as shown below.

Change this
*          soft    nproc    1024

To this
* - nproc 16384


Change the setting of SELINUX to disabled by editing the file "/etc/selinux/config",


SELINUX=disabled

Cat on this file looks as follows:

[root@san_rac_bbm yum.repos.d]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
#SELINUX=enforcing
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


[root@san_rac_bbm yum.repos.d]#



==================

Now lets disable the firewall using following commands as root:

service iptables stop
chkconfig iptables off

Here is the output of the commands:

[root@san_rac_bbm yum.repos.d]# service iptables stop
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[root@san_rac_bbm yum.repos.d]# chkconfig iptables off
[root@san_rac_bbm yum.repos.d]#

==================


Now let deconfigure the NTP using the following commands as root:

service ntpd stop
chkconfig ntpd off
mv /etc/ntp.conf /etc/ntp.conf.orig
rm /var/run/ntpd.pid

Following is outpuut of the commands:

[root@san_rac_bbm yum.repos.d]# service ntpd stop
Shutting down ntpd:                                        [FAILED]
[root@san_rac_bbm yum.repos.d]# chkconfig ntpd off
[root@san_rac_bbm yum.repos.d]# mv /etc/ntp.conf /etc/ntp.conf.orig
[root@san_rac_bbm yum.repos.d]# rm /var/run/ntpd.pid
rm: cannot remove `/var/run/ntpd.pid': No such file or directory
[root@san_rac_bbm yum.repos.d]#


===================

Now let’s install vmware tools on this machine as :

On vmware workstation window, do the following:
Click VM->Install VMware Tools


Now on putty window that we opened in the above step mount the cdrom as:
First create the cdrom directory with the following command as root:

mkdir /cdrom

Now execue the following command as root:

mount -t iso9660 -o ro /dev/cdrom /cdrom

Now shift to cdrom directory
cd /cdrom
and list the files in it using "ls -lart"

Here is the output of the above commands:

[root@san_rac_bbm ~]# mkdir /cdrom
[root@san_rac_bbm ~]# mount -t iso9660 -o ro /dev/cdrom /cdrom
[root@san_rac_bbm ~]# cd /cdrom
[root@san_rac_bbm cdrom]# ls -lart
total 64769
-r-xr-xr-x.  1 root root     1962 Aug  1  2012 manifest.txt
-r--r--r--.  1 root root 66314310 Aug  1  2012 VMwareTools-9.2.0-799703.tar.gz
dr-xr-xr-x.  2 root root     2048 Aug  1  2012 .
dr-xr-xr-x. 26 root root     4096 Jul 17 22:04 ..
[root@san_rac_bbm cdrom]#

So we have VMwareTools-9.2.0-799703.tar.gz, we will untar it to install VMware tools as follows:
shift to /tmp directory as:
cd /tmp

Unzip it as:
tar xf /cdrom/VMwareTools-9.2.0-799703.tar.gz
and now list the contents using "ls -lart" :

Here is the output of the above commands:

[root@san_rac_bbm cdrom]# ls -lart
total 64769
-r-xr-xr-x.  1 root root     1962 Aug  1  2012 manifest.txt
-r--r--r--.  1 root root 66314310 Aug  1  2012 VMwareTools-9.2.0-799703.tar.gz

dr-xr-xr-x. 26 root root     4096 Jul 17 22:04 ..
[root@san_rac_bbm cdrom]#
[root@san_rac_bbm cdrom]# cd /tmp
[root@san_rac_bbm tmp]# tar xf /cdrom/VMwareTools-9.2.0-799703.tar.gz
[root@san_rac_bbm tmp]# ls -lart
total 16
drwxr-xr-x.  7 root root 4096 Aug  1  2012 vmware-tools-distrib
-rw-------.  1 root root    0 Jul 14 23:17 yum.log
drwxrwxrwt.  2 root root 4096 Jul 17 18:50 .ICE-unix
dr-xr-xr-x. 26 root root 4096 Jul 17 22:04 ..
drwxrwxrwt.  4 root root 4096 Jul 17 22:10 .
[root@san_rac_bbm tmp]#


Now “cd vmware-tools-distrib” and execute “./vmware-install.pl” as root to install vmtools. When you execute “./vmware-install.pl” just accept all defaults by hiting “ENTER” key.

Here is the output of the above commands:

================

[root@san_rac_bbm tmp]# cd vmware-tools-distrib
[root@san_rac_bbm vmware-tools-distrib]# ls -lart
total 536
drwxr-xr-x. 15 root root   4096 Aug  1  2012 lib
-rwxr-xr-x.  1 root root 260711 Aug  1  2012 vmware-install.pl
drwxr-xr-x.  2 root root   4096 Aug  1  2012 installer
-rw-r--r--.  1 root root   2538 Aug  1  2012 INSTALL
-rw-r--r--.  1 root root 253897 Aug  1  2012 FILES
drwxr-xr-x.  4 root root   4096 Aug  1  2012 etc
drwxr-xr-x.  2 root root   4096 Aug  1  2012 doc
drwxr-xr-x.  2 root root   4096 Aug  1  2012 bin
drwxr-xr-x.  7 root root   4096 Aug  1  2012 .
drwxrwxrwt.  4 root root   4096 Jul 17 22:10 ..
[root@san_rac_bbm vmware-tools-distrib]# ./vmware-install.pl
Creating a new VMware Tools installer database using the tar4 format.

Installing VMware Tools.

In which directory do you want to install the binary files?
[/usr/bin]

What is the directory that contains the init directories (rc0.d/ to rc6.d/)?
[/etc/rc.d]

What is the directory that contains the init scripts?
[/etc/rc.d/init.d]

In which directory do you want to install the daemon files?
[/usr/sbin]

In which directory do you want to install the library files?
[/usr/lib/vmware-tools]

The path "/usr/lib/vmware-tools" does not exist currently. This program is
going to create it, including needed parent directories. Is this what you want?
[yes]

In which directory do you want to install the documentation files?
[/usr/share/doc/vmware-tools]

The path "/usr/share/doc/vmware-tools" does not exist currently. This program
is going to create it, including needed parent directories. Is this what you
want? [yes]

The installation of VMware Tools 9.2.0 build-799703 for Linux completed
successfully. You can decide to remove this software from your system at any
time by invoking the following command: "/usr/bin/vmware-uninstall-tools.pl".

Before running VMware Tools for the first time, you need to configure it by
invoking the following command: "/usr/bin/vmware-config-tools.pl". Do you want
this program to invoke the command for you now? [yes]

Initializing...


Making sure services for VMware Tools are stopped.

Stopping Thinprint services in the virtual machine:
   Stopping Virtual Printing daemon:                                   done
Stopping VMware Tools services in the virtual machine:
   Guest operating system daemon:                          [  OK  ]
   Unmounting HGFS shares:                                 [  OK  ]
   Guest filesystem driver:                                [  OK  ]


The VMware FileSystem Sync Driver (vmsync) allows external third-party backup
software that is integrated with vSphere to create backups of the virtual
machine. Do you wish to enable this feature? [no]

Found a compatible pre-built module for vmci.  Installing it...


Found a compatible pre-built module for vsock.  Installing it...


The module vmxnet3 has already been installed on this system by another
installer or package and will not be modified by this installer.  Use the flag
--clobber-kernel-modules=vmxnet3 to override.

The module pvscsi has already been installed on this system by another
installer or package and will not be modified by this installer.  Use the flag
--clobber-kernel-modules=pvscsi to override.

The module vmmemctl has already been installed on this system by another
installer or package and will not be modified by this installer.  Use the flag
--clobber-kernel-modules=vmmemctl to override.

The VMware Host-Guest Filesystem allows for shared folders between the host OS
and the guest OS in a Fusion or Workstation virtual environment.  Do you wish
to enable this feature? [yes]

Found a compatible pre-built module for vmhgfs.  Installing it...


Found a compatible pre-built module for vmxnet.  Installing it...


The vmblock enables dragging or copying files between host and guest in a
Fusion or Workstation virtual environment.  Do you wish to enable this feature?
[yes]

NOTICE:  It appears your system does not have the required fuse packages
installed.  The VMware blocking filesystem requires the fuse packages and its
libraries to function properly.  Please install the fuse or fuse-utils package
using your systems package management utility and re-run this script in order
to enable the VMware blocking filesystem.

!!! [EXPERIMENTAL] !!!
VMware automatic kernel modules enables automatic building and installation of
VMware kernel modules at boot that are not already present.  By selecting yes,
you will be enabling this experimental feature.  You can always disable this
feature by re-running vmware-config-tools.pl.

Would you like to enable VMware automatic kernel modules?
[no]

Thinprint provides driver-free printing. Do you wish to enable this feature?
[yes]

No X install found.

Creating a new initrd boot image for the kernel.
   Starting Virtual Printing daemon:                                   done
   Checking acpi hot plug                                  [  OK  ]
Starting VMware Tools services in the virtual machine:
   Switching to guest configuration:                       [  OK  ]
   VM communication interface:                             [  OK  ]
   VM communication interface socket family:               [  OK  ]
   Guest filesystem driver:                                [  OK  ]
   Mounting HGFS shares:                                   [FAILED]
   Guest operating system daemon:                          [  OK  ]
The configuration of VMware Tools 9.2.0 build-799703 for Linux for this running
kernel completed successfully.

You must restart your X session before any mouse or graphics changes take
effect.

You can now run VMware Tools by invoking "/usr/bin/vmware-toolbox-cmd" from the
command line.

To enable advanced X features (e.g., guest resolution fit, drag and drop, and
file and text copy/paste), you will need to do one (or more) of the following:
1. Manually start /usr/bin/vmware-user
2. Log out and log back into your desktop session; and,
3. Restart your X session.

Enjoy,

--the VMware team

Found VMware Tools CDROM mounted at /cdrom. Ejecting device /dev/sr0 ...
/sbin/restorecon:  Warning no default label for /tmp/vmware-block-restore0/tmp_file
[root@san_rac_bbm vmware-tools-distrib]#


================

Download and install Xming from:

http://sourceforge.net/projects/xming/









Uncheck “Launch Xming” as we will need to install its fonts as well.




Now start “Xming” from Start à Programs

===================


Now execute the following commands as root to synchronise time of this unix machine with host windows machine.

cd /tmp/vmware-tools-distrib
vmware-toolbox-cmd timesync status
vmware-toolbox-cmd timesync enable

[root@san_rac_bbm vmware-tools-distrib]# cd /tmp/vmware-tools-distrib
[root@san_rac_bbm vmware-tools-distrib]# vmware-toolbox-cmd timesync status
Disabled
[root@san_rac_bbm vmware-tools-distrib]# vmware-toolbox-cmd timesync enable
Enabled
[root@san_rac_bbm vmware-tools-distrib]# vmware-toolbox-cmd timesync status
Enabled
[root@san_rac_bbm vmware-tools-distrib]#


You can confirm by seeing in the file guest machine configuration file should have following entry now:

tools.syncTime = "TRUE"



===================


And now let’s install “xclock” as :


yum install xorg-x11-apps

Here is the output:

[root@san_rac_bbm vmware-tools-distrib]# yum install xorg-x11-apps
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: centos.mirror.freedomvoice.com
 * extras: mirrordenver.fdcservers.net
 * updates: mirror.thelinuxfix.com
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package xorg-x11-apps.x86_64 0:7.6-6.el6 will be installed
--> Processing Dependency: libxkbfile.so.1()(64bit) for package: xorg-x11-apps-7.6-6.el6.x86_64
--> Processing Dependency: libXaw.so.7()(64bit) for package: xorg-x11-apps-7.6-6.el6.x86_64
--> Running transaction check
---> Package libXaw.x86_64 0:1.0.11-2.el6 will be installed
--> Processing Dependency: libXpm.so.4()(64bit) for package: libXaw-1.0.11-2.el6.x86_64
---> Package libxkbfile.x86_64 0:1.0.6-1.1.el6 will be installed
--> Running transaction check
---> Package libXpm.x86_64 0:3.5.10-2.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================================================================================
 Package                                                      Arch                                                  Version                                                        Repository                                           Size
=============================================================================================================================================================================================================================================
Installing:
 xorg-x11-apps                                                x86_64                                                7.6-6.el6                                                      base                                                276 k
Installing for dependencies:
 libXaw                                                       x86_64                                                1.0.11-2.el6                                                   base                                                178 k
 libXpm                                                       x86_64                                                3.5.10-2.el6                                                   base                                                 51 k
 libxkbfile                                                   x86_64                                                1.0.6-1.1.el6                                                  base                                                 74 k

Transaction Summary
=============================================================================================================================================================================================================================================
Install       4 Package(s)

Total download size: 579 k
Installed size: 1.4 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): libXaw-1.0.11-2.el6.x86_64.rpm                                                                                                                                                                                 | 178 kB     00:00
(2/4): libXpm-3.5.10-2.el6.x86_64.rpm                                                                                                                                                                                 |  51 kB     00:00
(3/4): libxkbfile-1.0.6-1.1.el6.x86_64.rpm                                                                                                                                                                            |  74 kB     00:00
(4/4): xorg-x11-apps-7.6-6.el6.x86_64.rpm                                                                                                                                                                             | 276 kB     00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                        894 kB/s | 579 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : libxkbfile-1.0.6-1.1.el6.x86_64                                                                                                                                                                                           1/4
  Installing : libXpm-3.5.10-2.el6.x86_64                                                                                                                                                                                                2/4
  Installing : libXaw-1.0.11-2.el6.x86_64                                                                                                                                                                                                3/4
  Installing : xorg-x11-apps-7.6-6.el6.x86_64                                                                                                                                                                                            4/4
  Verifying  : libXpm-3.5.10-2.el6.x86_64                                                                                                                                                                                                1/4
  Verifying  : libXaw-1.0.11-2.el6.x86_64                                                                                                                                                                                                2/4
  Verifying  : xorg-x11-apps-7.6-6.el6.x86_64                                                                                                                                                                                            3/4
  Verifying  : libxkbfile-1.0.6-1.1.el6.x86_64                                                                                                                                                                                           4/4

Installed:
  xorg-x11-apps.x86_64 0:7.6-6.el6

Dependency Installed:
  libXaw.x86_64 0:1.0.11-2.el6                                                 libXpm.x86_64 0:3.5.10-2.el6                                                 libxkbfile.x86_64 0:1.0.6-1.1.el6

Complete!
[root@san_rac_bbm vmware-tools-distrib]#

==============


Now check the time and make sure X windows work, when you will execute the command “xclock” it should open the display and if doesn’t open the display, you need to check some previous steps and fix it.

[root@san_rac_bbm ~]#
[root@san_rac_bbm ~]# xclock
Warning: Missing charsets in String to FontSet conversion



========================



To setup SCAN IP for the cluster, for that we will install  “dnsmasq”

To install it use the following command as root:

yum install dnsmasq

Here is the output of the command.

[root@san_rac_bbm yum.repos.d]# yum install dnsmasq
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: centos.mirror.freedomvoice.com
 * extras: mirrordenver.fdcservers.net
 * updates: mirror.thelinuxfix.com
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package dnsmasq.x86_64 0:2.48-13.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================
 Package                                      Arch                                        Version                                             Repository                                 Size
==============================================================================================================================================================================================
Installing:
 dnsmasq                                      x86_64                                      2.48-13.el6                                         base                                      149 k

Transaction Summary
==============================================================================================================================================================================================
Install       1 Package(s)

Total download size: 149 k
Installed size: 293 k
Is this ok [y/N]: y
Downloading Packages:
dnsmasq-2.48-13.el6.x86_64.rpm                                                                                                                                         | 149 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : dnsmasq-2.48-13.el6.x86_64                                                                                                                                                 1/1
  Verifying  : dnsmasq-2.48-13.el6.x86_64                                                                                                                                                 1/1

Installed:
  dnsmasq.x86_64 0:2.48-13.el6

Complete!
[root@san_rac_bbm yum.repos.d]#



Turn on the Dnsmasq server and make sure it starts automatically on reboot execute following as root :

service dnsmasq start
chkconfig dnsmasq on

Output is as follows:

[root@san_rac_bbm vmware-tools-distrib]# service dnsmasq start
dnsdomainname: Unknown host
Starting dnsmasq:                                          [  OK  ]
[root@san_rac_bbm vmware-tools-distrib]# chkconfig dnsmasq on
[root@san_rac_bbm vmware-tools-distrib]#


To stop, start and restart dnsmasq use the following commands:
service dnsmasq stop
service dnsmasq start
service dnsmasq restart

Also for this to work we need to have firewall disabled which we already did earlier with following command and if we have not done it execute it now:
service iptables stop


Make sure /etc/hosts looks like above.

In file "/etc/resolv.conf" update ip infollowing line to the ip of the local machine:

i.e. update following line :

from:
nameserver 192.168.1.254

to:
nameserver 192.168.1.115

MAke sure "/etc/resolv.conf" looks as follows:

[root@san_rac_bbm vmware-tools-distrib]# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 192.168.1.115
[root@san_rac_bbm vmware-tools-distrib]#

Now do nslookup and it will pick scan ip's  as shown below:

[root@san_rac_bbm vmware-tools-distrib]# nslookup rac20-scan
Server:  192.168.1.115
Address: 192.168.1.115#53

Name: rac20-scan
Address: 192.168.1.114
Name: rac20-scan
Address: 192.168.1.112
Name: rac20-scan
Address: 192.168.1.113

[root@san_rac_bbm vmware-tools-distrib]#


Now shutdown the machine using command “init 0” as root.

On windows copy folder “san_rac_bbm” under directory  “C:\VM\rac_12c”  in same directory and name it as “san1” . I am naming it as “san1” as while building “11gR2” cluster I named san machine as “san”.
Now open bbm file with extention “Vmware virtual machine configuration file”  under directory “C:\VM\rac_12c\san1” in editor and update following line :
From:
scsi0:0.fileName = "C:\VM\rac_12c\bbm\bbm.vmdk"

to:
scsi0:0.fileName = "C:\VM\rac_12c\san1\bbm.vmdk"

and as mentioned earlier for machine “san_rac_bbm” , open the machine in console by opening file “C:\VM\rac_12c\san1\bbm.vmdk” from vmware window “File à Open” . I am not placing screenshots of this as we have already done similar thing earlier.  Then click “edit virtual machine settings” and then click “options” and here under “virtual machine name”  type “san1” and click “OK”. Now power on this virtual machine and click “I copied it”.

In console login as root and update following line in file “/etc/sysconfig/network”

HOSTNAME=san_rac_bbm.rac.meditate.com

To

HOSTNAME=san1.rac.meditate.com


Now vi file
/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network-scripts/ifcfg-eth1
/etc/sysconfig/network-scripts/ifcfg-eth2
/etc/sysconfig/network-scripts/ifcfg-eth3

And update ip from “192.168.1.115” to “192.168.1.116”

And now remove file “/etc/udev/rules.d/70-persistent-net.rules” using following command as root:
rm –f /etc/udev/rules.d/70-persistent-net.rules

Now reboot the machine using command “reboot” as root.

Now login through putty and use is 192.168.1.116 to connect to it.


=======================
Now repeat the same above steps to create machine rac20. i.e.

On windows copy folder “san_rac_bbm” under directory  “C:\VM\rac_12c”  in same directory and name it as “rac20” .
Now open bbm file with extention “Vmware virtual machine configuration file”  under directory “C:\VM\rac_12c\rac20” in editor and update following line :
From:
scsi0:0.fileName = "C:\VM\rac_12c\bbm\bbm.vmdk"

to:
scsi0:0.fileName = "C:\VM\rac_12c\rac20\bbm.vmdk"

and as mentioned earlier for machine “san_rac_bbm” , open the machine in console by opening file “C:\VM\rac_12c\rac20\bbm.vmdk” from vmware window “File à Open” . I am not placing screenshots of this as we have already done similar thing earlier.  Then click “edit virtual machine settings” and then click “options” and here under “virtual machine name”  type “rac20” and click “OK”. Now power on this virtual machine and click “I copied it”.

In console login as root and update following line in file “/etc/sysconfig/network”

HOSTNAME=san_rac_bbm.rac.meditate.com

To

HOSTNAME=rac20.rac.meditate.com


Now vi file
/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network-scripts/ifcfg-eth1
/etc/sysconfig/network-scripts/ifcfg-eth2
/etc/sysconfig/network-scripts/ifcfg-eth3

And update ip from “192.168.1.115” to “192.168.1.117”

And now remove file “/etc/udev/rules.d/70-persistent-net.rules” using following command as root:
rm –f /etc/udev/rules.d/70-persistent-net.rules

Now reboot the machine using command “reboot” as root.

Now login through putty and use is 192.168.1.117 to connect to it.

Now in next post we will complete the setup of san1 and rac20 machine.




As thirst is quenched with water, and the baby is satisfied with mother’s milk, and as the lotus does not exist without water, and as the fish dies without water—O Nanak, so does the Gurmukh live, receiving the Sublime Essence of the Lord, and singing the Glorious Praises of the Lord.






No comments:

Post a Comment