Tuesday, June 29, 2010

Install Oracle RAC 11gR2 on Vmware with Windows 7 64-bit as host OS and Linux as guest OS (Part 3, SAN Setup)

He is the Creator and is doing everything


Index of all the posts of Gurpartap Singh's Blog

Here are the steps to create the SAN machine.

After we copied the machine name that folder as SAN. Now open that machine with VMware workstation.



Here screen shot while opening the machine you need to browse to the location. Pick CentOS_5_BASE (Vmware file)



After opening it, click edit this machine.



click options and here change the name to "san"



Reduce the memory to 128MB and start the machine.



click I copied it on the following POP up.



In the following screen shot you can see the machine is rebooting.



After the machine reboots, login as root and execute the following command to see the ip address of the machine. in my case it picked 192.168.1.106



Login as root to this machine from putty. Putty is a small executable file that can be dowloaded easily and instead of working on machine directly, I will work with putty as its easy to copy paste stuff with that. Execute as root on putty window :
system-config-network



Hit edit devices and pick the first one as eth0 and hit enter to edit it.


Enter the following IP addresses for this machine.
1. Unselect Use DHCP
2. Static IP as 192.168.1.175
3. Netmask 255.255.255.0
4. Default Gateway 192.168.1.1
and then clcik ok.


Next select eth1 to edit i.e third row and enter the following ips.
1. Unselect Use DHCP.
2. Static IP 10.10.1.175
3. Netmask 255.255.255.0
Leave default gateway IP as empty. and click ok.


Now Edit DNS configuration

Here you will see that the hostname is already updated to "san.rac.meditate.com"
If not then update it. Just enter Primary DNS as 192.168.1.1
and click ok



And then on next screen click save and quit.

Now from m/c as root execute:
#service network restart

This will start the network and will make the ip of our machine as as 192.168.1.175

The old putty window will loose the connection so now open the new window and login as root and you should be able to ping oracle as follows:

login as: root
root@192.168.1.175's password:
Last login: Mon Jun 28 20:55:07 2010 from 192.168.1.103
[root@san ~]# ping www.oracle.com
PING a398.g.akamai.net (206.108.207.137) 56(84) bytes of data.
64 bytes from 206.108.207.137: icmp_seq=1 ttl=61 time=17.2 ms
64 bytes from 206.108.207.137: icmp_seq=2 ttl=61 time=16.2 ms

--- a398.g.akamai.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 16.219/16.726/17.233/0.507 ms


Edit /etc/hosts and it should look like:

[root@san ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 base.rac.meditate.com base localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
# SAN
192.168.1.175 san.rac.meditate.com san

Now as root, restart the machine:
shutdown -h now

and then login as root from putty and see all the network settings are ok as:

[root@san ~]# hostname
san.rac.meditate.com
[root@san ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 base.rac.meditate.com base localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
# SAN
192.168.1.175 san.rac.meditate.com san

[root@san ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:B2:8E:A6
inet addr:192.168.1.175 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feb2:8ea6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:210 errors:0 dropped:0 overruns:0 frame:0
TX packets:210 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:20520 (20.0 KiB) TX bytes:26784 (26.1 KiB)
Base address:0x2000 Memory:d8940000-d8960000

eth1 Link encap:Ethernet HWaddr 00:0C:29:B2:8E:B0
inet addr:10.10.1.175 Bcast:10.10.1.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feb2:8eb0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:46 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:7687 (7.5 KiB)
Base address:0x2080 Memory:d8960000-d8980000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:560 (560.0 b) TX bytes:560 (560.0 b)

[root@san ~]#




Install the iSCSI package and answer 'y' to the prompts.
Run the following command as root from the putty window as root:

yum install scsi-target-utils



Create a directory for the disks
mkdir /disks

Create a set of 24 disk files each 2Gb in size

for i in 1 to 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1117 18 19 20 21 22 23 24
do
dd if=/dev/zero of=/disks/disk$i.dat bs=1M count=2048
done



[root@san ~]# mkdir /disks
[root@san ~]# for i in 1 to 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
> do
> dd if=/dev/zero of=/disks/disk$i.dat bs=1M count=2048
> done

2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 4.66889 seconds, 460 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 7.42677 seconds, 289 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 4.88252 seconds, 440 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 7.27913 seconds, 295 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 6.02063 seconds, 357 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 5.77786 seconds, 372 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 5.81557 seconds, 369 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 6.69618 seconds, 321 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 6.01836 seconds, 357 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 5.76424 seconds, 373 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 5.98724 seconds, 359 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 6.99209 seconds, 307 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 5.88205 seconds, 365 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 5.93012 seconds, 362 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 5.73615 seconds, 374 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 5.65302 seconds, 380 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 7.10604 seconds, 302 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 6.1137 seconds, 351 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 5.80223 seconds, 370 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 5.94321 seconds, 361 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 7.27233 seconds, 295 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 5.9688 seconds, 360 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 6.6014 seconds, 325 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 6.27009 seconds, 342 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 6.1728 seconds, 348 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 6.04302 seconds, 355 MB/s
[root@san ~]#



Set the iscsi target demon to start automatically
chkconfig --level 345 tgtd on



Edit the following file:
/etc/rc.d/rc.local

So it has the following entries:

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# Create a target
tgtadm --lld iscsi --op new --mode target --tid 1 -T iqm.mgmt.volumes-san

# Create LUNs within the target
for i in 1 2 3 4 5 6 7 8 9 10 11 12
do
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun $i -b /disks/disk$i.dat
done

# Expose target to all
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL



Bounce the VM again with "reboot"



Verify target is ready and the LUNs are present:
tgtadm --lld iscsi --op show --mode target

You should see something like the following:

[root@san ~]# tgtadm --lld iscsi --op show --mode target

Target 1: iqm.mgmt.volumes-san
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: None
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk1.dat
LUN: 2
Type: disk
SCSI ID: IET 00010002
SCSI SN: beaf12
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk2.dat
.................
.................
...............
.................
LUN: 11
Type: disk
SCSI ID: IET 0001000b
SCSI SN: beaf111
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk11.dat
LUN: 12
Type: disk
SCSI ID: IET 0001000c
SCSI SN: beaf112
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk12.dat
Account information:
ACL information:
[root@san ~]#



Shut down the VM with "shutdown -h now"

Now we have configured our SAN and it has 12 luns of 2GB each and can be used by RAC.
Next we will configure RAC machines. We can add other 12 luns later if required.

Remember:
Where there is no mother, father, friends or siblings, there is only the Naam, the Name of the Lord, shall be with you as your help and support. So try to do as much meditaion in the rememberance of God as much you can.

11 comments:

  1. Mr Singh

    Is there a chance you would share in vmware image!
    I am a developer and dont want to go through the whole process of creating a vmware centos db to play around. Your help would be appreciated.

    Thanks

    ReplyDelete
  2. Hi Madhu,

    If the image would have been 2-3GB, I could have shared it. But the image is huge, I cannot do that.

    Good Luck.

    ReplyDelete
  3. i want ask someting,
    After reboot VM, there's problem when i check with command :
    tgtadm --lld iscsi --op show --mode target. Before that everything's fine.

    the error is :
    tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected

    Can u help me please....

    Jimmy Rekso,
    jimmy.rekso@berca.co.id

    ReplyDelete
  4. Hi Jimmy:

    Please check,

    1. If your ips are ok after the reboot with ifconfig command as root.

    2. Then check had you done the following:
    You can do it again.

    Set the iscsi target demon to start automatically
    chkconfig --level 345 tgtd on

    3. Also check if this file was properly updated. Is there is any mistake in this file?

    /etc/rc.d/rc.local

    Good Luck........and have fun.

    ReplyDelete
  5. Hi - I getting the following error tgtadm: invalid request when I run tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 -b /disks/disk1.dat

    I check the /var/log/messages and it has the following

    Oct 13 10:15:23 san tgtd: backed_file_open(92) Could not open /disks/disk1.dat, Permission denied.

    I change the permission to
    -rwxrw-r--. 1 root root 2147483648 Oct 12 09:39 disk10.dat

    and I still having he problem.

    I using oracle linux 64bit I didn't have this problem when I use oscent.

    ReplyDelete
  6. Hi Frank,

    Please use the following permissions:

    brw-r----- 1 oracle oinstall 65, 128 Oct 4 22:08 sdy

    This is what I have :

    brw-r----- 1 oracle oinstall 65, 160 Oct 4 22:08 sdaa
    brw-r----- 1 oracle oinstall 65, 176 Oct 4 22:08 sdab
    brw-r----- 1 oracle oinstall 65, 192 Oct 4 22:08 sdac
    brw-r----- 1 oracle oinstall 65, 208 Oct 4 22:08 sdad
    brw-r----- 1 oracle oinstall 65, 224 Oct 4 22:08 sdae
    brw-r----- 1 oracle oinstall 8, 16 Oct 4 22:08 sdb
    brw-r----- 1 oracle oinstall 8, 32 Oct 4 22:08 sdc
    brw-r----- 1 oracle oinstall 8, 48 Oct 14 07:42 sdd
    brw-r----- 1 oracle oinstall 8, 64 Oct 13 17:06 sde
    brw-r----- 1 oracle oinstall 8, 80 Oct 13 17:06 sdf
    brw-r----- 1 oracle oinstall 8, 96 Oct 4 22:08 sdg
    brw-r----- 1 oracle oinstall 8, 112 Oct 4 22:08 sdh
    brw-r----- 1 oracle oinstall 8, 128 Oct 4 22:08 sdi
    brw-r----- 1 oracle oinstall 8, 144 Oct 4 22:08 sdj
    brw-r----- 1 oracle oinstall 8, 160 Oct 4 22:08 sdk
    brw-r----- 1 oracle oinstall 8, 176 Oct 4 22:08 sdl
    brw-r----- 1 oracle oinstall 8, 192 Oct 4 22:08 sdm
    brw-r----- 1 oracle oinstall 8, 208 Oct 4 22:08 sdn
    brw-r----- 1 oracle oinstall 8, 224 Oct 4 22:08 sdo
    brw-r----- 1 oracle oinstall 8, 240 Oct 4 22:08 sdp
    brw-r----- 1 oracle oinstall 65, 0 Oct 14 07:42 sdq
    brw-r----- 1 oracle oinstall 65, 16 Oct 4 22:08 sdr
    brw-r----- 1 oracle oinstall 65, 32 Oct 14 07:42 sds
    brw-r----- 1 oracle oinstall 65, 48 Oct 14 07:42 sdt
    brw-r----- 1 oracle oinstall 65, 64 Oct 14 07:42 sdu
    brw-r----- 1 oracle oinstall 65, 80 Oct 14 07:42 sdv
    brw-r----- 1 oracle oinstall 65, 96 Oct 14 07:42 sdw
    brw-r----- 1 oracle oinstall 65, 112 Oct 13 17:06 sdx
    brw-r----- 1 oracle oinstall 65, 128 Oct 4 22:08 sdy
    brw-r----- 1 oracle oinstall 65, 144 Oct 4 22:08 sdz

    ReplyDelete
  7. Hi Mr. Singh,
    1. For some reasons I did not see eth1 listed in my network config area.
    2. I cannot connect to internet from my SAN machine.

    what am I missing?

    ReplyDelete
  8. Thanks for visitng and following the blog,
    Please check you eth0 settings, you should be able to ping outside world from your machine.

    Please redo teh steps you must have missed something.

    As root execute:
    service-config-network

    and go to edit dns configuration and check if primary dns and search entries are ok.

    Good luck with your install !

    Regards
    Gurpartap Singh

    ReplyDelete
  9. Hi Gurpartap,

    I am able to configure SAN successfully. Thanks for the crisp note. However, when i try to create an ASM disk on the RAC Nodes as Oracle USER, it is failing to mark the DISK as ASM. However, As ROOT i am able to create the ASM disk using oracleasm command. Do i need to set anything else ? am i missing something?

    ReplyDelete
    Replies
    1. My OS is RHEL5/RHEL6. I have tried on both but none of them works.

      Delete
  10. Hi Gurpartap, Thank You very very much for this documents.

    I am not able to see/list the target ( tgtadm --lld iscsi --op show --mode target ) after reboot. I am able to list the Target before i reboot my SAN VM.

    May i know where can i troubleshot this part ?

    Thanks
    Ryan

    ReplyDelete