we are looking at starting virtualizing our Oracle 11g systems. Some systems are RAC and some are standalone. Currently we use ASM with ASMLib to manage disk. I am trying to get some feedback on what other folks are doing in terms of their disk strategy. Here are my questions :
1) do you dedicate datastores for data files and separate datastore for log files ?
2) do you use paravirtualized disk ?
3) do you simply present a big vmdk file and then fdisk it and "oracleasm createdisk" or do you use LVM and then "oracleasm createdisk"
4) what other questions should i be asking ?
I've found some random papers online but nothing specific.
Although some crosstalk is inevitable, I should point out that a more appropriate forum for this question would be the Everything Oracle at EMC community, of which Sam Lucido and I are co-administrators. Having said that, when we become aware of a question on another forum such as this one, we are happy to respond.
I would say that the answer by Alan Robertson, above, is very helpful. As Alan points out, in an ASM context there are a couple of high-level architectural decisions:
Do you use RDM? VMware would say no, but so far all EMC Proven Solutions implementations have used RDM for block storage. RDM is required for several use cases:
Where you are using a physically booted database for source data and cloning that to be mounted on VMware
Where you need to be able to move the database transparently between physically booted hardware and virtualized hardware
Where you want a RAC environment with a mixture of physically booted and virtualized nodes
Do you use VMFS? With the multiwriter flag, it is now possible to use VMFS with RAC. There are a few issues here:
You can have no more than 8 nodes.
You cannot have physically-booted nodes.
You have another option as well: NFS using Direct NFS. You indicated that you want to use ASM, so I will not cover that further here.
All of these options, as well as some detailed best practices and recommendations regarding storage layout can be found in the Virtualizing Oracle 11g RAC on vSphere technical session that Sam and I co-presented at VMworld 2011. Feel free to take a look at this presentation, and please let Sam or me know if you have any questions on the content.
We are not sure customers have the DataGuard protection in their environment. If so, RAC->RAC and standalone->standalone failover should be done respectively. We took care of the similar cases before.
The answer for your #1 question is ture. General speaking, different datastore can satisfy different performance requirement of tiers.
I am fully understand your #2 question paravirtulized disks mean.
The #3 question gives me the concept that fdisk or VLM will bring more workload on the OS layer. Why not directly provide more VMDK for the different partitions.
Dynamo, I have some thoughts like to share with your. My thoughts is following your questions starting with “A:”
1) do you dedicate datastores for data files and separate datastore for log files ?
A: I suggest to leverage ESM separated store to hold Oracle data files, because Oracle instance has various behaviors on those file, that means IO profile is different. General speaking, Oracle Data file like R5 or R 6 FC/SAS, Redo file like R1/R10 FC/SAS, Archived logs like R6 NL_SAS. If you would use F_Cache to improve Ora performance, you would have a look at Oracle database on EMC VNX Best Practices http://powerlink.emc.com/km/live1/en_US/Offering_Technical/White_Paper/h8242-deploying-oracle-vnx-wp.pdf
2) do you use paravirtualized disk ?
A: This depend your favor on Oracle instance mobility or performance, paravirtualized disk / RDM would stick Ora instance on a Physical ESX server, and the performance with using paravirtualized disk / RDM would roughly 10% Oracle Performance improvement, if you have deployed F_Cache, the difference is small.
3) do you simply present a big vmdk file and then fdisk it and "oracleasm createdisk" or do you use LVM and then "oracleasm createdisk"
A: I prefer “a big vmdk file”, because VMFS would cross all spindles connected with ESX servers and co-exist of VMFS and ASM is a bit duplicated.
1) we only have one pool on VNX with EFD and SAS drives in raid-5 config. I am not going to put NL_SAS in the same pool until EMC allows pool configuration with different raid types (i am not using anythign but R6 for NL-SAS)
2) what do you mean mobility ? Are you saying i can't VMotion/sVMotion paravirtual controllers ? News to me
3) the question is what you do with vmdk devices presented to the Oracle box, do you fdisk that vmdk file into multiple partitions or do you use lvm. We are going to use ASM regardless.
Please have a look at my thoughts following your questions.
1) we only have one pool on VNX with EFD and SAS drives in raid-5 config. I am not going to put NL_SAS in the same pool until EMC allows pool configuration with different raid types (i am not using anythign but R6 for NL-SAS)
A: according to the whitepaper I referred to you, R5 SAS pool is configured for Oracle data flat files with F_Cahce, EFD enabled for better performance, R1 2+2 SAS Pool is configured for Oracle online redo log files, NL_SAS is considered for Archived log files.
I suggest Dev from every VNX pool are mapped with separated ASM DG, I,e Dev of R5 SAS pool with F_Cahce go to ASM DG for data flat files, etc.
2) what do you mean mobility ? Are you saying i can't VMotion/sVMotion paravirtual controllers ? News to me
A: The processor is paravirtualized within VMware (that's why you can't vmotion to too different processors) the drivers are not. If you use paravirtualized drivers (for SCSI, NIC, etc) you would be fine.
3) the question is what you do with vmdk devices presented to the Oracle box, do you fdisk that vmdk file into multiple partitions or do you use lvm. We are going to use ASM regardless.
ASM manage the dev presented to VM guest, single vmdk is a SCSI dev in VM guest, therefor from preventive, no fdisk action is required, but so far I don’t have any whitepaper talking about best practices of ASM on VMFS.
we are looking at starting virtualizing our Oracle 11g systems. Some systems are RAC and some are standalone. Currently we use ASM with ASMLib to manage disk. I am trying to get some feedback on what other folks are doing in terms of their disk strategy. Here are my questions :
I work in the team that builds, validates and produces many of EMC's Oracle the white papers. I have put my answers and links in line.
1) do you dedicate datastores for data files and separate datastore for log files ?
As far as I know, EMC do not have an official position on single or multiple datastores. If you are using vmdks on VMFS then I would recommend separate data stores.
3) do you simply present a big vmdk file and then fdisk it and "oracleasm createdisk" or do you use LVM and then "oracleasm createdisk"
If you are using ASM then you want multiple LUNs in your disk group for ASM to balance IO and prevent lun queuing on the guest. As above ASM is an volume manager so I would use fdisk with a 64KB offset on your partition. If you want to grow your disk group add another LUN.
4) what other questions should i be asking ?
Not an exhaustive list but some things I think you should consider
What size of cluster is being used, cpu/core count? Oracle Licensing
RDM or VMFS:Effects on Backup and Replication.
Migration to/from physical: downtime and mechanisms.
Build your VM template from a fresh Guest install or VMconverter.
Currently, I have only tested RAC with RDM (no vMotion due to SCSI bus sharing must set to physical) and NFS. However, if you want to use Oracle RAC and vMotion/DRS then you can now use vmdks on VMFS thanks to the multi-writer flag that enables sharing at the disk level, see vmware KB 1034165 and probably a future EMC whitepaper on the topic
thanks Eddy. I don't feel right about having one to one relationship between vmdk and ASM disk. If my standard is to present 100G vmdk file to the host, i would prefer to partition it so where would be more queues in the OS and not just one big 100G device. You know what i mean ?
jeff_browning
2 Intern
•
256 Posts
2559
0
Posted October 27th, 2011 12:00
Although some crosstalk is inevitable, I should point out that a more appropriate forum for this question would be the Everything Oracle at EMC community, of which Sam Lucido and I are co-administrators. Having said that, when we become aware of a question on another forum such as this one, we are happy to respond.
I would say that the answer by Alan Robertson, above, is very helpful. As Alan points out, in an ASM context there are a couple of high-level architectural decisions:
You have another option as well: NFS using Direct NFS. You indicated that you want to use ASM, so I will not cover that further here.
All of these options, as well as some detailed best practices and recommendations regarding storage layout can be found in the Virtualizing Oracle 11g RAC on vSphere technical session that Sam and I co-presented at VMworld 2011. Feel free to take a look at this presentation, and please let Sam or me know if you have any questions on the content.