Thanks Andre for your comment.

//Magnus

Hi Rodney,
thanks for your reply. I agree that three of my four suggestions does not actually secure VM priority to a specific HBA. As i mention in the blog post my suggestions are ways to work around the fact that there is no ESXi configuration available to do what my customer asked me to do. That’s why i have been throwing hardware to the design to solve the problems in some of the scenarios. Ultimately it will be up to my customer to select the solution they want and personally i have not seen the physical HBAs as the bottlenecks in a very long time.

My point is not to base a design around HBAs but i think you need to take every piece involved in the virtual infrastructure into account when creating a design. Specifically when we are virtualizing more and more critical systems and even the business critical once. If we don’t take all components into account and start making assumptions we better put those into the design and be ready to change it when necessary.
As far as i understand the disk.schednumreqoutstanding configuration are disabled when using Storage I/O Control

Again, thanks for your comment

//Magnus

If I understand your problem statement correctly, you must:

Prioritise access to a limited/shared resource (HBA) during times of contention for VMs based on a datastore layout.

This can be broken down into 3 assertions,

1. Prioritise access to a limited/shared resource (HBA)…
2. …during times of contention for VMs…
3. …based on a datastore layout,

In this case, you are correct, there are no options within the vSphere storage stack to do this with the layout above.

Some of your suggestions, however, do not meet your design assertions, specifically:

a. Place the most critical virtual machines on its own vSphere cluster. Depending on the importance e.g. mission critical workload, this might be a good choice.

This dedicates HBA resource to particular workloads and ultimately does not meet the 1st assertion as you are removing contention by throwing more hardware into the equation.

b. Add HBAs (4 ports in total) per ESXi host and make sure the SLA1 based datastores uses 2 HBA ports and the other datastores uses the 2 other HBA ports.

Again this invalidates the 1st assertion by adding HW and removing contention.

c. Do not separate the virtual machines on different datastores based on SLA/importance. This requires some intelligence, perhaps auto-tiering, on the storage array level to guarantee correct virtual machine performance at the storage level.

This is the only way to achieve the brief (if you are strictly adhering to the problem statement and assumptions above). And only if you establish a single large datastore (eg up to 64TB) and utilise SIOC to prioritise access to the datastore using VM storage shares. (note: introducing an assumption that vSphere 5x and VMFS5 is used)

In this way you meet the objective as you use a limited HBA resource appropriately during times of no contention and prioritise access during contention.

The datastore component of the design is aligned with the HBA component as the current controls available in the vSphere storage stack only operate at the datastore level. This will also involve fine tuning the storage device queue (disk.schednumreqoutstanding) within the hypervisors and the HBA maxqdepth to accommodate the concentration of IO to a single storage element and the lack of competing physical storage devices for HBA resources.

All in all it would be rather frivolous to base the design around HBA focused prioritisation as it leads to operational overheads that would be burdensome without much advantage. A modern 8GB or 16GB FC HBA has plenty of capacity for both throughput and bandwidth. Another outcome is that this would also invalidate any storage tiering strategy unless, as you point out, there was an effective auto-tiering mechanism in place.

d. Use the multipath policy Fixed or Most Recently Used (MRU) instead of the round robin multipathing policy. Two things before considering this option:
Create an automatic routine to ensure all SLA1 based datastores are accessed throught HBA port 1 and that all SLA based datastores are accessed through HBA port 2.
Make sure the selected multipath policy is supported by the storage vendor.

This is the same a the first two as you are dedicating HBA resource to certain workloads and removing the contention issue.

Not ideal as you potentially impact the resilience of the storage subsystem if not configured appropriately.

A final option to consider would be to adjust the queue depths on a datastore by datastore basis but this would really only achieve a throttling outcome rather than a sharing/prioritisation during contention outcome.

Regards,

Rodney

References:

Storage I/O Fairness (Duncan Epping)
http://www.yellow-bricks.com/2010/09/29/storage-io-fairness/

Setting the Maximum Outstanding Disk Requests for virtual machines (1268)
http://kb.vmware.com/kb/12678

Changing the queue depth for QLogic, Emulex and Brocade HBAs (1267)
http://kb.vmware.com/kb/1267

/Andre