Nutanix has seen requests for achieving all flash performance for specific workload(s) within a hybrid system meaning data for business critical and latency sensitive applications should not be migrated from hot data tier, backed by SSD disks, to cold data tier, backed by HDD (spinning) disks. Because of this increasing demand Nutanix has created a feature called VM Flash Mode (VFM) that makes it possible to guarantee that an entire virtual machine (VM) virtual disk or a portion of a VM virtual disk always resides in the hot tier. VM Flash Mode was previously called Pin to SSD.
The figure to the right shows the options we have:
|
 |
This is a great feature and can be handy i some cases but you should evaluate available Nutanix data reduction technologies such as Compression, Deduplication and EC-X before enabling VM Flash Mode. The three data reduction technologies mention provide an effective increase in the usable SSD tier and as a result this will allow Nutanix Information Lifecycle Manager (ILM) to maintain more data in the hot tier giving a flash performance experience to a larger working set.
How it works
A maximum of 25% of the available Acropolis Distributed Storage Fabric (DSF), formerly known as Nutanix Distributed File System (NDFS), SSD capacity can be used for the VM Flash Mode feature but there will be no specific partition in the software set aside for this. VM Flash Mode uses the SSD part of the Nutanix extent store capacity and is completely controlled by software.
When using the virtual disk portion configuration option it is critical to understand that data classified as hot day 1 might not be classified as hot data day 1+X and based on that it can be migrated from hot tier to cold tier by DSF ILM. This is because data is assigned to hot tier based on a first-in first-out (FIFO) basis by default and this applies to VM Flash Mode portion configuration as well.
The following example explains what can happen when using the VM Flash Mode portion configuration for a virtual disk.
 |
 |
 |
 |
As previously mention the VM Flash Mode feature works on a per virtual machine virtual disk level and they can be pinned either when they are created or for an existing virtual disk. If enabling VM Flash Mode on an existing virtual disk and its data reside on cold tier, it will not be migrated to SSD by the VM Flash Mode feature. Data will be moved to SSD as it is migrated up to hot tier using the normal ILM mechanism.
Nutanix stores two or three identical blocks of the same data depending on Nutanix container configuration. For new writes, both or all three blocks will be placed in hot tier by VM Flash Mode but that does not apply when data is being migrated from cold to hot tier.
During a node failure scenario the available SSD capacity will be reduced and a situation where more than 25% of the SSD tier is actually already used by VM Flash Mode can occur e.g.:
- 4 Nutanix nodes provide 600 XG VM Flash Mode capacity.
- 500 XG is used by VM Flash Mode.
- 1 Nutanix node fails and only 450 XG is available for VM Flash Mode.
When the failure scenario appears, the soft limit will increase to 50% of total SSD capacity meaning data will not be migrated from hot tier to cold tier. When the VM Flash Mode feature uses more than 50% of the SSD space during a node failure scenario DSF will start migrating data form hot tier to cold tier.
The virtual machine configuration is managed via NCLI using the following command:
- ncli virtual-disk update-pinning id=<virtual-disk-id> tier_name=SSD-SATA pinned-space=<size>
Note: If the entire disk is pointed to VM Flash Mode initially there is no need to run the command again if/when the virtual disk is extended, the new size will be included automatically.
The configuration can be verified using e.g: the command disk_config_printer:
- This output shows a virtual disk that is completely assigned to the VM Flash Mode:
- params {
total_reserved_capacity: 42949672960
random_io_tier_preference: “SSD-SATA”
sequential_io_tier_preference: “SSD-SATA”
}
- params {
- This output shows a virtual disk with a portion (4 GB) assigned to VM Flash Mode:
- params {
tier_params {
tier_name: “SSD-SATA”
min_pinned_usage_bytes: 4294967296
}
}
- params {
The VM Flash Mode feature is hypervisor agnostic meaning you can take advantage of the feature when running the Acropolis Base Software version 4.5 and any of the supported hypervisors:
- Acropolis
- ESXi
- Hyper-V
There are a lot of features included in the different supported hypervisors and in DSF. This section outlines the interoperability for VM Flash mode and certain features:
- Block awareness – No effect in homogenous Nutanix clusters. In heterogeneous Nutanix clusters a virtual machine virtual disk completely placed on hot tier by VM Flash Mode might fail a write IO based on unavailable SSD space.
- Clone – Newly cloned virtual disk will not preserve the VM Flash Mode configuration, the VM Flash Mode configuration will be applicable to the original virtual disk. Configure VM Flash Mode for the newly cloned virtual disk to take the advantage of the feature.
- Compression – Has no affect on VM Flash Mode. Less SSD space will be used based on the DSF compression feature.
- Deduplication – Has no affect on VM Flash Mode. Less SSD space will be used based on the DSF deduplication feature.
- Snapshot – A DSF VM snapshot preserves the virtual disk VM Flash Mode configuration. Restore a DSF VM snapshot will not preserve the VM Flash Mode configuration. Configure VM Flash Mode for the restored virtual disk to take the advantage of the feature.
- Replication – Remote copy will not have VM Flash Mode enabled. Original virtual disk will keep its VM Flash Mode configuration.
- Disaster Recovery – Remote copy will not have VM Flash Mode enabled. Original virtual disk will keep its VM Flash Mode configuration. A restore operation will not preserve the VM Flash Mode configuration.
- Live Migration/vMotion – Does not affect VM Flash Mode configuration.
- Storage Migration/Storage vMotion – The new virtual disk created will not have VM Flash Mode enabled. Configure VM Flash Mode for the newly created virtual disk to take the advantage of the feature.
- Metro availability – VM Flash Mode configuration will not be propagated to the VM running in the second Nutanix cluster.
Final word
VM Flash Mode should be used as a last resort as it reduces the ability for the DSF to manage workloads in a dynamic manner and hot tier capacity might be allocated for workload(s) that not required hot tier performance.
Do not use the VM Flash Mode feature in chargeback purposes meaning you always pin X portion of a VM virtual disk to be able to charge customer or internal department more money for hosting your VMs.
Always perform a root cause analyse before enabling VM Flash Mode and engage the excellent Nutanix support for such an analyse.
If VM Flash Mode is the way to go the following guidelines should be considered:
- Use for latency sensitive applications.
- Use for workloads that runs on a schedule that makes its data being migrated to cold tier between the scheduled jobs.
- Activate the entire virtual disk if possible.
- When activating a portion of the virtual diska the data you want to be hot might be migrated to cold tier from hot tier. This happens when other none critical data is written to the virtual disk and uses the virtual disk VM Flash Mode hot tier space.
- Activate for read workloads. Write intensive workload will hit SSD either via random writes that goes to SSD because of oplog or sequential writes that goes to hot tier in extent store.
- It is very important to monitor the hot tier usage in a Nutanix Xtreme Computing Platform environment no matter if you use VM Flask Mode or not.