Upgrade Nutanix Operating System to version 4.0

A few days ago i did a Nutanix Operating System (NOS) upgrade from version 3.5.4 to version 4.0. This blog post will describe the steps i took during the upgrade of a Nutanix 1350 Block just to outline how easy it actually is to upgrade the distributed Nutanix storage system.
There are a few steps you manually need to take but many (most) of them are  there to make sure we provide the upgrade process with the best possible conditions to complete a successful upgrade.

Some additional tips before starting the upgrade:

• Always run the cluster status command to verify that all Controller VMs are up before performing a Controller VM or host shutdown or restart.
• Never shut down or restart multiple Controller VMs or hosts simultaneously.

These are the steps i took when upgrading from NOS 3.5.4 to 4.0

1. Download the NOS 4.0 release from the Nutanix web site.
   
2. Upload the NIS 4.0 release to any of the Controller Virtual Machines (CVMs). I used the pscp command “pscp.exe nutanix_installer_package-danube-4.0-8b40.tar.gz nutanix@10.x.y.68:/home/nutanix/
   
3. Log on, via SSH, to one of my CVM.
4. Run the cluster command “ncli cluster version” to verify the running Nutanix cluster version.
   
5. Verify the Nutanix cluster status via the command “ncli cluster status”
   Make sure all checks, apart from the name (IP address) of each CVM is up.
6. Verify your Nutanix cluster don’t have any specific parameters that will prevent a successful upgrade by running the following command/syntax:
   for i in `svmips`; do echo $i; ssh $i “if [ -e
   ~/config/stargate.gflags.zk ]; then grep extent_cache
   ~/config/stargate.gflags.zk; fi”; done
   Verify only the IP addresses of the CVMs and no configuration parameters are listed as output.
   
7. If all the CVMs IP addresses are listed with Status “Up”, and State “Normal” while running the command “nodetool -h localhost ring” you know that all CVMs are part of the metadata store.
   
8. For all CVMs (via SSH) run the command “sudo smartctl -a /dev/sda | grep result” and verify no errors are reported. This will make sure the CVMs boot drive is healthy. Expected output from the command is:
   SMART overall-health self-assessment test result: PASSED
9. Run the command “cluster enable_auto_install” to enable automatic installation.
10. Time synchronization is important for all systems and this applies to NOS as well. Verify the NTP settings before continue by running the “ncli cluster get-ntp-servers” command.
11. Expand the tar.gz file by running the “tar -zxf nutanix_installer_package-danube-4.0-8b40.tar.gz” command.
12. Start the upgrade process by running the “/home/nutanix/install/bin/cluster -i /home/nutanix/install upgrade” command.
13. The upgrade process can be monitored via the “upgrade_status” command. You can also tail the log file /home/nutanix/data/logs/install.out
14. All CVMs will be rebooted and when completed you can run the “upgrade_status” command and the output should lok similar to:
    2014-06-30 15:36:39 INFO upgrade_status:38 Target release version: 1.7-release-danube-4.0-stable-3b5fa1d2b4d200b04c47b9e9cb4411624d198b40
    2014-06-30 15:36:39 INFO upgrade_status:43 Cluster upgrade method is set to: automatic rolling upgrade
    2014-06-30 15:36:39 INFO upgrade_status:96 SVM 10.x.y.66 is up to date
    2014-06-30 15:36:39 INFO upgrade_status:96 SVM 10.x.y.67 is up to date
    2014-06-30 15:36:39 INFO upgrade_status:96 SVM 10.x.y.68 is up to date
15. When all the CVMs have been rebooted you can verify the new NOS version by running “ncli cluster version” command. The output should be similar to:
    

You should verify things like e.g. CVMs virtual hardware (CPU, RAM, NIC driver) CVM timezone configuration when the upgrade is finished but all the steps are outlined in the Nutanix official upgrade document found here.

Disclaimer: I strongly advice you to conduct the Nutanix official upgrade document, again found here, when upgrading your system to make sure you don’t miss anything critical. This upgrade was conducted in a proof of concept (POC) environment without e.g.:

• No remote replication setup
• Email alert enabled at present time
• No iSCSI LUNs