Migrate Nutanix AHV Network Traffic From 2×1 Gbps NICs To 2×10 Gbps NICs

A while back a Nutanix AHV customer implemented 10 Gbps at their remote office so we had to change the 2×1 Gbps NICs used by their Nutanix cluster to 2×10 Gbps NICs. This blog post will cover the steps taken to achieve that and it will include Controller Virtual Machine (CVM) and Prism configuration and verification actions.

The steps described below are the once I used and you can use whatever steps you feel necessary in your environment. If you don’t believe any network connectivity tests are needed just jump to the end where we change the 2×1 Gbps NICs to the 2×10 Gbps NICs for bond br0-up:)

The initial configuration looked like this for all three AHV hosts.

The above figure can be reached in Prism via the Network option in the drop down box in the top left corner and then select the AHV host.

The following steps were taken to perform the migration from 2×1 Gbps NICs to 2×10 Gbps NICs and the commands were executed from one of the Nutanix cluster CVMs.

Note: Make sure to use two minus signs before e.g. command switch interfaces meaning since the formatting might not work if you copy and past from the blog post.

Create a new bridge on the AHV hosts using one of the following commands:
1. hostssh /usr/bin/ovs-vsctl add-br br1
  1. Will create a new bridge on all AHV Hosts in the Nutanix Cluster
2. ssh root@192.168.5.1 /usr/bin/ovs-vsctl add-br br1
  1. Will create a new Open vSwitch bridge on the AHV host where the CVM you ran the command runs
From now on I will use the hostssh command and perform all actions on all AHV hosts in the Nutanix cluster. Use ssh root@192.168.5.1 to configure one AHV host at the time. Kind of recommended:)
Since the command from Step 1 does not provide any output you can verify you now have two bridges called br0 and the new one called br1 via:
1. Command line using
  1. hostssh /usr/bin/ovs-vsctl list bridge | grep name
2. PRISM
List available 10 Gbps interfaces using the below command and the figure below shows expected output from one of the AHV hosts in my case:
1. allssh manage_ovs show_interfaces | grep 10000
Create a bond, br1-up, one the newly created bridge, br1, and add both 10 Gbps NICs.
1. allssh manage_ovs –bridge_name br1 –interfaces eth2,eth3 –bond_name br1-up –require_link=false update_uplinks
Verification done via one of the following:
1. allssh manage_ovs show_uplinks
2. PRISM
Create as many new Acropolis networks needed so you can verify all VLANs used in your environment. Do not forget the one where the AHV hosts and the CVMs are running. In our case we only had two VLANs, both using VLAN tags, in used and created them using the following commands:
1. acli net.create vlan201 vlan=201 vswitch_name=br1
  Note: what you can do if you were about to keep both bridges was to name the network vlan201.br1 so you immediately recognise all Networks created on bridge 1.
2. acli net.create vlan401 vlan=401 vswitch_name=br1
Verification of the newly created networks via one of the following:
1. acli net.list | grep -e 201 -e 401
2. PRISM via Gear icon in the top right corner -> Network Configuration
Create 2 VMs and place them on the newly created networks
Start a ping session to both VMs
Verify the VLAN are accessible to both eth2 and eth3 by shutting down the interfaces using the following command sequence:
1. hostssh ifconfig | grep eth3 ; hostssh ifconfig eth3 down; hostssh ifconfig | grep eth3
  1. This will shut down the eth3 interface and verify eth2 can manage the traffic. The ping sessions will continue to run successfully. A few echo replies might get lost if the session(s) was running on eth3
    1. Command “allssh manage_ovs show_interfaces” will show the following for eth3:
      1. link=false
      2. speed=none
2. hostssh ifconfig | grep eth2 ; hostssh ifconfig eth2 down; hostssh ifconfig | grep eth2
  1. This will shut down eth2 interface so now the ping sessions times out.
    1. Command “allssh manage_ovs show_interfaces” will show the following for eth2:
      1. link=false
      2. speed=none
3. hostssh ifconfig | grep eth3 ; hostssh ifconfig eth3 up; hostssh ifconfig | grep eth3
  1. Now the eth3 interface will be active again and the ping sessions will start receiving responses. Might take a few seconds depending on physical network infrastructure.
4. hostssh ifconfig | grep eth2 ; hostssh ifconfig eth2 up; hostssh ifconfig | grep eth2
  1. Now the eth2 interface will be active again
Delete the following items you created in the above steps:
1. VMs
2. Acropolis networks
  1. acli net.delete vlan201
  2. acli net.delete vlan401
3. Bridge 1
  1. hostssh /usr/bin/ovs-vsctl del-br br1
  2. Now we are back with initial network configuration which can be verified using:
    1. Command line using:
      1. hostssh /usr/bin/ovs-vsctl list bridge | grep name
    2. PRISM
Now when we have verified that the 2×10 Gbps NICs can manage both our VLANs and cleaned up what we did earlier we can change the 2×1 Gbps NICs for the 2×10 Gbps NICs on the production bond br0-up.
1. allssh manage_ovs –interfaces eth2,eth3 update_uplinks
  I created a separate blog post about Nutanix AHV Network Configuration – Only Use 10Gbps NICs
Verify the configuration changes via:
1. Command line using:
  1. allssh manage_ovs show_uplinks
  2. PRISM
Configure Network Teaming accordingly and in our case we wanted an active/active NIC configuration without LACP so we were using the below commands which aligns with Nutanix recommendations for the balance-sob mode. Last command is for verification purposes. I created a separate blog post about Nutanix AHV Network Configuration – Change Load Balancing
1. hostssh ovs-vsctl set port br0-up bond_mode=balance-slb
2. hostssh ovs-vsctl set port br0-up other_config:bond-rebalance-interval=30000
3. hostssh ovs-appctl bond/show br0-up
Clear Prism Alerts to get rid of all the once we have created by e.g. taking interfaces up and down.
And as always after configuration changes , RUN NCC via one of the following:
1. Command line:
  1. ncc health_checks run_all
2. PRISM -> Health -> Actions – Run Checks

Copyright