vmnic Switches During vMotion: Check ESXi TX Hang and NIC Firmware

Early answer: When a vmnic unexpectedly switches during vMotion and you see physical switch port down events plus ESXi redundancy loss alerts, check /var/log/vmkernel.log for TX hang entries. A TX hang indicates the NIC driver/firmware combination is failing under high-throughput conditions, triggering a firmware-level NIC reset that causes the vmnic change and network disruption. The resolution is updating the NIC driver and firmware to a VMware HCL-listed version obtained from the hardware vendor.

Symptoms and Initial Observations

The documented case presents a specific symptom set:

• Physical switch ports report down randomly during vMotion
• ESXi hosts report loss of redundancy during migration
• vMotion may fail outright, or succeed with noticeable network fluctuation
• esxtop shows the vmkernel adapter changing its active vmnic mid-migration

In the source case, vmk2 (the dedicated vMotion vmkernel adapter) was originally using vmnic3. During migration, esxtop revealed that vmk2 would suddenly switch to a different vmnic, and this switch correlated with physical network link state changes. Further examination of the vmkernel log revealed frequent TX hang errors—a key clue that narrowed the investigation from a broad network problem to a specific driver/firmware condition.

Distinguishing TX Hang from Normal NIC Teaming Failover

This distinction matters for accurate diagnosis. Normal NIC teaming failover—triggered by a genuine link loss or by configured failover order—is expected behavior. In those cases, the vmnic switch is a response to a network event, and the teaming policy governs which adapter takes over.

In the TX hang scenario, the vmnic switch is not a normal failover event. The NIC driver detects that its transmit queue has stalled, and the driver initiates a firmware-level NIC reset to recover. This reset causes the link to drop, which then triggers the teaming failover. The physical switch port down event and the ESXi redundancy loss alert are downstream effects of the NIC reset, not the original cause. If you treat this as a teaming policy problem, you will spend time adjusting failover orders and uplink configurations without addressing the underlying driver/firmware defect.

Confirming TX Hang in the vmkernel Log

The key diagnostic step is searching the vmkernel log for TX hang entries:

1. SSH to the ESXi host
2. Navigate to the log directory: cd /var/log
3. View the vmkernel log: cat vmkernel.log | less
4. Search for TX hang: type /tx hang (case-insensitive)

The log entry typically resembles:

ixgben 0000:##:00.0: TX hang detected

The exact format may vary by hardware. The ixgben entry in the source case points to an Intel 10GbE NIC using the native ixgben driver, but the same TX hang pattern can surface with other driver/firmware combinations. Note the driver name and PCI address in the log entry—these identify which physical NIC is affected.

Root Cause: Driver/Firmware Behavior Under High Throughput

The confirmed root cause in the source case is at the NIC driver and firmware layer. During high-throughput operations like vMotion, specific driver/firmware combinations expose timing sensitivity or buffer management defects. The NIC transmit queue hangs, and the driver responds by triggering a firmware-level NIC reset. This reset causes the vmnic to drop, the physical switch port to report down, and the vmkernel adapter to fail over to another vmnic in the teaming group.

In the documented case, the evidence pointed below the vSwitch configuration layer. Driver and firmware remediation should be coordinated with the server or NIC vendor and checked against the Broadcom Compatibility Guide. The resolution requires working with the hardware vendor and referencing the VMware Hardware Compatibility List (HCL).

Timeline-Based Troubleshooting Method

Organize the investigation chronologically to correlate events across systems:

Phase 1—Capture symptoms during or immediately after the event:

• Note the exact time of the vMotion attempt
• Record which vmnic was active before the switch using esxtop (press n for the network view)
• Record which vmnic became active after the switch
• Note whether vMotion succeeded or failed
• Check the physical switch logs for port down events at the same timestamp

Phase 2—Examine the vmkernel log:

• SSH to the host and run cd /var/log
• Run cat vmkernel.log | less and search /tx hang
• Correlate TX hang timestamps with the vMotion event time
• Note the driver name and PCI address from the log entry

Phase 3—Identify the driver and firmware versions:

• Run esxcli network nic list to identify the NIC model and current driver
• Run esxcli network nic get -n vmnicX for detailed driver and firmware version information
• Cross-reference the results against the VMware HCL
• Check whether the current driver/firmware combination is listed as tested