VMware Workstation – Virtual Ethernet failed – sometimes for good!

I use VMware Workstation as a hypervisor for hosting a few MS Windows guests, which I need in customer projects. As I do not trust Windows systems I sometimes place such guests in different virtual and isolated host-only networks on my workstation or a dedicated server. On other systems I run virtualized Linux server systems for production and tests with the help of KVM/QEMU, LXC and libvirt. Some time ago I learned the hard way that I have to keep track of the different "local" virtual networks more thoroughly. As the VMware side was affected by a misconfiguration in a rather peculiar way I thought this experience might be interesting for others, too.

Host interface to VMware virtual network sometimes not available?

The whole thing coincided with an upgrade of one of my Opensuse workstations. I am always a bit nervous whether and how such an upgrade may impact the VMware WS installation. Quite often I experienced problems with the automatic compilation in the past. In addition the start of some modules lead to secondary errors. However, at a first test VMware WS 14 compiled without problems on a system with Opensuse Leap 15. No wonder, I thought, as the kernel version 4.12 of Leap15 is relatively old.

Then I had to set up a Windows guest. I gave it an IP address within a virtual VMware host-only network, which covered an IPv4 address range of a class C network. VMware uses a kind of virtual bridge to support such a network; normally one associates a virtual network device on the host with it, to which you can assign a certain IP address in the net's address ange. For a C-net VMware uses yyy.yyy.yyy.1 as a default on the host's interface (with yyy.yyy.yyy defining the C-network). In my case the device on the host was named "vmnet6". In the beginning this virtual interface also worked as expected.

However, during the following days I noticed trouble with "vmnet6". In a normal host configuration the VMware WS initialization script ("/etc/init.d/vmware") is started as a LSB service by systemd. The script loads VMware modules and configures the defined virtual networks. Unfortunately, relatively often, my "vmnet6" interface did not become directly available after the start of the Linux host. Some experiments showed however that I could circumvent this problem by some "dubious" action:

I had to enter the "Virtual Network Editor" with root rights and save the already existing virtual network again. Then ifconfig or the ip command enlisted interface "vmnet6" - which also seemed to work normally.

You get a strange feeling when you are forced to remedy something as root .... However, I ignored this problem, which I could not solve directly, for a while. I also noticed that the problem did not occur all the time. This should have rang a bell ... but I was too stupid. Until, yesterday, when I needed to set up another special MS WIN guest in the same address range.

Virtual Ethernet failed

In addition I upgraded to VMware WS Version 14.1.3. The (automatic) compilation of the VMware WS modules on Opensuse Leap 15 again worked without major problems. But, when I manually (re-) started the VMware modules via "/etc/init.d/vmware restart" I saw an error message:

"Virtual Ethernet failed".

Such a message makes you nervous because you assume some real big problem with the VMware modules. However, starting some of my VMware guests (not the ones linked to vmnet6) proved that these guests operated flawlessly and could communicate via their and the hosts network devices. But my "vmnet6" did not appear in the output of "ip a s".

The problem "Virtual Ethernet failed" is reported in some Internet articles; however without a real solution. See e.g. https://ubuntuforums.org/showthread.php?t=1592977; https://www.linuxquestions.org/questions/slackware-14/vmware-workstation-unable-to-start-services-4175547448/; https://communities.vmware.com/thread/264235.

Now, I seriously began to wonder whether and how this problem was related to the already known problem with my virtual device "vmnet6". Some tests quickly showed that the error message disappeared when I eliminated the host-only network related to "vmnet6". Then I tried a new host-only test network with a device "vmnet7" and a different IP range. Also then the "virtual Ethernet" started flawlessly. So, what was wrong with "vmnet6" and/or its IP range? Some residual garbage from old installations? A search for configuration files gave me no better ideas.

vnetlib-log

After a minute I thought: Maybe VMware is right. A look into the log-file "/var/log/vnetlib" revealed:

Oct 14 14:22:49 VNL_Load - LOG_ERR logged
Oct 14 14:22:49 VNL_Load - LOG_WRN logged
Oct 14 14:22:49 VNL_Load - LOG_OK logged
Oct 14 14:22:49 VNL_Load - Successfully initialized Vnetlib
...
Oct 14 14:22:49 VNL_StartService - Started "Bridge" service for vnet: vmnet0
Oct 14 14:22:49 VNLPingAndCheckSubnet - Return value of vmware-ping: 0
...
Oct 14 14:22:50 VNL_CheckSubnetAvailability - Subnet: xxx.xxx.xxx.xxx on vnet: vmnet2 is available
...
Oct 14 14:22:49 VNL_CheckSubnetAvailability - Subnet: yyy.yyy.yyy.yyy on vnet: vmnet6 is not available
Subnet on vmnet6 is no longer available for usage, please run the network editor to reconfigure different subnet
Oct 14 14:22:50 VNL_CheckSubnetAvailability - Subnet: xxx.xxx.xxx.xxx on vnet: vmnet7 is available
Oct 14 14:22:50 VNL_CheckSubnetAvailability - Subnet: xxx.xxx.xxx.xxx on vnet: vmnet8 is available
.....

(I have replaced real addresses by xxx and yyy.) So, obviously VMware at startup performs a kind of ping check beyond the locally defined virtual devices and networks. Interesting! Why do they ping?

The answer is trivial: When you set up the (virtual) internal bridge for the host-only network you may want to guarantee that the IP-address for the respective host device ("vmnet6") does not exist anywhere else in the reachable network - especially as the host's interface of a host-only network may be used for a host controlled routing (and NAT) of the otherwise isolated VMware guests to the outside world.

Address overlap

And this resolved my problem: A manual ping for the address of the planned host's interface address yyy.yyy.yyy.1 indeed gave me a result. I located the source on another server, where I sometimes perform tests of different virtual network configurations with KVM guests, LXC containers and libvirt. Unfortunately, I had not disabled all of my test networks after my last tests. Due to default DHCP-settings a virtual interface with address yyy.yyy.yyy.1 got established on the test server whenever it was booted. This also explained the finding that the VMware WS problem did not occur always - it only happened when the test server was active in my physical LAN!

Then I actually remembered that I had had a similar problem once in 2016, whilst experimenting with QEMU/VMware-bridge-coupling. (See: Opensuse/Linux – KVM, VMware WS – virtuelle Brücken zwischen den Welten). So, I should have known better and planned my "local" virtual experiments a bit more carefully to avoid address overlaps with other potential virtual networks on different hosts in the LAN.

Stupid me ... The VMware WS startup script was absolutely right to not establish a second address of that kind on my workstation! This lead to the error message "Virtual ethernet failed" - which in my opinion should include a bit more detailed information or a hint to a log-file. But the fault clearly was on my side: One should not think in terms of a local host environment when using VMware WS for virtual networks, but consider the global network configuration.

Still, the whole VMware handling of a possible IP address overlap left me a bit puzzled :

Why can you by saving the questionable virtual network again establish a host interface despite the fact that another interface with the same address is running somewhere in the network? OK, you are root when you do it - but why is no warning given at this point? (VMware could do a ping check there, too .... )

A second question also worried me: Why did the existence of two devices with the same IP-address did not lead to more chaos in the network? One reason probably was that I had allowed pinging but no general TCP-transport to the virtual device on the test server. And explicit routes were otherwise defined properly on the different hosts. However, I could bet on some problems on the ARP-level when the test server was up and running. Anyway - such a basic misconfiguration in a a network may lead to security holes, too, and should, of course, be avoided.

A third question that came up for a second was: How do you avoid overlaps in case one wants to assign KVM/QEMU-guests and VMware guests IP addresses within the same network address space on one and the same virtualization host? Such a scenario is not at all as far fetched as it may seem at first sight. One reason for such a configuration could be the EU GDPR (DSGVO):

If you need to guarantee a customer confidentiality and are nevertheless forced to use a standard MS Windows client, you have a problem as MS may in an uncontrollable way transfer data to their own servers (outside the EU). Just read the license and maintenance agreements you sign with the operation of a standard Win 10 client! Therefore, you may want to isolate such clients drastically and only allow for communication with certain IP-addresses on the Internet (and NOT with MS servers). You may allow communication with some (virtualized) Linux machines in the same sub-network and one of them may serve as a gateway and perimeter firewall with strict filters. You can build up such scenarios by coupling a QEMU-virtual bridge to a VMware virtual bridge. At the same time, however, you need full control over the DHCP-systems on both sides (besides a bit of scripting) to avoid address overlaps. But this is actually easy: the DHCP-control files on the VMware side are found under the directory "/etc/vmware/vmnetX/dhcpd", with "X" standing for a virtual VMware interface number. On the QEMU side you find the files at "/etc/libvirt/qemu/networks". There you can control your IP assignments (or even no assignments for some special interfaces). Ok, but this is the beginning of another story.

Conclusion

Never think "local" or host based when working with virtual networks! Always include local virtual test networks in your documentation of your global network landscape! An do not always just accept the standard address assignment for host interfaces to virtual networks without thinking.

KVM – Anlegen eines privaten, isolierten Netzwerks mit virt-manager

Im vorhergehenden Artikel dieses Blogs
Opensuse – manuelles Anlegen von Bridge to LAN Devices (br0, br1, …) für KVM Hosts
hatte ich 2 Arten der Anbindung eines KVM-Gastsystems an die physikalische Umwelt des KVM-Hostes diskutiert. Ich hatte angemerkt, dass es unter KVM/libvirt neben einem direkten Bridging zu einer physikalischen Host-NIC natürlich auch die Möglichkeit gibt, KVM-Gäste an ein sog. "Host-Only-Network" [HON] anzubinden. Intern wird dieses Netzwerk durch eine virtuelle Bridge repräsentiert. Soll man aus dem HON heraus mit der physikalischen Umwelt (LAN) kommunizieren, muss man auf dem KVM-Host Routing zwischen einer bereitgestellten virtuellen Host-NIC der Bridge zu einer physikalischen NIC des Hostes ermöglichen. Letztere leitet die Pakete dann ins LAN.

Unterlässt man das Routing (und/oder filtert man Pakete aus dem HON) auf dem Host, so befinden sich Gastsystem und Host in einem isolierten virtuellen Netz, aus dem nach außen ohne weitere Vorkehrungen nicht kommuniziert werden kann. Dem isolierten Netz können natürlich weitere Gäste beitreten.

Wegen der Nachfrage einer Leserin, zeige ich nachfolgend kurz die Anlage eines virtuellen Host-Only-Netzwerks unter KVM mittels "virt-manager". Ich setze voraus, dass der "libvirtd"-Daemon läuft.

Man ruft als root "virt-manager" auf und geht im Übersichtsfenster auf "Edit >> Connection Details" und dort auf den Reiter "Virtual Networks". Dort findet man unter der Übersichtsliste zu bereits vorhandenen Netzwerken, einen Button mit einem "+" Symbol zum Anlegen eines neuen Netzwerks. Ich zeige nachfolgend die Dialogsequenz:

new_network_1

new_network_2

new_network_3

new_network_4

new_network_5

Die Bridge und eine zugehörige Host-NIC tauchen dann auch in der Liste der vorhandenen Netzwerk-Devices auf. Unter Opensuse zeigt das Komamndo "wicked show all" dann etwa ein virbr-Device (virtual bridge - im Beispiel ein "virbr2"):

virbr2          device-unconfigured
      link:     #73, state device-up, mtu 1500
      type:     bridge
      addr:     ipv4 192.168.120.1/24

virbr2-nic      device-unconfigured
      link:     #74, state down, mtu 1500, master virbr2
      type:     tap, hwaddr 52:54:00:c9:bd:24

 
Das neue Netzwerk findet sich dann auch in Form einer XML-Netzwerk-Konfigurations-Datei unter "/etc/libvirt/qemu/networks/host2.xml" wieder:

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh net-edit host2
or other application using the libvirt API.
-->

<network>
  <name>host2</name>
  <uuid>f47c2d04-b1d6-48bf-a6dc-a643d28b38d3</uuid>
  <bridge name='virbr2' stp='on' delay='0'/>
  <mac address='52:54:00:c9:bd:24'/>
  <domain name='host2'/>
  <ip address='192.168.120.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.120.128' end='192.168.120.254'/>
    </dhcp>
  </ip>
</network>

 
Will man eine solche Host-Only-Bridge von einem KVM-Gast aus nutzen, so muss man für diesen Gast ein entsprechendes "Netzwerk"-Device (NIC) anlegen, das der Bridge zugeordnet wird. "virt-manager" bietet auch hierfür entsprechende grafische Dialoge an. Ich gehe davon aus, dass eine virtuelle Maschine (z.B. namens "kali2") bereits existiert. Man öffnet deren Konfigurations-Oberfläche durch Doppelklick auf den entsprechenden Eintrag in der Liste aller KVM-Instanzen unter "virt-manager". Im sich öffnenden Fenster klickt man weiter auf den Button mit dem "i"-Symbol:

new_network_6

Danach taucht das Device auch in der XML-Konfigurationsdatei für den Gast auf - in meinem Beispiel etwa für einen Kali-Gast mit der Datei "/etc/libvirt/qemu/kali.xml" - ich zeige nur den relevanten Ausschnitt:

    <interface type='network'>
      <mac address='52:54:00:0d:3c:8b'/>
      <source network='host2'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
    </interface>

 
Danach muss man den Gast neu starten und in ihm natürlich das neu aufgetauchte Netzwerk-Interface manuell konfigurieren - falls man nicht auf DHCP setzt und/oder besondere Einstellungen benötigt. Man merke sich hierzu die MAC-Adresse, um bei mehreren NICs nicht den Überblick zu verlieren! Man achte auch auf das Default-Gateway, wenn Routing über den Host gewünscht ist.

Achtung: Das neue virtuelle Netzwerk des Gastes ist mit "ifconfig", "ip" oder "wicked" Kommandos erst als ein "vnetN"-Device sichtbar, wenn der Gast gestartet und aktiv ist. Das "N" steht dabei für eine fortlaufende Nummer, die vom System (libvirt) vergeben wird. Z.B. taucht unter "wicked show all" dann ggf. ein Device "vnet3" auf:

virbr2          device-unconfigured
      link:     #73, state device-up, mtu 1500
      type:     bridge
      addr:     ipv4 192.168.120.1/24

virbr2-nic      device-unconfigured
      link:     #74, state down, mtu 1500, master virbr2
      type:     tap, hwaddr 52:54:00:c9:bd:24
...
...
vnet3           device-unconfigured
      link:     #17, state up, mtu 1500, master virbr2
      type:     tap, hwaddr fe:54:00:0d:3c:8b

 
Man beachte, dass sowohl die Host-Nic, als auch das Device des Gastes "tap"-Devices sind. Allg. Infos zu tap-devices finden sich hier:
https://www.kernel.org/doc/Documentation/networking/tuntap.txt
https://de.wikipedia.org/wiki/TUN/TAP

Hinweise für Leser, die die Konfiguration lieber manuell und über die Kommandozeile durchführen mögen

Wie man eine virtuelle Bridge auf dem KVM-Host mittels des "brctl"-Kommandos einrichtet, benennt und wie man ihr "tap"-Devices zuordnet, habe ich im Prinzip bereits früher am Beispiel einer direkten Bridge zu einem physikalischen Device beschrieben:
Opensuse – manuelles Anlegen von Bridge to LAN Devices (br0, br1, …) für KVM Hosts
Siehe aber auch hier:
http://www.linux-kvm.org/page/Networking#Configuring_Guest_Networking - Abschnitt (Private Network)

"tap"-Devices kann man manuell und temporär über das Kommando "tunctl" auf dem Host erzeugen.
Siehe etwa :
http://unix.stackexchange.com/questions/86720/can-i-create-a-virtual-ethernet-interface-named-eth0
http://serverfault.com/questions/347895/creating-tun-tap-devices-on-linux
http://www.naturalborncoder.com/virtualization/2014/10/17/understanding-tun-tap-interfaces/
http://blog.elastocloud.org/2015/07/qemukvm-bridged-network-with-tap.html

Die Zuordnung von "tap"-Devices zu einer virtuellen Linux-Bridge erfolgt über das "brctl addif"-Kommando. Nun fehlt also nur noch eine Methode, um einmal erzeugte "tap"-Devices in die Konfiguration eines Gastes einzubinden.

Ich kenne einige Leute, die starten ihre virtuellen Maschinen lieber eigenhändig und über Scripts statt über virt-manager. Dann kann die Netzwerk-Konfiguration des Gastsystems in Form passender Optionsparameter des Kommandos "qemu-system-x86_64" (mit weiteren Optionen für KVM-Unterstützung; s.u.) oder des Kommandos "qemu-kvm" zum Starten eines KVM-Gastes geschehen. Eine Übersicht über diese Möglichkeit findet man hier:
http://qemu-buch.de/de/index.php?title=QEMU-KVM-Buch/_Netzwerkoptionen/_Virtuelle_Netzwerke_konfigurieren
In abgekürzter Form auch hier :
https://bbs.archlinux.org/viewtopic.php?pid=1148335#p1148335
https://bbs.archlinux.org/viewtopic.php?pid=1424044#p1424044

Eine weitere Alternative ist hier beschrieben (s. den Abschnitt zu "Private Networking"):
http://www.linux-kvm.org/page/Networking#Configuring_Guest_Networking

Übrigens: "qemu-kvm" ist auf aktuellen Linux-Systemen meist nur ein kleines Shell-Script-Kommando, dass "qemu-system-x86_64" mit KVM-Hardware-Unterstützungsoptionen aufruft! Siehe zum Unterschied zw. "qemu-system-x86_64" und "qemu-kvm" etwa
http://www.linux-kvm.com/content/qemu-kvm-or-qemu-system-x8664%EF%BC%9F).
Die Optionen des "qemu-kvm"-Kommandos sind z.B. hier beschrieben:
https://www.suse.com/documentation/sles11/book_kvm/data/cha_qemu_running_gen_opts.html
Eine Zusammenfassung zu tap-Devices und verschiedenen Bridging-Varianten gibt auch
https://www.suse.com/documentation/sles11/book_kvm/data/cha_qemu_running_networking.html

Will man die Tools von "libvirt/virt-manager" zum Starten der virtuellen Maschine benutzen, dann kann die "manuelle" Definition von virtuellen NICs für einen KVM-Gastes aber auch mittels des "virsh edit"-Kommandos zur Manipulation des XML-Files für die Gastkonfiguration durchgeführt werden.
Siehe:
http://serverfault.com/questions/665440/set-up-network-interfaces-in-ubuntu-for-kvm-virtual-machine

In einigen Situationen kann es auch erforderlich sein, den Gast im laufenden Betrieb um ein Netzwerkinterface zu einem neu definierten virtuellen Netz zu erweitern. Informationen hierzu findet man hier:
http://www.linuxwave.info/2014/12/hot-attach-and-hot-detach-network.html
https://kashyapc.fedorapeople.org/virt/add-network-card-in-guest.txt