I use VMware Workstation as a hypervisor for hosting a few MS Windows guests, which I need in customer projects. As I do not trust Windows systems I sometimes place such guests in different virtual and isolated host-only networks on my workstation or a dedicated server. On other systems I run virtualized Linux server systems for production and tests with the help of KVM/QEMU, LXC and libvirt. Some time ago I learned the hard way that I have to keep track of the different “local” virtual networks more thoroughly. As the VMware side was affected by a misconfiguration in a rather peculiar way I thought this experience might be interesting for others, too.
Host interface to VMware virtual network sometimes not available?
The whole thing coincided with an upgrade of one of my Opensuse workstations. I am always a bit nervous whether and how such an upgrade may impact the VMware WS installation. Quite often I experienced problems with the automatic compilation in the past. In addition the start of some modules lead to secondary errors. However, at a first test VMware WS 14 compiled without problems on a system with Opensuse Leap 15. No wonder, I thought, as the kernel version 4.12 of Leap15 is relatively old.
Then I had to set up a Windows guest. I gave it an IP address within a virtual VMware host-only network, which covered an IPv4 address range of a class C network. VMware uses a kind of virtual bridge to support such a network; normally one associates a virtual network device on the host with it, to which you can assign a certain IP address in the net’s address ange. For a C-net VMware uses yyy.yyy.yyy.1 as a default on the host’s interface (with yyy.yyy.yyy defining the C-network). In my case the device on the host was named “vmnet6”. In the beginning this virtual interface also worked as expected.
However, during the following days I noticed trouble with “vmnet6”. In a normal host configuration the VMware WS initialization script (“/etc/init.d/vmware”) is started as a LSB service by systemd. The script loads VMware modules and configures the defined virtual networks. Unfortunately, relatively often, my “vmnet6” interface did not become directly available after the start of the Linux host. Some experiments showed however that I could circumvent this problem by some “dubious” action:
I had to enter the “Virtual Network Editor” with root rights and save the already existing virtual network again. Then ifconfig or the ip command enlisted interface “vmnet6” – which also seemed to work normally.
You get a strange feeling when you are forced to remedy something as root …. However, I ignored this problem, which I could not solve directly, for a while. I also noticed that the problem did not occur all the time. This should have rang a bell … but I was too stupid. Until, yesterday, when I needed to set up another special MS WIN guest in the same address range.
Virtual Ethernet failed
In addition I upgraded to VMware WS Version 14.1.3. The (automatic) compilation of the VMware WS modules on Opensuse Leap 15 again worked without major problems. But, when I manually (re-) started the VMware modules via “/etc/init.d/vmware restart” I saw an error message:
“Virtual Ethernet failed”.
Such a message makes you nervous because you assume some real big problem with the VMware modules. However, starting some of my VMware guests (not the ones linked to vmnet6) proved that these guests operated flawlessly and could communicate via their and the hosts network devices. But my “vmnet6” did not appear in the output of “ip a s”.
The problem “Virtual Ethernet failed” is reported in some Internet articles; however without a real solution. See e.g. https://ubuntuforums.org/showthread.php?t=1592977; https://www.linuxquestions.org/questions/slackware-14/vmware-workstation-unable-to-start-services-4175547448/; https://communities.vmware.com/thread/264235.
Now, I seriously began to wonder whether and how this problem was related to the already known problem with my virtual device “vmnet6”. Some tests quickly showed that the error message disappeared when I eliminated the host-only network related to “vmnet6”. Then I tried a new host-only test network with a device “vmnet7” and a different IP range. Also then the “virtual Ethernet” started flawlessly. So, what was wrong with “vmnet6” and/or its IP range? Some residual garbage from old installations? A search for configuration files gave me no better ideas.
vnetlib-log
After a minute I thought: Maybe VMware is right. A look into the log-file “/var/log/vnetlib” revealed:
Oct 14 14:22:49 VNL_Load - LOG_ERR logged Oct 14 14:22:49 VNL_Load - LOG_WRN logged Oct 14 14:22:49 VNL_Load - LOG_OK logged Oct 14 14:22:49 VNL_Load - Successfully initialized Vnetlib ... Oct 14 14:22:49 VNL_StartService - Started "Bridge" service for vnet: vmnet0 Oct 14 14:22:49 VNLPingAndCheckSubnet - Return value of vmware-ping: 0 ... Oct 14 14:22:50 VNL_CheckSubnetAvailability - Subnet: xxx.xxx.xxx.xxx on vnet: vmnet2 is available ... Oct 14 14:22:49 VNL_CheckSubnetAvailability - Subnet: yyy.yyy.yyy.yyy on vnet: vmnet6 is not available Subnet on vmnet6 is no longer available for usage, please run the network editor to reconfigure different subnet Oct 14 14:22:50 VNL_CheckSubnetAvailability - Subnet: xxx.xxx.xxx.xxx on vnet: vmnet7 is available Oct 14 14:22:50 VNL_CheckSubnetAvailability - Subnet: xxx.xxx.xxx.xxx on vnet: vmnet8 is available .....
(I have replaced real addresses by xxx and yyy.) So, obviously VMware at startup performs a kind of ping check beyond the locally defined virtual devices and networks. Interesting! Why do they ping?
The answer is trivial: When you set up the (virtual) internal bridge for the host-only network you may want to guarantee that the IP-address for the respective host device (“vmnet6”) does not exist anywhere else in the reachable network – especially as the host’s interface of a host-only network may be used for a host controlled routing (and NAT) of the otherwise isolated VMware guests to the outside world.
Address overlap
And this resolved my problem: A manual ping for the address of the planned host’s interface address yyy.yyy.yyy.1 indeed gave me a result. I located the source on another server, where I sometimes perform tests of different virtual network configurations with KVM guests, LXC containers and libvirt. Unfortunately, I had not disabled all of my test networks after my last tests. Due to default DHCP-settings a virtual interface with address yyy.yyy.yyy.1 got established on the test server whenever it was booted. This also explained the finding that the VMware WS problem did not occur always – it only happened when the test server was active in my physical LAN!
Then I actually remembered that I had had a similar problem once in 2016, whilst experimenting with QEMU/VMware-bridge-coupling. (See: Opensuse/Linux – KVM, VMware WS – virtuelle Brücken zwischen den Welten). So, I should have known better and planned my “local” virtual experiments a bit more carefully to avoid address overlaps with other potential virtual networks on different hosts in the LAN.
Stupid me … The VMware WS startup script was absolutely right to not establish a second address
of that kind on my workstation! This lead to the error message “Virtual ethernet failed” – which in my opinion should include a bit more detailed information or a hint to a log-file. But the fault clearly was on my side: One should not think in terms of a local host environment when using VMware WS for virtual networks, but consider the global network configuration.
Still, the whole VMware handling of a possible IP address overlap left me a bit puzzled :
Why can you by saving the questionable virtual network again establish a host interface despite the fact that another interface with the same address is running somewhere in the network? OK, you are root when you do it – but why is no warning given at this point? (VMware could do a ping check there, too …. )
A second question also worried me: Why did the existence of two devices with the same IP-address did not lead to more chaos in the network? One reason probably was that I had allowed pinging but no general TCP-transport to the virtual device on the test server. And explicit routes were otherwise defined properly on the different hosts. However, I could bet on some problems on the ARP-level when the test server was up and running. Anyway – such a basic misconfiguration in a a network may lead to security holes, too, and should, of course, be avoided.
A third question that came up for a second was: How do you avoid overlaps in case one wants to assign KVM/QEMU-guests and VMware guests IP addresses within the same network address space on one and the same virtualization host? Such a scenario is not at all as far fetched as it may seem at first sight. One reason for such a configuration could be the EU GDPR (DSGVO):
If you need to guarantee a customer confidentiality and are nevertheless forced to use a standard MS Windows client, you have a problem as MS may in an uncontrollable way transfer data to their own servers (outside the EU). Just read the license and maintenance agreements you sign with the operation of a standard Win 10 client! Therefore, you may want to isolate such clients drastically and only allow for communication with certain IP-addresses on the Internet (and NOT with MS servers). You may allow communication with some (virtualized) Linux machines in the same sub-network and one of them may serve as a gateway and perimeter firewall with strict filters. You can build up such scenarios by coupling a QEMU-virtual bridge to a VMware virtual bridge. At the same time, however, you need full control over the DHCP-systems on both sides (besides a bit of scripting) to avoid address overlaps. But this is actually easy: the DHCP-control files on the VMware side are found under the directory “/etc/vmware/vmnetX/dhcpd”, with “X” standing for a virtual VMware interface number. On the QEMU side you find the files at “/etc/libvirt/qemu/networks”. There you can control your IP assignments (or even no assignments for some special interfaces). Ok, but this is the beginning of another story.
Conclusion
Never think “local” or host based when working with virtual networks! Always include local virtual test networks in your documentation of your global network landscape! An do not always just accept the standard address assignment for host interfaces to virtual networks without thinking.