More fun with veth, network namespaces, VLANs – I – open questions

Posted on 14. February 2024 by eremo

With this post I start a new series on the topic of virtual networks which we can create on a Linux virtualization host with commonly available and relatively simple standard tools. Lately, I have received relatively many questions of blog readers on this topic and meanwhile gathered enough new stuff that setting up a second post series seemed to be appropriate.

Introduction

Modern Linux systems use virtualization. This includes the use of containers as well as KVM-based virtual guest systems. Even in small companies a system admin will organize virtual clients and servers in groups. She/he will put virtualized guest systems into virtual network segments – often coinciding with IP-subnets. As Linux offers several classes of virtual switches the resulting virtual network can reach the same complexity as a real network.

Regarding security such a virtual network requires the application of all the measures which we take and apply in a real network environment. Single virtualized Linux hosts or groups of such hosts must be protected against each other. The dictum to follow is: Assume that the attacker is already somewhere on site.

Therefore, on the one hand the admin has to fully implement security appliances as e.g. firewalls and IDS systems on all virtualized hosts and at key points in the network. On the other hand it may be required to isolate groups of virtualized hosts and their group-internal communication against members of other groups. You normally would use IP-subnets and (V)LAN-segments plus routers and firewalls to achieve this.

Virtual VLANs for the isolation of groups of virtualized hosts

Virtual VLANs support the isolation of certain groups of virtualized hosts on the Link layer, i.e. even within a defined IP-subnet. VLANs define separate Ethernet broadcast domains on an otherwise normal LAN-segment. VLANs together with VLAN-aware switches may therefore help to reduce the overall network traffic and, at the same time, protect the communication inside VLAN-segments. But VLANs also introduce a new level of complexity for virtual networking.

The additional complexity has three aspects:

The commands for the setup of a virtual network may strongly depend on the Linux tool set used.
The configuration of virtual bridges/switches may differ from that of commercially available HW-based L2/L3-switches.
At hosts (or Linux network namespaces) with multiple attached (V)LAN-segments ambiguities regarding the choice of a proper interface for sending elementary packets may come up. Veth devices with VLAN interfaces almost naturally produce such a kind of ambiguity.
In some situation with ICMP- and ARP-requests and replies the Linux kernel may run through a specific chain of decisions regarding the transmission of VLAN-tagged packets through a particular VLAN-interface among multiple available ones. This chain may depend upon the type of virtual network interface or of the type of virtual bridge/switch used.

Hosts (or namespaces) with multiple attached (V)LAN-segments are critical points in a network. They require a careful configuration, e.g. regarding routes. This is valid for virtual networks, too. Remember that a such a host or namespace with multiple (V)LAN-segments need not be a router. It could just be a host responsible for the administration of other hosts in different (V)LAN-segments. In any case one needs to get a clear idea about the flow of key protocol packets at such hosts or network namespaces.

Namespaces instead of hosts for virtual (V)LAN planning and testing

Planning and testing a secure and segmented virtual network is a major task. A quick method to test the basic (V)LAN-segment layout would be very welcome. In case that you are just interested in the communication between hosts or segments without caring about details of the OS setup of a container or a KVM guest, then configuring a set of Linux network namespaces coupled by Linux bridges and veth devices may reduce your work load for testing significantly. A network namespace can replace or represent a virtualized Linux host in very many aspects regarding network traffic.

Older posts series of 2017 and motivation for a continuation

I have already written some posts series on virtualized (V)LANs and the usage of veth-devices, network namespaces and Linux bridges. See e.g.
Fun with veth-devices, Linux bridges and VLANs in unnamed Linux network namespaces – I
and
Linux bridges – can iptables be used against MiM attacks based on ARP spoofing ? – I
plus related posts.

The motivation behind the first of the named series was the definition of separate packet paths in complex virtual VLANs based on Linux bridges. Amongst other measures VLAN tagging at bridge ports is a central method to define certain allowed and disallowed paths for the propagation of Ethernet packets between virtual hosts or network namespaces within a given IP-subnet. Additional iptables, ebtables or nftables rules may help to control the traffic between certain bridge-ports and the segments behind them.

However, my old series did not fully cover some interesting points at namespaces where multiple VLAN lines terminate.

I want to mention that some really clever and serious questions of a reader, Joshua Greenberg, motivated me to consider related problems at namespaces with and without forwarding between VLAN-segments once again. His questions concerning some of my old statements gave me a bit of a headache – in particular with respect to ARP and routes. We agreed upon that some of the questions could only be answered by further experiments. So, many thanks to Joshua for giving me a push to turn my head towards virtual networking again.

This new post series deepens and extends the discussion of VLAN-related information given in my older post series. Among other things I will have a closer look on scenarios in which two or more separate LAN- and VLAN-segments are connected to a namespace with routing rules. All NICs will have IPs of one and the same IP-subnet (C-class net). In addition the behavior of ARP and ICMP packets on Linux bridges with iptables and ebtables rules will be analyzed.

On our way I will give you further simple examples for virtualized network setups – including VLANs – based on very elementary and generally available Linux tools (see below for a short list). The posts will again demonstrate how to use these tools to quickly define a segmented (V)LAN environment. But I will look deeper into some aspects than before – and thereby answer some open questions of readers. For example, I will have a look on some Linux kernel parameters.

Although I will repeat some basic points, my older posts are not obsolete. In particular a view into the introductory posts of the first series may be worth it. There I have given some basic information on veths and Linux bridges. I have also defined VLAN-configuration rules for bridge ports. These rules will be used in the present series, too.

In other posts of the previous series some special, but simple configurations were analyzed. The discussions there may help you to understand the new lines of thoughts in the forthcoming posts better. And I will also come back to the relatively big scenario discussed in my old post series.

Restrictions to simple virtual devices

This post series is definitely related to Linux systems. I do not care about MS Windows. Linux offers a big palette of tools and different virtual devices for virtual networking. I will restrict all efforts and considerations to a simple set of tools:

Linux network namespaces (with routing capabilities),
Linux virtual bridges,
Ethernet-capable devices,
Linux veth devices (with two peer devices and the capability to support VLAN sub-devices),
iptables and ebtables as packet filters.
Some real Ethernet NIC with contact to a real network segment and a router to the Internet.

All these elements should be available on a modern Linux system.

Focus on ARP and ICMP as key protocols at critical points with potential ambiguities

To set a focus for this new post series I will analyze predominantly ARP and ICMP traffic at critical places in our virtual network, e.g. at network namespaces where multiple (V)LAN segments come together. Other critical points mark the transition to real networks in some forwarding namespaces.

The experiments will cast light on possible ambiguities which arise due to spanning a IP-subnet over multiple L2-segments or over multiple VLAN segments. We will see that the capability of veth devices to support virtual VLANs introduces a basic ambiguity which the Linux kernel has to resolve – even on the level f the ARP protocol. We will in particular analyze the role of routes at such points and – as dubious as it may sound – the impact of routes on ARP traffic.

Open points of my previous post series and related questions

Both of my old post series on virtual networking got a long way, but were not finalized. I briefly outline some of the questions that remained open.

Multiple L2-segments attached to a common network namespace

I define a L2-segment as a LAN-segment, in which packets on the Link Layer travel freely. A L2-segment forms an Ethernet broadcast domain; Ethernet broadcast packets reach all NICs attached to a L2-segment. A good introduction to L1- and L2-segments and related Ethernet broadcasts is given here. Note that a complexly and hierarchically structured L2-segment may be created by connecting real or virtual linear Ethernet bus-lines by Linux bridges.

An exciting area of unusual scenarios opens up when so called L2-segments do not coincide with logical IP sub-nets.

Multiple different and originally separated virtual L2-segments with IPs of the same IP-subnet may be coupled by routing namespaces or (VLAN-aware) bridges/switches. On one side we may e.g. have a namespace with two attached standard L2-segments where untagged packets flow. What happens with ARP requests and replies in such a namespace?

On the other side NICs attached to one and the same L2-segment may belong to different IP sub-nets. Which on first sight appears to be a stupid mis-configuration; but such configurations may occur for a variety of reasons. What about ARP then?

What about VLANs across segments belonging to different logical IP sub-nets? Do such configurations make sense at all?

Termination of multiple VLANs via related sub-devices of veth endpoints in a network namespace

Two separated L2-segments may have NICs with IPs belonging to one and the same IP network class. The attentive reader of my 2017 series has of course noted that this was the case in all VLAN scenarios discussed at that time. Actually, this was the clue of the setups:

I used VLANs to isolate packet paths within one and the same IP sub-net ( a class C net) and within originally coherent L2-segments against each other. This worked out pretty well – as e.g. experiments with ICMP packets demonstrated.

However, I admit that I should have analyzed virtual veth connections which support multiple VLANs more thoroughly for some of my scenarios discussed in 2017. In particular a closer look at ARP-traffic in scenarios where a single veth-endpoint puts multiple VLAN-related sub-devices into one and the same target namespace (or into a bridge) would have been helpful. A discussion would probably have to avoid confusion of several readers regarding the impact of routes on ARP packets.

Before you think this topic may be boring, note that a veth endpoint device itself may get just one IP-address which it shares between all of its VLAN sub-interfaces and its trunk interface. See e.g. post. We will in addition find that a veth endpoint has just one MAC – which is also shared amongst all sub-interfaces. So the MAC/IP-tuple is not a unique identifier for any of a veth’s VLAN-interfaces or its neutral trunk interface. This is a major deviation from normal non-tagging NICs!

Thus, a VLAN-aware veth endpoint comes with multiple tagging NIC devices which cannot be identified by a MAC/IP-tuple alone. Somehow we (or more precisely some network namespaces) need additional rules to control the selection of a specific VLAN interface, such that our Ethernet packets get the right VLAN tag (VID) and propagate through the right VLAN to a target IP address.

In this context the coupling of network layer 2 (Link layer) to layer 3 (IP or network layer) is of special interest. Layer coupling by evaluating information about IP/MAC-relations is done by the ARP protocol. We all know that ARP is often used in hacker attacks. So it might be a good idea to know what happens with ARP in routing namespaces with multiple VLAN aware veth-end-points, a multitude of VLAN-interfaces and maybe a common NIC to reach real network segments. This is a major objective of this post series:

We want to thoroughly understand what ARP packets (requests and replies) do in the Linux network namespace with multiple available VLAN interfaces, but with just one IP/MAC-tuple shared between these interfaces.

In my old posts I speculated that ICMP and ARP answers in such unclear situations may depend on defined routes in the namespace. We shall find out in how far this is true by two corresponding experiments. As a contrast we will also look at situations where we give each of the VLAN-interfaces and individual IP address.

Connection of the virtual LAN to real LANs and the Internet

Scenarios with some ambiguities regarding packet paths are also of interest when traffic of different virtual VLANs must be directed to or through a common namespace which contains both VLAN end-points and a real NIC; the latter to enable communication to the outside world of a Linux virtualization host. Such a namespace would be a routing and forwarding one.

The connection of (virtual) VLANs to external real (physical) networks via namespaces that contain one or more real NICs were unfortunately not discussed in my old post series. (I had to work on project for a customer at this point). It is not at all clear whether we can separate the traffic between the VLANs and the outer world without the help of firewalls. In this context we may also have to look closer at the relation of VLANs to IP-subnets.

Bridges, iptables, VLANs and ARP

Readers have also sent me questions regarding VLAN-aware bridges and the propagation of ARP requests and ARP answer packets when IPtables rules control the packet traffic between bridge ports via “physdev“-related commands. What about tagged packets on a bridge with filtering IPtables rules?

Summary

Linux network namespaces, virtual Linux bridges and veth network devices make it possible to realize complex virtual network scenarios on a virtualization host – including virtual VLANs. Also virtual networks must be well protected against hacker attacks. Therefore, we should first understand packet transport through virtual (V)LAN devices and in particular across Linux bridges sufficiently well. Critical points are virtualized hosts or namespaces with multiple attached (V)LAN segments. Ambiguities regarding the packet path may come up, in particular in namespaces with multiple veh-based VLAN-interfaces.

Linux network namespaces and veths allow us to study packet transfer on different OSI layers in elementary network scenarios for L2-segments with and without virtualized VLANs in detail. The questions which remained open in my old post series and some new questions of readers invite us to study a bunch of further scenarios in a new post series.

In the next post

More fun with veth, network namespaces, VLANs – II – two L2-segments attached to a common network namespace

I will pose and study a first scenario without VLANs and respective tags. I will just attach veth end-points of two otherwise separated L2-segments to a common network namespace. Nevertheless, this very simple experiment will shed some light on open questions regarding routes, ARP and ICMP requests and answers. It will also lead us to aspects of PROXY ARP in (routing and forwarding) network namespaces.

KVM/Qemu VMs with a multi-screen Spice console – VIII – VM and user specific restrictions for remote-viewer connections – iptables and sudo

Posted on 10. May 2021 by eremo

The Spice console allows users to access the graphical desktop of a Qemu based virtual machines [VM]. The performance with both data encryption and data compression is excellent, audio is no problem and the required data transfer rates to client-applications as “remote-viewer” are within reasonable limits for a (switched) LAN. In virtualization scenarios where you can organize tasks according to a scheme “one user per VM” Spice is actually an attractive tool – and even professional virtualization environments a “Proxmoxx” and “oVirt” make use of it.

In this series we look at basic setups in self-administered Intranet environments. So far we had some ups and downs regarding the tool remote-viewer. If you want to use it in a desktop virtualization environment it is an almost perfect tool. We also found it to be a very convenient and efficient remote client-tool in an Intranet when we combined it with SSH and internal data compression of the Spice protocol. See:

KVM/Qemu VMs with a multi-screen Spice console – V – remote access via remote-viewer, a network port and a SSH-tunnel
KVM/Qemu VMs with a multi-screen Spice console – IV – remote access via SSH, remote-viewer and a Unix socket
KVM/Qemu VMs with a multi-screen Spice console – III – local access with remote-viewer via a Unix socket

SSH gave us all options we needed to take care of various security issues in remote access scenarios. The KVM/Qemu server could control the interaction of remote-users with VMs by applying user-specific SSH restrictions to port-forwarding. We could establish rules to bind the Spice access to a specific VM to a specific user.

Regarding the last point the combination of remote-viewer, TLS and SASL came as a solid disappointment. TLS worked perfectly. But we had problems with SASL; see:

KVM/Qemu VMs with a multi-screen Spice console – VII – remote-viewer, qemu and SASL authentication
KVM/Qemu VMs with a multi-screen Spice console – VI – remote access with remote-viewer and TLS encryption

We got SASL authentication working in two different ways. The frustrating result of our efforts, however, was that we could not confine the access to the Spice-console of a specific VM to exactly one user.

In the end we found that it might even be better to use TLS and a VM-specific Spice password set in the VM’s XML-domain file. The problem then still is that users could share the password for a specific VM. So, we still could not ensure a scenario “one VM – one UID, only”.

In this article we, therefore, look at measures on remote client-systems to allow a TLS connection to a specific VM on the KVM/Qemu host for exactly one user. We use two different Linux tools – namely iptables and sudo – to achieve this. The recipes given below are interesting in themselves and can be applied to other scenarios where the admin wants to restrict outgoing TCP-connection on a user specific level.

Why do we care about user-specific control for Spice at all?

Assume a situation
in a small Linux oriented office where you run a bunch of VMs for special purposes on a central KVM-server.
You provide e.g. a VM (VM1) with Windows 10 ( 🙁 ) for book-keeping with a program your tax advisor understands. A user “Charlie” with a defined UID should use the VM’s Spice console, but basically only he (and his boss in the evening hours).
You have a second VM (VM2) with a Web-server for development and test purposes. Only user “Maggie” shall have access to the Spice console of VM2 on Mondays and Tuesdays and a user “Anne” on Wednesdays and Thursdays and no one else.
And then there is yet another VM (VM3) which only user “Ralph” shall access via Spice to create documents of a confidential analysis for a governmental organization.
There are multiple Linux client PCs available in the office; any of the user can use and login to any free Linux workstations. None of the user has root rights on the clients and the server.

We have thus defined a scenario of the type

One VM + Spice console < = > one user or a group of selected users.

Can you cover such a situation with remote-viewer, TLS and server-based SASL, only?
The answer of my last article was: NO. At least not by simple means. (By the way: It is not the fault of SASL. Remote-viewer, simply and unfortunately, does not provide any system-based data for the realm or for the remote-system which SASL could evaluate. The user has too much freedom …)

Do we need measures on the client-systems?

It is interesting that even Opensuse’s “Virtualization Guide” writes about some required measures on the client (restricting access to specific client-certificates for selected users) when discussing the security for libvirt and vnc as tools to access the graphical desktop of a VM. We have not come to libvirt-based tools in this series, yet, but this is already some indication that restrictions of desktop access to KVM/Qemu based VMs may sometimes require measures on the client. So, let us turn to the client-systems … Of course, we assume that such clients have a Linux OS, in my case a Opensuse Leap 15.2.

Schematic drawing

The following schematic drawing reflects the situation we want to control:
We want to make it impossible user “mybro” to access the Spice console of a VM1. Only “myself” should be able to use VM1’s Spice console via a TLS connection. The other way round: Only user “mybro” shall be allowed to connect to the the Spice console of VM2 with remote-viewer. Cross-connections are forbidden; each user gets access to one defined VM only.

We use the same systems as in the last articles: A KVM/Qemu server host “MySRV” (IP: 192.168.2.2) with a Leap 15.2 OS on it, a test-VM1 “debianx” with a Kali-OS on it and a client-system “MyLAP” (a laptop with a Leap 15.2 OS; IP: 192.168.2.22). On MyLap we have a user “myself” and a user “mybro”. VM1 (“debianx”) gets the (TLS-) port 20002 on the server, and VM2 (“leaptest”) the (TLS-) port 20004.

We shall perform the experiments in this article with a Qemu set up without SASL, but each VM configured for an individual password. (As we already know it will only means a small difference in the form of the authentication dialog. We will not be asked for a username.) The scenario can easily be changed to SASL authentication.

Strategies to allow Spice access to a VM’s desktop only for a specific user

There are three major strategies we can follow:

Strategy 1: We could think about a specific TLS client certificate for a connection to a VM and to make it accessible only to the user in question. Such a
strategy would require that client certificates are supported by Qemu and Spice.

Strategy 2: We could use iptables and add user-specific rules for outgoing data to certain TLS-ports on the server. As user-related rules can only be set for the “output chain” we cannot set any such user-specific filters for our problem on the server.

Strategy 3: We take the choice of the target port on the server away from the user as well as any free access to the remote-viewer command namely by changing file permissions, setting up special sudo rules and/or enforcing him/her to run a shell script which starts remote-viewer with a pre-defined target port for him/her.

The strategies (if working) can be combined. Even if the firewall of strategy 2 failed the user would only get access to a specific VM by restrictions of strategy 3. But all these things are a bit complicated and they all depend on the user not being able to get root rights, the firewall being set up at every system-startup and the sudo rules being tight. This is crucial; our KVM/Qemu server with TLS and SASL could only block connections from certain IPs, but not really users.

We could extend strategy 3 to a network namespace – but it won’t help too much as we then would again need to apply routing and related firewall rules. Or to sudo rules. We could also think about using different networks for different VMs and thus a port-IP-VM-relation – but we would still depend on user-specific firewall rules on the clients.

Strategy 1: What about Spice and TLS client certificates?

Here I come with bad news:
If you activate the option “default_tls_x509_verify = 1” in “/etc/livirt/qemu.conf” it is simply ignored. One can guess it from a lack of required parameters in the qemu-command created by virt-manager. Even if and when you provide client certificates in the required directories on the server and the client. In contrast to VNC-settings there is no option like “spice_tls_x509_verify = 1” available. If you set it by yourself, it is silently ignored.

So, client-certificates will be of no help with Spice and remote-viewer. The situation my be different for the libvirt-dependent “virt-viewer” tool. But this is the topic of a future blog post.

Strategy 2: netfilter/iptables to the rescue

I assume that you have some tool in place on your Linux systems to configure your own iptables-rules. Professionals may write their own scripts. Independent of your interface to netfilter/iptables, it is obvious that working with firewall rules in a productive environment is a risky business. Therefore:

Disclaimer: I take no responsibility whatever for the consequences of the approach describes below and its application to your computers. The iptables-rules have to be tested carefully before making them productive and their setup on Linux hosts must be supervised by an expert.

In my network environments I still cling to the tool “fwbuilder” because it gives you a good graphical overview for most purposes and it allows for more complex configurations than e.g. frontends to firewalld or ufw. So, let me show you how to set up a basic set of user-specific iptables rules with fwbuilder and then have a closer look at the contents of the created statements of the firewall script.

fwbuilder allows you to set up “users” with a defined UID in the object catalog for “services”:

This may seem strange; but the meaning is the following:

Using a “user” as a “service” will trigger the creation of a rule for the OUTPUT chain which matches the user of outgoing connection
packets against an owner with a defined UID and then triggers a reaction.
To cover the reaction in fwbuilder we “branch off” into a specific rule-set for the user. On the iptables level this corresponds to a user-defined rule-chain as a reaction target.
The filter-rules for data packets in the user-defined chain themselves then lead to certain final reactions in the sense of “allow” or “deny”.

In pseudo-code: IF a user matches an UID THEN apply a set of filter rules with their own final reactions.

In Fwbuilder a set of filter-rules is usually put into an object called “policy”. The following image shows some rules of the main “policy” on “MyLap”:

You see that rules 4 and 5 include users “myself” and “mybro”; the rules branch off to policies “Spice_VM_1” and “Spice_VM_2”, respectively.

Rule 6 blocks a whole bunch of TCP-ports used for VMs on the KVM/Qemu server for any other user than “myself” and “mybro”.

Policy “Spice_VM_1” contains one rule, only:

This rule specifies a certain interface and the IP of the target host “mysrv”. We allow access to port 20002 – corresponding to the TLS port defined for our test VM “debianx” (VM1). Something analogous holds for our second user “mybro”. He gets access to another virtual machine “VM2” which we have bound to port 20004 on the KVM-server for external TLS connections.

Now, let us look at the iptables statements generated by fwbuilder:


    # ================ Table 'filter', automatic rules
    # accept established sessions
    $IPTABLES -A INPUT   -m state --state ESTABLISHED,RELATED -j ACCEPT 
    $IPTABLES -A OUTPUT  -m state --state ESTABLISHED,RELATED -j ACCEPT 
    $IPTABLES -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT 
    # backup ssh access from a admin host 
    $IPTABLES -A INPUT  -p tcp -m tcp  -s 192.168.44.12/255.255.255.255  --dport 22  -m state --state NEW,ESTABLISHED -j  ACCEPT 
    $IPTABLES -A OUTPUT  -p tcp -m tcp  -d 192.168.44.12/255.255.255.255  --sport 22  -m state --state ESTABLISHED,RELATED -j ACCEPT


    # ================ Table 'filter', rule set Spice_VM_1
    # 
    # Rule Spice_VM_1 0 (eth0)
    # 
    echo "Rule Spice_VM_1 0 (eth0)"
    # 
    $IPTABLES -N Spice_VM_1
    $IPTABLES -N Cid40091X30615.0
    $IPTABLES -A Spice_VM_1 -o eth0  -p tcp -m tcp  -d 192.168.2.2   --dport 20002  -m state --state NEW  -j Cid40091X30615.0
    $IPTABLES -A Cid40091X30615.0  -s 192.168.2.22   -j ACCEPT
    $IPTABLES -A Cid40091X30615.0  -s 192.168.2.22   -j ACCEPT
    
    # ================ Table 'filter', rule set Spice_VM_2
    # 
    # Rule Spice_VM_2 0 (eth0)
    # 
    echo "Rule Spice_VM_2 0 (eth0)"
    # 
    $IPTABLES -N Spice_VM_2
    $IPTABLES -N Cid40661X30615.0
    $IPTABLES -A Spice_VM_2 -o eth0  -p tcp -m tcp  -d 192.168.2.2   --dport 20004  -m state --state NEW  -j Cid40661X30615.0
    $IPTABLES -A Cid40661X30615.0  -s 192.168.2.22   -j ACCEPT
    $IPTABLES -A Cid40661X30615.0  -s 192.168.2.22 
  -j ACCEPT
    
    # ================ Table 'filter', rule set Main_Policy
    # 
    # Rule Main_Policy 0 (eth0)
    # 
    echo "Rule Main_Policy 0 (eth0)"
    # 
    # Antispoofing
    $IPTABLES -N In_Main_Policy_0
    $IPTABLES -A INPUT -i eth0   -s 192.168.2.22   -j In_Main_Policy_0
    $IPTABLES -A INPUT -i eth0   -s 192.168.4.22   -j In_Main_Policy_0
    $IPTABLES -A FORWARD -i eth0   -s 192.168.2.22   -j In_Main_Policy_0
    $IPTABLES -A FORWARD -i eth0   -s 192.168.4.22   -j In_Main_Policy_0
    $IPTABLES -A In_Main_Policy_0  -j LOG  --log-level info --log-prefix "RULE 0 -- DENY "
    $IPTABLES -A In_Main_Policy_0  -j DROP
    # 
    # Rule Main_Policy 1 (lo)
    # 
    echo "Rule Main_Policy 1 (lo)"
    # 
    $IPTABLES -A INPUT -i lo   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -o lo   -m state --state NEW  -j ACCEPT
    # 
    # Rule Main_Policy 2 (global)
    # 
    echo "Rule Main_Policy 2 (global)"
    # 
    #  ICMP, DNS, DHCP 
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 3  -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 0/0   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 8/0   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 11/0   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 11/1   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p tcp -m tcp  --dport 53  -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p udp -m udp  -m multiport  --dports 68,67,53  -m state --state NEW  -j ACCEPT
    # 
    # Rule Main_Policy 3 (global)
    # 
    echo "Rule Main_Policy 3 (global)"
    # 
    $IPTABLES -A OUTPUT -p udp -m udp  --dport 123  -m state --state NEW  -j ACCEPT
    # 
    # Rule Main_Policy 4 (global)
    # 
    echo "Rule Main_Policy 4 (global)"
    # 
    $IPTABLES -A OUTPUT -m owner --uid-owner 1021  -j Spice_VM_1
    # 
    # Rule Main_Policy 5 (global)
    # 
    echo "Rule Main_Policy 5 (global)"
    # 
    $IPTABLES -A OUTPUT -m owner --uid-owner 1022  -j Spice_VM_2
    # 
    # Rule Main_Policy 6 (global)
    # 
    echo "Rule Main_Policy 6 (global)"
    # 
    $IPTABLES -N Out_Main_Policy_6
    $IPTABLES -A OUTPUT -p tcp -m tcp  --dport 20001:20010  -j Out_Main_Policy_6
    $IPTABLES -A Out_Main_Policy_6  -j LOG  --log-level info --log-prefix "RULE 6 -- DENY "
    $IPTABLES -A Out_Main_Policy_6  -j DROP
    # 
...
...

These are simple rules which the experienced reader will have no difficulty to interpret. (The reader also sees that I have multiple IPs on eth0, but this does not affect our present topic).

I leave the test to you. You should see that user “myself” can access VM1 (debianx) whilst user “mybro” and any other users cannot. User “mybro” instead can access VM2 on the server.

Important note:

Do not forget to write a small systemd service which starts your firewall automatically during the startup of your client-system.

I have written about his topic somewhere else in this blog already. believe me its easy.

Strategy 3: Using sudo

Working with “sudo” and manipulating the file “/etc/sudoers” is somewhat risky. Therefore:

Disclaimer: I take no responsibility whatever for the consequences of the sudo approach describe below and its application to your computers. The sudoer rules have to be tested carefully before the are used in a production environment and their setup must be supervised by an expert.

The settings below work for an Opensuse Leap system – partially due to the default settings there.

A standard problem with sudoer rules for graphical applications is the handling of the access to the X11
display if you need to start programs as another user (e.g. root). (Things may be even worse with Wayland; but I have no experience with it). To keep things simple it is a worthwhile investment to think a bit about the precise nature of your sudo-objective.

In our case we want enforce a user-specific usage of the command remote-viewer. More precisely:

We want to disallow the usage of remote-viewer for most users. And even selected users shall not be able to call or invoke remote-viewer directly and freely.
We want to enforce user-specific arguments to the command “remote-viewer”, if executed by certain selected users.
If possible we do not want to run any remote-tool with root-rights at all.
We want to keep our user-specific firewall rules in place and not to create new ones.
X11-display access shall be possible.

The answer to this challenge is a bit tricky. It first looks like you need to write a separate script (accessible to root) which evaluates UIDs or SUD_UIDs and then calls remote-viewer with appropriate arguments.

But you will then be confronted with problems to access the X11-display of the user issuing the script. In addition rule 3 above forces you to start the required remote-viewer command in the end as a specific user via “sudo -u USER” or “su -c ‘…’ USER”. This in turn forces you to allow “USER” to execute the remote-viewer command anyway according to a specific sudo (!) rule for USER. Therefore, he/she must be able to read and execute the remote-viewer command. But then he/she could execute it outside freely without sudo – and again use any arguments he/she likes. It took me a while to find a way out.

The right approach is to find a rule working for the user in question, first, before you turn to a script as a mediator. On the other side: The user must NOT be allowed to use “remote-viewer” freely – neither by user or group access rights. This seeming contradiction is solved by the following steps on our client-system; we describe the rules below for user “myself”. You must execute most of the commands as root:

Step 1: Create a special group, e.g. named “spicegrp”. Do NOT make user “myself” a member of this group!
Step 2: Change the ownership and access rights of “/usr/bin/remote-viewer” according to
- chown root.spicegrp /usr/bin/remote-viewer
- chmod 750 /usr/bin/remote-viewer
Step 3: Check that remote-viewer can no longer be executed by user “myself”.
Step 4: Start editing the file “/etc/sudoers” with visudo.

Step 5: Add the following lines and turn some existing lines into comments:

....
Defaults env_reset
Defaults env_keep = "LANG LC_ADDRESS LC_CTYPE LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_ATIME LC_ALL LANGUAGE LINGUAS XDG_SESSION_COOKIE"

Defaults:myself env_keep += "DISPLAY"

#Defaults targetpw   # ask for the password of the target user i.e. root
#ALL   ALL=(ALL) ALL

myself ALL=(myself:spicegrp) /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002
# myself ALL=(myself:spicegrp) NOPASSWD: /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002
# mybro ALL=(mybro:spicegrp) NOPASSWD: /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20004

This is all we basically need for the user myself. An extension to
user “mybro” should be clear. See the commented line for mybro.

The sudoer statements – also the commented ones – deserve some short explanation:
We reset the environment, but we keep some language settings for sudoer users. More important: We keep the “DISPLAY” variable of the environment of sudoer user “myself”. Meaning, it will be available when commands are executed with his/her UID. Hereby, we avoid major X11 trouble. The two commented lines in the middle correspond to the the objective that sudoer users should provide their own passwords instead of the root password to execute the commands in the prescribed form.

And then comes some sudo vodoo:

We only define a rule for the user “myself”: We allow him/her to execute the command remote-viewer with his own UID, but with a different group. This gives him access to “/usr/bin/remote-viewer” under “sudo” conditions ( only !). But, in no way else on a shell’s command line, as he/her no longer has access rights to “/usr/bin/remote-viewer” and is no member of “spicegrp”!

The other part of the magic is that the sudoer-mechanism checks the precise form by which the user in question (here: myself) executes the command. It compares the command exactly to the form given in the rule line including the command’s arguments!

myself ALL=(myself:spicegrp) /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002

The command and the arguments to it are together handled as one string for comparison!
Thus we have a user “myself” who can only use remote-viewer with “sudo” and he/she is forced to provide a specific argument. And when he issues the command the firewall rules defined in the first part of this post should open doors to the VM as the execution is done with his/her UID!

What will the required sudo command for a successful access to the VM1 (debianx) at TLS-port 20002 on server MySRV look like:

sudo -u myself -g spicegrp remote-viewer — spice://mysrv.anraconc.de?tls-port=20002

You took notice of the argument for the group?

Test for a sudoer user

Let us test our theory (sorry for the German system messages, tried to translate them):

myself@mylap:~> remote-viewer -- spice://mysrv.anraconc.de?tls-port=20002
-bash: /usr/bin/remote-viewer: Keine Berechtigung
myself@mylap:~> # Keine Berechtigung = No access right
myself@mylap:~> # Now with sudo - but wrong port 20004
myself@mylap:~> sudo -u myself -g spicegrp remote-viewer -- spice://mysrv.anraconc.de?tls-port=20004
[sudo] Passwort für myself: 
Leider darf der Benutzer myself »/usr/bin/remote-viewer -- spice://mysrv.anraconc.de?tls-port=20004« als myself:spicegrp auf mylap.anraconc.de nicht ausführen.
myself@mylap:~> # Translation: Sorry, but user myself is not allowed to execute »/usr/bin/remote-viewer -- spice://beta.rux.anraconb.de?tls-port=20004« as myself:spicegrp on mylap.anraconc.de.
myself@mylap:~> # Now with sudo - but the right port 20002
myself@mylap:~> sudo -u myself -g spicegrp remote-viewer -- spice://mysrv.anraconc.de?tls-port=20002

And now we get the familiar dialog to authenticate :

Our firewall obviously has let us through. And – after filling in the VM-specific password (we work without SASL here; see above), we get:

Good!
I leave
it to the reader extend the whole thing to two users and to test all combinations out.

Making things a bit easier for the users

An authorized user has to type a lot of things to make the sudo command work. One thing, you can think about, is to not require a password for the sudo command. (Personally, I would not do this; a password is a last security barrier in case of configuration mistakes. But, in principle it is possible to work without a password in this specific case – after you have checked out and tested all security implications). The entry in the “/etc/sudoers” file then would look like

....
myself ALL=(myself:spicegrp) NOPASSWD: /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002
...

Another reduction of typing work comes through a script which readers can read and execute, but not change. The script could make all the relevant decisions for a user. A very simplified version would in my test scenario contain statements like:

#!/bin/bash
host="myserv.anraconc.de"
myself_port="20002"
mybro_port="20004"
Myself_UID="1021"
Mybro_UID="1022"
if  test $SUDO_UID
then
        if test  $UID -ne $SUDO_UID
        then
                echo "error - you are not allowed to run this command as another user"
                exit
        fi
fi
if  test $UID -eq $Myself_UID
then
        echo "Hello Ralph"
        msg="You are entering the Spice console of VM \"debianx\" - happy working"
        echo $msg
        sudo -u "#$UID" -g spicegrp /usr/bin/remote-viewer -- spice://$host?tls-port=$myself_port
elif  test $UID -eq $Mybro_UID
then
        echo "Hello Brother"
        msg4="You are entering the Spice console of VM \"leaptest\" - happy working"
        echo $msg4
        sudo -u "#$UID" -g spicegrp /usr/bin/remote-viewer -- spice://$host?tls-port=$mybro_port
else
        echo "Sorry, you are not allowed to access VMs"
fi

Note that the script would ask for a password before executing the sudo statement enclosed – if you had defined a password request in the sudoer file.
If you absolutely wanted to obfuscate the information contained in this script you could again use the trick with sudo and the special group spicegrp. You would the add lines

myself ALL=(myself:spicegrp) /usr/bin/rviewer
myself ALL=(mybro:spicegrp) /usr/bin/rviewer

to the “/etc/sudoers”-file

Important note:

You must not forget to check the fie permission of the file “/usr/bin/remote-viewer” after SW-updates or upgrades of your system.

I would recommend to start a small scheduled job or a service to check the rights settings frequently.

Conclusion

Remote-viewer connections to VMs cannot be controlled on a user level by the KVM/Qemu server if we just used TLS and SASL. We can set up a VM specific password. But external connections to a VM specific TLS port can only be blocked for external systems and IPs on the server.
However, on Linux client-systems iptables helps us to allow access to the Spice console of a specific VM for a selected user, only. This can be achieved by setting up user specific iptables rules on client-systems. This post has shown you how to create such rules for the OUTPUT chain with fwbuilder. We must set up a systemd service to implement these rules automatically at system (and network) startup.

To restrict users on Linux client systems even more we applied the sudo mechanism in a very specific way: we enforced the usage of certain arguments to the remote-viewer command for specific users. I think the method I discussed is safe; if you find a caveat please send me a mail.

Both strategies can and should be applied in Intranets where we want to provide remote-viewer and the Spice consoles of Qemu VMs as real working instruments.

KVM/Qemu VMs with a multi-screen Spice console – VII – remote-viewer, qemu and SASL authentication

Posted on 1. May 2021 by eremo

I continue with my exploration of remote access methods to the Spice console of a VM run on a KVM/Qemu-server with Opensuse Leap 15.2 as OS.

KVM/Qemu VMs with a multi-screen Spice console – VI – remote access with remote-viewer and TLS encryption
KVM/Qemu VMs with a multi-screen Spice console – V – remote access via remote-viewer, a network port and a SSH-tunnel
KVM/Qemu VMs with a multi-screen Spice console – IV – remote access via SSH, remote-viewer and a Unix socket
KVM/Qemu VMs with a multi-screen Spice console – III – local access with remote-viewer via a Unix socket
KVM/Qemu VMs with a multi-screen Spice console – II – local access with remote-viewer via a network port
KVM/Qemu VMs with a multi-screen Spice console – I – Overview over local and remote access methods

The topic of my last post was the combination of remote-viewer with “TLS” encryption. We configured libvirt such that virt-manager or virsh started the qemu-process for our test-VM with support for TLS/SSL. But: Remote-viewer itself does not use any libvirt-layer; regarding connections to a VM-specific network port, it is the qemu-emulator-process of the VM which does the encryption business for us on the server’s side – not libvirt.

In comparison to SSH-based methods discussed earlier a major disadvantage of the TLS-solution was the lack of any authentication on the server. The TLS-port for Spice on the server was open for anybody in our Intranet – not a nice situation for a “one seat” tool as Spice. We could, of course, have restricted access to certain client-systems by a firewall and by setting a general or a VM-specific password for the Spice console(s); but still there would not be any user-specific restrictions in place.

As a first improvement we would like to establish some server-side authentication as a condition for access to a Spice console. With SSH this was a peace of cake – and we could in addition set user-specific options for the SSH daemon to combine authentication with a user-specific access to the Spice console of a selected VM, only. Something similar is much harder to achieve with TLS – as long as we cling to libvirt-tools for starting VMs and remote-viewer as a client. I have not succeeded with it by simple means. Actually, I will need another article to describe some measures for real user specific restrictions – on the remote-client and, unfortunately, without reaching the same clear association of a user’s authentication on the server with the access to a specific VM. But one step after the other.

In this article I am going to discuss the combination of

remote-viewer, Qemu, TLS and SASL.

We first take a look at a most simple SASL authentication mechanisms based on a sasldb2-file which contains the credentials. One could also directly involve a Kerberos system by just changing some settings in prepared configuration
files. But Kerberos is beyond the scope of this series. Instead I want to show you how to configure the invocation of the saslauthd daemon as a general bridge to whatever authentication method you prefer in your network or on the server for special services. For reasons of simplicity we pick the local PAM machinery on the server as an example, but MySQL, LDAP or LDAP combined with Kerberos would be other options.

We are going to see that these two simple approaches to SASL based authentication – though working – come with a big disadvantage regarding Spice: The authentication is unfortunately global for all VMs. We won’t get a restriction to the Spice console of a specific VM with remote-viewer and SASL. And we do not really restrict the access to the console of a specific VM to a defined UID on the server.

But, let us first focus on the problem of getting SASL active in combination with remote-viewer. We work again with our test-VM “debianx” with a Kali system on it, a laptop MyLAP as a remote client-system with OS Leap 15.2 and a KVM/Qemu server, also with Opensuse Leap 15.2 as OS (see the previous articles of this series).

Schematic overview

The following drawing gives you an overview about the involved components on the KVM/Qemu server and on the remote client-system.

This is more or less the graphics from my last article with additional elements. I have indicated that both remote-viewer and the Qemu-hypervisor must be configured to use SASL in addition to OpenSSL. This in turn means that these components must support SASL and its challenge/response mechanisms. On the server-side we need a SASL compatible backend for keeping credential data and performing authentication against them. Some Qemu-configuration options allow for a direct and seamless access of certain backends – as sasldb2 or Kerberos. However, the “saslauthd” daemon with its manifold of connectors to different internal or external backends can be used as a mediator, too.

Note that we would create severe security holes if SASL related credentials or other authentication data were transferred between the client-system and the server without encryption. SASL is basically independent of OpenSSL-libraries. It provides its own methods (“mechanisms”) for data exchange – some with encryption, but some without. And saslauthd works with clear text data, only.

Therefore, any client-application using SASL is well advised to check that a TLS-tunnel really is in place if the used or available SASL mechanisms imply plain text transfer of challenge/response data. It is the sole responsibility of the applications to evaluate the circumstances. And remote-viewer indeed seems to take care ….

TLS was not really enforced by our settings … so far

I am lazy and I use libvirt tools to start qemu-based VMs. The TLS configuration is, therefore, in a way a global one for all VMs. It activates (potential) TLS support for the Spice console whether the VM in the end really offers a special TLS-port or not. In this case the global configuration has no consequences in the sense of TLS enforcement; we must still define a VM-specific TLS-port to really enable TLS for a chosen VM. And as long as a standard network port is defined for the VM and opened on the KVM/Qemu-server the usage of TLS is not enforced. A good reason for remote-viewer to be careful with regard to SASL.

SASL RPMs

We have to install packages for SASL on the server and also on the client. Opensuse provides SASL in form of “cyrus-sasl“. Required packages for our
tests are:
cyrus-sasl, cyrus-sasl-digestmd5, cyrus-sasl-plain, cyrus-sasl-saslauthd, cyrus-sasl-scram, libgsasl-lang, libgsasl7, libsasl2-3, libsas2-3-32bit, perl-Authen-SASL.

Some of these packages provide specific SASL “mechanisms” for the exchange of challenge/response data.

Basic preparations for Qemu and SASL on the KVM/Qemu-server

In previous articles we learned that remote-viewer accesses the Spice console of a VM directly via Qemu (and not libvirt). So we must prepare the start of a qemu-process for our VM such that parameters enable the use of SASL. As with TLS we use the file “/etc/libvirt/qemu.conf” for the configuration. For our present purposes we activate the following options:

spice_tls = 1
...
spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice"
...
spice_sasl = 1
...

You are already familiar with the TLS-related options. The only new point is the activation of SASL as the authentication mechanism for the Spice console. Folks who start qemu processes manually or with the help of scripts should read
https://www.spice-space.org/spice-user-manual.html
for the relevant options to add to the “/usr/bin/qemu-system-x86_64”-command. Spice related parameters may look similar to what I get in may case, namely

/usr/bin/qemu-system-x86_64 ….. -spice tls-port=20002,addr=0.0.0.0,sasl,disable-ticketing,x509-dir=/etc/pki/libvirt-spice,tls-channel=default,image-compression=auto_glz,seamless-migration=on ……

Note the location of the x509-directory; we got already acquainted with it it in the last article: It is used by qemu to read in TLS/SSL-certificates and key information which we must place there in advance, e.g for the CA and the server. If we started VMs directly with “qemu”-comands on a CLI we could change this directory individually for VMs. Using virt-manager or virsh forces us to define one x509 directory for all VMs, instead. Keep this in mind.

The commenting text in the file “/etc/libvirt/qemu.conf” tells you already that additional configuration steps are required in a file just below “/etc/sasl2/“. This directory gathers SASL configuration files for various services that make use of SASL.

On a Leap 15.2 system no special default configuration file for “qemu” is provided at “/etc/sasl2/”, but for “libvirt”. For our present purposes we just copy the existing file “/etc/sasl2/libvirt.conf” to a new file “/etc/sasl2/qemu.conf“. Why “qemu.conf”? Well, this is the instruction given in Opensuse’s virtualization documentation – and it seems completely logical. This may lead you to the assumption that the “service” in question always has to be qualified as “qemu” in relation to SASL. We shall later see that this assumption is wrong when we invoke the “saslauthd”-daemon …

For our first tests we activate the following two options, only, in the file “/etc/sasl2/qemu.conf” by un-commenting respective lines, out-commenting others and a minimum of editing:

mech_list:  plain scram-sha-256 
# mech_list:  plain scram-sha-256 digest-md5
..
sasldb_path: /var/spice/sasl/passwd.db
...

With the “mech_list” we obviously define a list of SASL mechanisms. We shall see how we specify which of these mechanism remote-viewer should use in a minute.

The mechanisms “plain” and “scram-sha-256” lead to an exchange of (salted) challenge/response data in clear text. They offer no real security for man-in-the-middle attacks in our Intranet. So, these mechanisms must be used in combination with a TLS tunnel, only. For possible attack-vectors see e.g.
https://tools.ietf.org/html/rfc7677
r
and
https://tools.ietf.org/html/rfc5802

The good message is: Remote-viewer will block the access to the Spice console of a VM if the connection is not secured by TLS/SSL!

Note the out-commented line with a crypto-mechanism specified in addition, namely “digest-md5“. We shall use this line for tests of situations where a standard connection without TLS is established or might be established as a fallback. We shall see that remote-viewer will pick “digest-md5” automatically if TLS is not guaranteed and if the mechanism is available.

However, “digest-md5” must itself be regarded as insecure due to its weak cryptography. In a productive environment you should simply forget it – and best eliminate it from your configuration! Let remote-viewer check for sufficient TLS conditions – and let it refuse access to the target VM if necessary.

The parameter sasldb_path specifies the path to sasldb2-files which should contain the credentials. In my settings the reader recognizes the directory below which I placed a folder for a pure Unix socket in local access scenarios (the previous articles). You may, however, specify whatever directory you like on your system. But be aware of required access rights; see below.

How to fill sasldb2 with credentials? What access rights are required?

Some basic information about the usage of sasldb2-files is given e.g. at
https://blog.sys4.de/cyrus-sasl-sasldb-konfigurieren-de.html.

According to the test settings above we place our sasldb-files into a folder “/var/spice/sasl/” (see below). Before you blindly experiment with adding users, consider some hints regarding the entries:

SASL distinguishes between users of the same name for different systems or application environments. Users have to be qualified with a user-name and a realm in the form

username@realm

Regarding the “user“:
It does not matter what usernames we choose; we can use fictitious users – they need not exist as valid users on the KVM/Qemu server (or the client-system) at all. However, as remote-viewer normally “suggests” the present username in a dialog for authentication (see below) it may be clever to add users with these suggested names – just to save some time with typing. In my test environment I work as user “myself” on the client “MyLAP”, but sometimes also as user “uvmb” on the server “MySRV”. So, I should have valid entries for both users.

Regarding the “realm“:
For a simple sasldb you can basically define as a realm whatever you want – but the user who wants to authenticate has to know what to provide. So he/she should be able to associate it with some meaning, e.g. the VM’s name. The “realm” is, unfortunately, NOT set or defined automatically later by the remote-viewer client and is in no way related to the contents of TLS certificates or target domain names presented during the authentication process. It can be freely set by the user in an SASL authentication dialog presented by remote-viewer: The user there can fill in anything after “username@…”. To become authenticated, however, what he/she fills in must match an entry in the sasldb-file.

Thus, we could define a fantasy user “tom@waits” with a password “the-piano-has-been-drinking”; it would work – as long as the remote-viewer user has all this information and provides it properly in the authentication dialog.

There is, however, one more rule to keep in mind during your experiments:

If the user provides just a username without any realm in the SASL authentication dialog then the hostname will be added automatically
on the server-side. And the hostname is the one in “/etc/hostname” on the server.

(The qemu-end of the SASL authentication does not seem to differentiate in cases where we use multiple IPs and different FQDN entries at a DNS server or in /etc/hosts for the host.)

In my test case the hostname is “myserv.anraconc.de”. But be careful: You may have the short-name (“mysrv”) standing in “/etc/hostname” on your Opensuse Leap 15.2 system! You should have a proper entry “myself@hostname” in your sasldb. If in doubt enter entries with both the short form and the FQDN of your KVM-host.

As root we now prepare a folder “/var/spice/sasl/” on the server (we have a folder “/var/spice/” already from previous efforts) and add some users with the help of the “saslpasswd2“-command :

mysrv:/var/spice # mkdir /var/spice/sasl
mysrv:/var/spice # chown root.qemu /var/spice/sasl
mysrv:/var/spice # chmod 750 /var/spice/sasl
mysrv:/var/spice # saslpasswd2  -a qemu myself@mysrv.anraconc.de -f /var/spice/sasl/passwd.db
Password: 
Again (for verification): 
mysrv:/var/spice # saslpasswd2  -a qemu myself -f /var/spice/sasl/passwd.db
...
mysrv:/var/spice # saslpasswd2  -a qemu uvmb@mysrv.anraconc.de -f /var/spice/sasl/passwd.db
...
mysrv:/var/spice # saslpasswd2  -a qemu uvmb -f /var/spice/sasl/passwd.db
...
mysrv:/var/spice # saslpasswd2  -a qemu xen -f /var/spice/sasl/passwd.db
...
mysrv:/var/spice # saslpasswd2  -a qemu xen@xensrv -f /var/spice/sasl/passwd.db
...

Afterward, you will find two new files in “/var/spice/sasl”:

mysrv:/var/spice/sasl # la 
mysrv:/var/spice/sasl # la
total 28
drwxr-x--- 2 root qemu  4096 Apr 18 18:09 .
drwxr-xr-x 3 root root  4096 Apr 18 18:07 ..
-rw------- 1 root root    16 Apr 18 18:09 passwd.db.dir
-rw------- 1 root root 23384 Apr 18 18:09 passwd.db.pag
mysrv:/var/spice/sasl #

Note the rights: The passwd-files can only be read by root. Why? Well, such a SASL file contains the credentials in clear text! It must be protected!

But, with these settings we would again run into a problem, which we have already seen in previous articles: The user of the qemu-processes is “qemu” on Leap 15.2-systems – at least when qemu-processes are started by libvirt-tools! This user “qemu” must be allowed to read critical files, in our present situation the “passwd.db”-files. Therefore, we change the group to “qemu” and the access rights to “640”:

mysrv:/var/spice/sasl # chown root.qemu pass*
mysrv:/var/spice/sasl # chmod 640 pass*
mysrv:/var/spice/sasl # la
total 28
drwxr-x--- 2 root qemu  4096 Apr 18 18:09 .
drwxr-xr-x 3 root root  4096 Apr 18 18:07 ..
-rw-r----- 1 root qemu    16 Apr 18 18:09 passwd.db.dir
-rw-r----- 1 root qemu 16384 Apr 18 18:09 passwd.db.pag
mysrv:/var/spice/sasl #

Warning: Please check that the qemu-group contains the user “qemu”, ONLY !

To check which users are enlisted in our backend we use the command “sasldblistusers2“.

mysrv:/var/spice/sasl # sasldblistusers2 -f /var/spice/sasl/passwd.db 
xen@xensrv: userPassword
xen@mysrv: userPassword
uvmb@mysrv: userPassword
uvmb@mysrv.anraconc.de: userPassword
myself@mysrv: userPassword
myself@mysrv.anraconc.de: userPassword
mysrv:/var/spice/sasl #

You see that users provided without a system realm automatically got the present system’s name attached!

If we wanted to delete a user we would use:

mysrv:/var/spice/sasl # saslpasswd2  -d xen@xen -f /var/spice/sasl/passwd.db
mysrv:/var/spice/sasl # sasldblistusers2 -f /var/spice/sasl/passwd.db 
xen@mysrv: userPassword
uvmb@mysrv: userPassword
uvmb@mysrv.anraconc.de: userPassword
myself@mysrv: userPassword
myself@mysrv.anraconc.
de: userPassword
mysrv:/var/spice/sasl #

For tests you should vary the passwords of usernames defined for different systems.

Indications of basic security problems

As long as a standard TCP port is defined for Spice fallback-situations may occur without any announcement. But remote-viewer checks whether the tunnel really is TLS secured or not. If not, it will automatically try to choose a SASL mechanism with encryption – as e.g. digest-md5 – if such a mechanism is available to the qemu-process on the server. If such a mechanism is not available remote-viewer will block the access to the Spice console:

A typical indication that something is wrong is a decline to enter a username in the dialog window for authentication opened by remote-viewer.

Problematic SASL mechanisms of course are PLAIN and all SCRAM-mechanisms. A noteworthy fact about remote-viewer is:

remote-viewer will try to use a SASL mechanism with encryption even if you have explicitly specified something else!

Problematic situation arise for a remote-viewer command (executed locally or on a real remote client) which contains the specification of a standard non-TLS port and a TLS-port – e.g.:

remote-viewer spice://myserv.anraconc.de:20001?tls-port=20002

for a VM with a Spice configuration that includes a standard non-TLS port and “defaultMode” set to “insecure” or “any”. The same is true for:

remote-viewer spice://myserv.anraconc.de:20001?tls-port=20002&sasl=plain

If remote-viewer in its background dialog with the server does not find a “worthy” mechanism it will give you an error hint:

Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)

Spice configuration to be used with plain text SASL mechanisms

To be on the safe side we configure Spice in the XML-definition file for a VM as follows:

   <graphics type='spice' tlsPort='20002' autoport='no' listen='0.0.0.0' keymap='de' defaultMode='secure' >
      <listen type='address' address='0.0.0.0'/>
      <image compression='auto_glz'/>
      <gl enable='no'/>
    </graphics>

The difference in comparison to specifications in previous articles is that we do not specify a standard “port” at all – but still have “autoport=’no’” in place AND set “defaultMode=’secure'”.

How do we specify a SASL mechanism when starting remote-viewer?

The general form is as simple as this

remote-viewer spice://FQDN??tls-port=NR_OF_TLS_PORT&sasl=MECHANISM

where “FQDN” refers to the address of the server, “NR_OF_TLS_PORT” to the tlsPort settings for Spice in the VM’s configuration and “MECHANISM” to one of the SASL mechanisms in the list of the “/etc/sasl2/qemu.config”-file.

Remote test

We are confident and test our present settings. We restart libvirtd on the server. A privileged user “uvma” starts the test VM there for us. Then we login into the remote system – in my case as user “myself” into “MyLAP”. On a terminal we enter:

myself@mylap:~> remote-viewer spice://mysrv.anraconc.de?tls-port=20002&sasl=plain

We then get an authentication dialog:

We enter the password defined in the credentials file – in my case for “myself@mysrv.anraconc.de” and not for “myself@mysrv” (!!) – and there we go:

After a login we can work with the desktop of the VM as described in previous articles and open up more screens.

Tests for realm dependence of the authentication

We are now able to test the dependency of the authentication on the “realm”; we just type in the “username@realm” which we want to test and provide the related password. Try also cases without providing a “realm”. I leave it to you to perform such tests.

Afterward it should be clear that the FQDN shown in the authentication dialog is important for the validity of the TLS server certificate – but it has nothing to do with the SASL realms!

A dis-functionality: Freezing Spice windows if the Spice session is taken by another user

Now you could try to do the same shortly after and locally on the server “mysrv” – i.e. you try to steal the “seat” in front of the Spice console. In previous articles we have seen that the Spice windows of the user, from which the seat was taken, were just closed without any warning. This was not the case on my systems with TLS and SASL. Instead and unfortunately the Spice window of the original user froze; he/she could not enter anything anymore. One could not even close it explicitly without killing the job.

Otherwise the handling of an active Spice session is as we are used to.

What do we get for situations with TLS not fully enforced?

Just for testing purposes we change the security situation by adding a standard port to the Spice configuration and set the defaultMode in the VM’s XML-configuration to ‘any’:

   <graphics type='spice' port='20001' tlsPort='20002' autoport='no' listen='0.0.0.0' keymap='de' defaultMode='any' >
      ...
    </graphics>

Then a dubious situation as provoked by

myself@mylap:~> remote-viewer spice://mysrv.anraconc.de:20001?tls-port=20002&sasl=plain

The TLS-option is ignored in this case and this in turn leads to a decline to enter a user@realm in the first field of the dialog:

accompanied by a warning on the terminal window :

(remote-viewer:21318): GSpice-CRITICAL **: 17:33:36.750: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)

However, if you now change the “mech_list” for SASL in “/etc/sasl2/qemu.config” such that it includes “digest-md5”, then

myself@mylap:~> remote-viewer spice://mysrv.anraconc.de:20001?tls-port=20002

would work. And also – somewhat unexpected – the following works:

myself@mylap:~> remote-viewer spice://mysrv.anraconc.de:20001?tls-port=20002&sasl=plain

Wireshark, however, reveals that the specified sasl-mechanism is ignored and digest-md5 is taken in this case:

Well, this is a bit confusing as remote-viewer does not inform the user about the measures taken. My advice is to prepare for clarity: If you want to work with SASL then disable the standard non-TLS port for Spice and do not include weak SASL crypto-mechanisms in the mech-list at all. It is better that remote-viewer
stops the conversation ….

Failure for local access to the Spice console from virt-manager

There is a problem with our qemu-setting that enforces SASL and virt-manager. Afterwards we can no longer access the Spice console locally on the KVM/Qemu-server with the built-in Spice client of virt-manager. We get a screen that wants a password from us – but no user and realm:

You can fill in whatever you want of known passwords for root, qemu and other users which you may have filled in into your sasldb. (You could have added an entry for “root@mysrv.anraconc.de” with some password, because virt-manager runs with user root). It is not going to work.

There seems to be a basic problem; we get a log message as

  
2021-04-30T17:05:20.745776Z qemu-system-x86_64: warning: Spice: red-stream.c:1039:red_sasl_start_auth: sasl context setup failed -7 (invalid parameter supplied)

The funny thing is that another libvirt-dependent tool, namely “virt-viewer” works correctly with SASL. You can test it locally on the KVM/Qemu-server, e.g. for the following situation:

Allow for a standard Spice port without TLS (in our test case 20001; in our series we have not yet come to the point where we use libvirt-tools remotely together with TLS 🙂 ).
Let Spice listen on all on all network devices (including 127.0.0.1).
Activate SASL for qemu (via settings in “/etc/libvirt/qemu.conf”; see above).
Allow for a SASL mechanism with encryption.

If you then enter e.g. the command

mysrv:~ # virt-viewer -c qemu:///system

(e.g. as user root) an authentication dialog opens, where we can fill in a username (“root” is suggested) plus a realm, if we want, and a valid password for the combination.

Enter a valid password for some entry (e.g. for root@mysrv.anraconc.de”) in the sasldb and the Spice console comes up as usual. So – there is some dis-functionality in virt-viewer, when SASL is activated for qemu (and not a libvirt port or local libvirt Unix socket).

Switching to the saslauthd daemon

“/etc/sasl2/qemu.config” was copied from a libvirt-oriented file. We can, of course, specify other SASL rules for the service “qemu”. In particular we can invoke a “saslauthd” daemon. We achieve this by changing the settings in “/etc/sasl2/qemu.config” to:

mech_list:  plain scram-sha-256
pwcheck_method: saslauthd

Please out-comment other lines. Why did I only specify clear text mechanisms? Well, saslauthd simply does not work with crypto-mechanisms! It is not designed for it. It is invoked locally on our KVM server where it listens on a Unix socket after having been started:

mysrv:~ # netstat -ax | grep sasl
unix  2      [ ACC ]     STREAM     LISTENING     603842   /run/sasl2//mux

To get that far there are a few more things to do, of course: We must configure saslauthd and start the related service. The basic configuration of “saslauthd” on a Leap 15.2 system is done
via settings in the file “/etc/sysconfig/saslauthd“.

For a simple scenario, which uses the standard PAM mechanisms on the server, we can leave the default settings there as they are:

SASLAUTHD_AUTHMECH=pam
SASLAUTHD_THREADS=5
SASLAUTHD_PARAMS=""

What is the expected effect? Well, afterward, we should be able to authenticate ourselves with credentials (usernames and passwords) already defined for valid accounts on the server (e.g. in “/etc/passwd” and “/etc/shadow”). Note:

No login shell is required for our special Spice users!

So, we can create some (harmless) user accounts on the server to test the Spice/SASL authentication (with some preset complicated passwords).

Before testing we should not forget to start the daemon by “systemctl start saslauthd” or “rcsaslauthd start“. But before we do this there is one more step required ….

PAM configuration to support authentication for Spice by saslauthd

PAM works for services – and what PAM should do for a service has to be defined. The configuration files are placed in “/etc/pam.d/“. Naively, I had expected that we would need a “qemu” configuration file there. Well this is wrong. I could see it from resulting error messages in “/var/log/messages”

... saslauthd[25571]:                 : auth failure: [user=myself] [service=spice] [realm=] [mech=pam] [reason=PAM auth error]

The “service” to be configured is “spice” ! Ok, the machine gets what it wants. I created a simple file “/etc/pam.d/spice” with the following standard contents:

#%PAM-1.0
auth     requisite      pam_nologin.so
auth     include        common-auth
account  include        common-account
password include        common-password
session  required       pam_loginuid.so
session  include        common-session

This is enough to support our SASL operations; some lines as e.g. for the password could even have been omitted for our present objectives. Note hat a “realm” does not make much sense if we work with a local PAM for Linux password files on a Linux host, only, and no external backend.

Test of salsauthd

Stop your VM if it were running. Restart libvirtd, start saslauthd, start the VM.
You can pre-test your saslauthd configuration for the service “spice” with existing user accounts on the server with the command

mysrv:~ # testsaslauthd  -s spice -u uvmb -p ******** 
0: OK "Success."

You must of course replace “uvmb” without a username defined on your server and the “******” with the related password. .

Now you can try it for a remote-viewer session. The same authentication dialog we got before for our sasldb-experiment should appear. You then only enter a username (no realm) into the filed for the username. Should work without any problems.

Major deficits of the SASL-authentication for remote-viewer

During his/her tests of the authentication with sasldb2 the reader has certainly realized the following point:

Any valid combination of a username, a related realm and a password authorizes us to get access to the Spice console of our test-VM.

There is no unique relation between a UID on the remote-client or on the server and the “username@realm” combination. And there is no real relation of a username/user with a specific VM. Therefore, the situation is actually worse:

A user providing a valid “username@realm” (existing in sasldb2) has access to the Spice consoles of all VMs running on the server!

Activating SASL in the general way we did it for the startup of all qemu-processes
establishes an authentication and authorization mechanism for the Spice consoles of all VMs on a KVM server – if the user knows the relevant TLS-ports or just checks out ports systematically.

Obviously, the logic for our present setup is that we authenticate a general Spice user via SASL on the server. We cannot establish a relation between the Spice related SASL “username@realm” and a specific VM. The present version of remote-viewer does not provide other data (as an rhost) which we could e.g. use in some cleverly crafted PAM rules for special modules. (You can see the information provided from log-entries for your VM and wrong authentications)

In my opinion an authorization should be specific for a user (or a group of selected users) and a specific target service. And it should be such that the server alone can control the service access. For a bunch of VMs on the KVM/Qemu-server we would assume that we could establish a specific user authentication for the Spice console of a specific VM and no other VMs.

So, it is very unfortunate that “realm” delivered by remote-viewer can freely be set by the user and that it is independent of the TLS parameters and/or data of the remote-host. Thus, there is no way to force the user on the client-system to enter a VM-specific user@realm combination. And any other user sharing one valid username/password combination known to him opens up access to all Spice consoles for other users – if no additional measures were taken.

If you wonder how a coupling of certificate data to the realm would have helped: We could have achieved a relation between a realm and a VM, if we had coupled the VM to a sub-domain for a wildcard certificate on the server and to a specific IP on the server – and if the realm were taken from the “common name” (i.e. the FQDN) in the certificate. Then we had just needed a specific username@ realm entry in the sasldb – and we would have gotten a VM-specific authentication. In the case of saslauthd the delivery of information about the remote host would have helped. I leave it to the user to figure that out. Its fruitless thinking anyway, …

Actually, setting a password int VM’s XML file would have given us a VM-specific authentication. Unfortunately, we are not allowed to combine the setting of a VM-specific password in the XML-file in addition to SASL-authentication. The VM can not be started by virt-manager or virsh then. So, it is a bit of a mess with SASL and remote-viewer …

Are there valid counter arguments?
Well, you could argue that we could hinder access to specific VMs by blocking ports on the server for certain client-systems. Well, and what for situations where multiple users are allowed to login to any client-machine in an office?
And you could argue: But the user has certainly to login into the VM as well? Yeah, but there is the problem with Spice – if someone legitimately had already opened a a session on the VM via a Spice console this session could be taken from him/her in the present open status by an other user authenticated via SASL for some other entry in the sasldb. Security is something else …

Conclusion

To activate SASL for some general authentication to get access to the Spice console of a chosen VM is simple. However, the SASL authentication does not allow us

to restrict the access to the Spice console of a specific VM to a specific username@realm combination.
to restrict the access to the Spice console to a specific UID on either the remote client or the KVM-server.

So, TLS in combination with SASL is to a certain degree a disappointment. A VM-specific password has more advantages. Note, that for present versions of “qemu” it is not displayed anymore in the output of “ps aux | grep qemu”.

The combination of TLS and SASL with remote-viewer is not really useful for situations where we want Intranet users to
work with VMs via Spice sessions – each of them with a (user-) specific VM or some (user-) specific VMs, only.

If you wanted to make Spice sessions available to a selected user only for specific VMs we thus end up with the question: How can we become more restrictive?

In the next article

KVM/Qemu VMs with a multi-screen Spice console – VIII – VM and user specific restrictions for remote-viewer connections – iptables and sudo

I will therefore discuss the application of user specific iptables- and sudo-rules on remote client-systems. Not ideal – but a step forward.