KVM/Qemu VMs with a multi-screen Spice console – VIII – VM and user specific restrictions for remote-viewer connections – iptables and sudo

The Spice console allows users to access the graphical desktop of a Qemu based virtual machines [VM]. The performance with both data encryption and data compression is excellent, audio is no problem and the required data transfer rates to client-applications as “remote-viewer” are within reasonable limits for a (switched) LAN. In virtualization scenarios where you can organize tasks according to a scheme “one user per VM” Spice is actually an attractive tool – and even professional virtualization environments a “Proxmoxx” and “oVirt” make use of it.

In this series we look at basic setups in self-administered Intranet environments. So far we had some ups and downs regarding the tool remote-viewer. If you want to use it in a desktop virtualization environment it is an almost perfect tool. We also found it to be a very convenient and efficient remote client-tool in an Intranet when we combined it with SSH and internal data compression of the Spice protocol. See:

KVM/Qemu VMs with a multi-screen Spice console – V – remote access via remote-viewer, a network port and a SSH-tunnel
KVM/Qemu VMs with a multi-screen Spice console – IV – remote access via SSH, remote-viewer and a Unix socket
KVM/Qemu VMs with a multi-screen Spice console – III – local access with remote-viewer via a Unix socket

SSH gave us all options we needed to take care of various security issues in remote access scenarios. The KVM/Qemu server could control the interaction of remote-users with VMs by applying user-specific SSH restrictions to port-forwarding. We could establish rules to bind the Spice access to a specific VM to a specific user.

Regarding the last point the combination of remote-viewer, TLS and SASL came as a solid disappointment. TLS worked perfectly. But we had problems with SASL; see:

KVM/Qemu VMs with a multi-screen Spice console – VII – remote-viewer, qemu and SASL authentication
KVM/Qemu VMs with a multi-screen Spice console – VI – remote access with remote-viewer and TLS encryption

We got SASL authentication working in two different ways. The frustrating result of our efforts, however, was that we could not confine the access to the Spice-console of a specific VM to exactly one user.

In the end we found that it might even be better to use TLS and a VM-specific Spice password set in the VM’s XML-domain file. The problem then still is that users could share the password for a specific VM. So, we still could not ensure a scenario “one VM – one UID, only”.

In this article we, therefore, look at measures on remote client-systems to allow a TLS connection to a specific VM on the KVM/Qemu host for exactly one user. We use two different Linux tools – namely iptables and sudo – to achieve this. The recipes given below are interesting in themselves and can be applied to other scenarios where the admin wants to restrict outgoing TCP-connection on a user specific level.

Why do we care about user-specific control for Spice at all?

Assume a situation
in a small Linux oriented office where you run a bunch of VMs for special purposes on a central KVM-server.
You provide e.g. a VM (VM1) with Windows 10 ( 🙁 ) for book-keeping with a program your tax advisor understands. A user “Charlie” with a defined UID should use the VM’s Spice console, but basically only he (and his boss in the evening hours).
You have a second VM (VM2) with a Web-server for development and test purposes. Only user “Maggie” shall have access to the Spice console of VM2 on Mondays and Tuesdays and a user “Anne” on Wednesdays and Thursdays and no one else.
And then there is yet another VM (VM3) which only user “Ralph” shall access via Spice to create documents of a confidential analysis for a governmental organization.
There are multiple Linux client PCs available in the office; any of the user can use and login to any free Linux workstations. None of the user has root rights on the clients and the server.

We have thus defined a scenario of the type

One VM + Spice console    < = >    one user or a group of selected users.

Can you cover such a situation with remote-viewer, TLS and server-based SASL, only?
The answer of my last article was: NO. At least not by simple means. (By the way: It is not the fault of SASL. Remote-viewer, simply and unfortunately, does not provide any system-based data for the realm or for the remote-system which SASL could evaluate. The user has too much freedom …)

Do we need measures on the client-systems?

It is interesting that even Opensuse’s “Virtualization Guide” writes about some required measures on the client (restricting access to specific client-certificates for selected users) when discussing the security for libvirt and vnc as tools to access the graphical desktop of a VM. We have not come to libvirt-based tools in this series, yet, but this is already some indication that restrictions of desktop access to KVM/Qemu based VMs may sometimes require measures on the client. So, let us turn to the client-systems … Of course, we assume that such clients have a Linux OS, in my case a Opensuse Leap 15.2.

Schematic drawing

The following schematic drawing reflects the situation we want to control:
We want to make it impossible user “mybro” to access the Spice console of a VM1. Only “myself” should be able to use VM1’s Spice console via a TLS connection. The other way round: Only user “mybro” shall be allowed to connect to the the Spice console of VM2 with remote-viewer. Cross-connections are forbidden; each user gets access to one defined VM only.

We use the same systems as in the last articles: A KVM/Qemu server host “MySRV” (IP: with a Leap 15.2 OS on it, a test-VM1 “debianx” with a Kali-OS on it and a client-system “MyLAP” (a laptop with a Leap 15.2 OS; IP: On MyLap we have a user “myself” and a user “mybro”. VM1 (“debianx”) gets the (TLS-) port 20002 on the server, and VM2 (“leaptest”) the (TLS-) port 20004.

We shall perform the experiments in this article with a Qemu set up without SASL, but each VM configured for an individual password. (As we already know it will only means a small difference in the form of the authentication dialog. We will not be asked for a username.) The scenario can easily be changed to SASL authentication.

Strategies to allow Spice access to a VM’s desktop only for a specific user

There are three major strategies we can follow:

Strategy 1: We could think about a specific TLS client certificate for a connection to a VM and to make it accessible only to the user in question. Such a
strategy would require that client certificates are supported by Qemu and Spice.

Strategy 2: We could use iptables and add user-specific rules for outgoing data to certain TLS-ports on the server. As user-related rules can only be set for the “output chain” we cannot set any such user-specific filters for our problem on the server.

Strategy 3: We take the choice of the target port on the server away from the user as well as any free access to the remote-viewer command namely by changing file permissions, setting up special sudo rules and/or enforcing him/her to run a shell script which starts remote-viewer with a pre-defined target port for him/her.

The strategies (if working) can be combined. Even if the firewall of strategy 2 failed the user would only get access to a specific VM by restrictions of strategy 3. But all these things are a bit complicated and they all depend on the user not being able to get root rights, the firewall being set up at every system-startup and the sudo rules being tight. This is crucial; our KVM/Qemu server with TLS and SASL could only block connections from certain IPs, but not really users.

We could extend strategy 3 to a network namespace – but it won’t help too much as we then would again need to apply routing and related firewall rules. Or to sudo rules. We could also think about using different networks for different VMs and thus a port-IP-VM-relation – but we would still depend on user-specific firewall rules on the clients.

Strategy 1: What about Spice and TLS client certificates?

Here I come with bad news:
If you activate the option “default_tls_x509_verify = 1” in “/etc/livirt/qemu.conf” it is simply ignored. One can guess it from a lack of required parameters in the qemu-command created by virt-manager. Even if and when you provide client certificates in the required directories on the server and the client. In contrast to VNC-settings there is no option like “spice_tls_x509_verify = 1” available. If you set it by yourself, it is silently ignored.

So, client-certificates will be of no help with Spice and remote-viewer. The situation my be different for the libvirt-dependent “virt-viewer” tool. But this is the topic of a future blog post.

Strategy 2: netfilter/iptables to the rescue

I assume that you have some tool in place on your Linux systems to configure your own iptables-rules. Professionals may write their own scripts. Independent of your interface to netfilter/iptables, it is obvious that working with firewall rules in a productive environment is a risky business. Therefore:

Disclaimer: I take no responsibility whatever for the consequences of the approach describes below and its application to your computers. The iptables-rules have to be tested carefully before making them productive and their setup on Linux hosts must be supervised by an expert.

In my network environments I still cling to the tool “fwbuilder” because it gives you a good graphical overview for most purposes and it allows for more complex configurations than e.g. frontends to firewalld or ufw. So, let me show you how to set up a basic set of user-specific iptables rules with fwbuilder and then have a closer look at the contents of the created statements of the firewall script.

fwbuilder allows you to set up “users” with a defined UID in the object catalog for “services”:

This may seem strange; but the meaning is the following:

  1. Using a “user” as a “service” will trigger the creation of a rule for the OUTPUT chain which matches the user of outgoing connection
    packets against an owner with a defined UID and then triggers a reaction.
  2. To cover the reaction in fwbuilder we “branch off” into a specific rule-set for the user. On the iptables level this corresponds to a user-defined rule-chain as a reaction target.
  3. The filter-rules for data packets in the user-defined chain themselves then lead to certain final reactions in the sense of “allow” or “deny”.

In pseudo-code: IF a user matches an UID THEN apply a set of filter rules with their own final reactions.

In Fwbuilder a set of filter-rules is usually put into an object called “policy”. The following image shows some rules of the main “policy” on “MyLap”:

You see that rules 4 and 5 include users “myself” and “mybro”; the rules branch off to policies “Spice_VM_1” and “Spice_VM_2”, respectively.

Rule 6 blocks a whole bunch of TCP-ports used for VMs on the KVM/Qemu server for any other user than “myself” and “mybro”.

Policy “Spice_VM_1” contains one rule, only:

This rule specifies a certain interface and the IP of the target host “mysrv”. We allow access to port 20002 – corresponding to the TLS port defined for our test VM “debianx” (VM1). Something analogous holds for our second user “mybro”. He gets access to another virtual machine “VM2” which we have bound to port 20004 on the KVM-server for external TLS connections.

Now, let us look at the iptables statements generated by fwbuilder:

    # ================ Table 'filter', automatic rules
    # accept established sessions
    # backup ssh access from a admin host 
    $IPTABLES -A INPUT  -p tcp -m tcp  -s  --dport 22  -m state --state NEW,ESTABLISHED -j  ACCEPT 
    $IPTABLES -A OUTPUT  -p tcp -m tcp  -d  --sport 22  -m state --state ESTABLISHED,RELATED -j ACCEPT

    # ================ Table 'filter', rule set Spice_VM_1
    # Rule Spice_VM_1 0 (eth0)
    echo "Rule Spice_VM_1 0 (eth0)"
    $IPTABLES -N Spice_VM_1
    $IPTABLES -N Cid40091X30615.0
    $IPTABLES -A Spice_VM_1 -o eth0  -p tcp -m tcp  -d   --dport 20002  -m state --state NEW  -j Cid40091X30615.0
    $IPTABLES -A Cid40091X30615.0  -s   -j ACCEPT
    $IPTABLES -A Cid40091X30615.0  -s   -j ACCEPT
    # ================ Table 'filter', rule set Spice_VM_2
    # Rule Spice_VM_2 0 (eth0)
    echo "Rule Spice_VM_2 0 (eth0)"
    $IPTABLES -N Spice_VM_2
    $IPTABLES -N Cid40661X30615.0
    $IPTABLES -A Spice_VM_2 -o eth0  -p tcp -m tcp  -d   --dport 20004  -m state --state NEW  -j Cid40661X30615.0
    $IPTABLES -A Cid40661X30615.0  -s   -j ACCEPT
    $IPTABLES -A Cid40661X30615.0  -s 
    # ================ Table 'filter', rule set Main_Policy
    # Rule Main_Policy 0 (eth0)
    echo "Rule Main_Policy 0 (eth0)"
    # Antispoofing
    $IPTABLES -N In_Main_Policy_0
    $IPTABLES -A INPUT -i eth0   -s   -j In_Main_Policy_0
    $IPTABLES -A INPUT -i eth0   -s   -j In_Main_Policy_0
    $IPTABLES -A FORWARD -i eth0   -s   -j In_Main_Policy_0
    $IPTABLES -A FORWARD -i eth0   -s   -j In_Main_Policy_0
    $IPTABLES -A In_Main_Policy_0  -j LOG  --log-level info --log-prefix "RULE 0 -- DENY "
    $IPTABLES -A In_Main_Policy_0  -j DROP
    # Rule Main_Policy 1 (lo)
    echo "Rule Main_Policy 1 (lo)"
    $IPTABLES -A INPUT -i lo   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -o lo   -m state --state NEW  -j ACCEPT
    # Rule Main_Policy 2 (global)
    echo "Rule Main_Policy 2 (global)"
    #  ICMP, DNS, DHCP 
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 3  -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 0/0   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 8/0   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 11/0   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 11/1   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p tcp -m tcp  --dport 53  -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p udp -m udp  -m multiport  --dports 68,67,53  -m state --state NEW  -j ACCEPT
    # Rule Main_Policy 3 (global)
    echo "Rule Main_Policy 3 (global)"
    $IPTABLES -A OUTPUT -p udp -m udp  --dport 123  -m state --state NEW  -j ACCEPT
    # Rule Main_Policy 4 (global)
    echo "Rule Main_Policy 4 (global)"
    $IPTABLES -A OUTPUT -m owner --uid-owner 1021  -j Spice_VM_1
    # Rule Main_Policy 5 (global)
    echo "Rule Main_Policy 5 (global)"
    $IPTABLES -A OUTPUT -m owner --uid-owner 1022  -j Spice_VM_2
    # Rule Main_Policy 6 (global)
    echo "Rule Main_Policy 6 (global)"
    $IPTABLES -N Out_Main_Policy_6
    $IPTABLES -A OUTPUT -p tcp -m tcp  --dport 20001:20010  -j Out_Main_Policy_6
    $IPTABLES -A Out_Main_Policy_6  -j LOG  --log-level info --log-prefix "RULE 6 -- DENY "
    $IPTABLES -A Out_Main_Policy_6  -j DROP

These are simple rules which the experienced reader will have no difficulty to interpret. (The reader also sees that I have multiple IPs on eth0, but this does not affect our present topic).

I leave the test to you. You should see that user “myself” can access VM1 (debianx) whilst user “mybro” and any other users cannot. User “mybro” instead can access VM2 on the server.

Important note:

Do not forget to write a small systemd service which starts your firewall automatically during the startup of your client-system.

I have written about his topic somewhere else in this blog already. believe me its easy.

Strategy 3: Using sudo

Working with “sudo” and manipulating the file “/etc/sudoers” is somewhat risky. Therefore:

Disclaimer: I take no responsibility whatever for the consequences of the sudo approach describe below and its application to your computers. The sudoer rules have to be tested carefully before the are used in a production environment and their setup must be supervised by an expert.

The settings below work for an Opensuse Leap system – partially due to the default settings there.

A standard problem with sudoer rules for graphical applications is the handling of the access to the X11
display if you need to start programs as another user (e.g. root). (Things may be even worse with Wayland; but I have no experience with it). To keep things simple it is a worthwhile investment to think a bit about the precise nature of your sudo-objective.

In our case we want enforce a user-specific usage of the command remote-viewer. More precisely:

  1. We want to disallow the usage of remote-viewer for most users. And even selected users shall not be able to call or invoke remote-viewer directly and freely.
  2. We want to enforce user-specific arguments to the command “remote-viewer”, if executed by certain selected users.
  3. If possible we do not want to run any remote-tool with root-rights at all.
  4. We want to keep our user-specific firewall rules in place and not to create new ones.
  5. X11-display access shall be possible.

The answer to this challenge is a bit tricky. It first looks like you need to write a separate script (accessible to root) which evaluates UIDs or SUD_UIDs and then calls remote-viewer with appropriate arguments.

But you will then be confronted with problems to access the X11-display of the user issuing the script. In addition rule 3 above forces you to start the required remote-viewer command in the end as a specific user via “sudo -u USER” or “su -c ‘…’ USER”. This in turn forces you to allow “USER” to execute the remote-viewer command anyway according to a specific sudo (!) rule for USER. Therefore, he/she must be able to read and execute the remote-viewer command. But then he/she could execute it outside freely without sudo – and again use any arguments he/she likes. It took me a while to find a way out.

The right approach is to find a rule working for the user in question, first, before you turn to a script as a mediator. On the other side: The user must NOT be allowed to use “remote-viewer” freely – neither by user or group access rights. This seeming contradiction is solved by the following steps on our client-system; we describe the rules below for user “myself”. You must execute most of the commands as root:

  • Step 1: Create a special group, e.g. named “spicegrp”. Do NOT make user “myself” a member of this group!
  • Step 2: Change the ownership and access rights of “/usr/bin/remote-viewer” according to
    • chown root.spicegrp /usr/bin/remote-viewer
    • chmod 750 /usr/bin/remote-viewer
  • Step 3: Check that remote-viewer can no longer be executed by user “myself”.
  • Step 4: Start editing the file “/etc/sudoers” with visudo.
  • Step 5: Add the following lines and turn some existing lines into comments:
    Defaults env_reset
    Defaults:myself env_keep += "DISPLAY"
    #Defaults targetpw   # ask for the password of the target user i.e. root
    #ALL   ALL=(ALL) ALL
    myself ALL=(myself:spicegrp) /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002
    # myself ALL=(myself:spicegrp) NOPASSWD: /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002
    # mybro ALL=(mybro:spicegrp) NOPASSWD: /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20004

This is all we basically need for the user myself. An extension to
user “mybro” should be clear. See the commented line for mybro.

The sudoer statements – also the commented ones – deserve some short explanation:
We reset the environment, but we keep some language settings for sudoer users. More important: We keep the “DISPLAY” variable of the environment of sudoer user “myself”. Meaning, it will be available when commands are executed with his/her UID. Hereby, we avoid major X11 trouble. The two commented lines in the middle correspond to the the objective that sudoer users should provide their own passwords instead of the root password to execute the commands in the prescribed form.

And then comes some sudo vodoo:

We only define a rule for the user “myself”: We allow him/her to execute the command remote-viewer with his own UID, but with a different group. This gives him access to “/usr/bin/remote-viewer” under “sudo” conditions ( only !). But, in no way else on a shell’s command line, as he/her no longer has access rights to “/usr/bin/remote-viewer” and is no member of “spicegrp”!

The other part of the magic is that the sudoer-mechanism checks the precise form by which the user in question (here: myself) executes the command. It compares the command exactly to the form given in the rule line including the command’s arguments!

myself ALL=(myself:spicegrp) /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002

The command and the arguments to it are together handled as one string for comparison!
Thus we have a user “myself” who can only use remote-viewer with “sudo” and he/she is forced to provide a specific argument. And when he issues the command the firewall rules defined in the first part of this post should open doors to the VM as the execution is done with his/her UID!

What will the required sudo command for a successful access to the VM1 (debianx) at TLS-port 20002 on server MySRV look like:

sudo -u myself -g spicegrp remote-viewer — spice://mysrv.anraconc.de?tls-port=20002

You took notice of the argument for the group?

Test for a sudoer user

Let us test our theory (sorry for the German system messages, tried to translate them):

myself@mylap:~> remote-viewer -- spice://mysrv.anraconc.de?tls-port=20002
-bash: /usr/bin/remote-viewer: Keine Berechtigung
myself@mylap:~> # Keine Berechtigung = No access right
myself@mylap:~> # Now with sudo - but wrong port 20004
myself@mylap:~> sudo -u myself -g spicegrp remote-viewer -- spice://mysrv.anraconc.de?tls-port=20004
[sudo] Passwort für myself: 
Leider darf der Benutzer myself »/usr/bin/remote-viewer -- spice://mysrv.anraconc.de?tls-port=20004« als myself:spicegrp auf mylap.anraconc.de nicht ausführen.
myself@mylap:~> # Translation: Sorry, but user myself is not allowed to execute »/usr/bin/remote-viewer -- spice://beta.rux.anraconb.de?tls-port=20004« as myself:spicegrp on mylap.anraconc.de.
myself@mylap:~> # Now with sudo - but the right port 20002
myself@mylap:~> sudo -u myself -g spicegrp remote-viewer -- spice://mysrv.anraconc.de?tls-port=20002

And now we get the familiar dialog to authenticate :

Our firewall obviously has let us through. And – after filling in the VM-specific password (we work without SASL here; see above), we get:

I leave
it to the reader extend the whole thing to two users and to test all combinations out.

Making things a bit easier for the users

An authorized user has to type a lot of things to make the sudo command work. One thing, you can think about, is to not require a password for the sudo command. (Personally, I would not do this; a password is a last security barrier in case of configuration mistakes. But, in principle it is possible to work without a password in this specific case – after you have checked out and tested all security implications). The entry in the “/etc/sudoers” file then would look like

myself ALL=(myself:spicegrp) NOPASSWD: /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002

Another reduction of typing work comes through a script which readers can read and execute, but not change. The script could make all the relevant decisions for a user. A very simplified version would in my test scenario contain statements like:

if  test $SUDO_UID
        if test  $UID -ne $SUDO_UID
                echo "error - you are not allowed to run this command as another user"
if  test $UID -eq $Myself_UID
        echo "Hello Ralph"
        msg="You are entering the Spice console of VM \"debianx\" - happy working"
        echo $msg
        sudo -u "#$UID" -g spicegrp /usr/bin/remote-viewer -- spice://$host?tls-port=$myself_port
elif  test $UID -eq $Mybro_UID
        echo "Hello Brother"
        msg4="You are entering the Spice console of VM \"leaptest\" - happy working"
        echo $msg4
        sudo -u "#$UID" -g spicegrp /usr/bin/remote-viewer -- spice://$host?tls-port=$mybro_port
        echo "Sorry, you are not allowed to access VMs"

Note that the script would ask for a password before executing the sudo statement enclosed – if you had defined a password request in the sudoer file.
If you absolutely wanted to obfuscate the information contained in this script you could again use the trick with sudo and the special group spicegrp. You would the add lines

myself ALL=(myself:spicegrp) /usr/bin/rviewer
myself ALL=(mybro:spicegrp) /usr/bin/rviewer

to the “/etc/sudoers”-file

Important note:

You must not forget to check the fie permission of the file “/usr/bin/remote-viewer” after SW-updates or upgrades of your system.

I would recommend to start a small scheduled job or a service to check the rights settings frequently.


Remote-viewer connections to VMs cannot be controlled on a user level by the KVM/Qemu server if we just used TLS and SASL. We can set up a VM specific password. But external connections to a VM specific TLS port can only be blocked for external systems and IPs on the server.
However, on Linux client-systems iptables helps us to allow access to the Spice console of a specific VM for a selected user, only. This can be achieved by setting up user specific iptables rules on client-systems. This post has shown you how to create such rules for the OUTPUT chain with fwbuilder. We must set up a systemd service to implement these rules automatically at system (and network) startup.

To restrict users on Linux client systems even more we applied the sudo mechanism in a very specific way: we enforced the usage of certain arguments to the remote-viewer command for specific users. I think the method I discussed is safe; if you find a caveat please send me a mail.

Both strategies can and should be applied in Intranets where we want to provide remote-viewer and the Spice consoles of Qemu VMs as real working instruments.

Linux bridges – can iptables be used against MiM attacks based on ARP spoofing ? – II

In the last article
Linux bridges – can iptables be used against MiM attacks based on ARP spoofing ? – I
we saw that iptables rules with options like

-m physdev –physdev-in/out device

may help in addition to other netfilter tools, which work on network level 2, to block redirected traffic to a “man in the middle system” on a bridge.

Tools like FWbuilder support the creation of such “physdev”-related rules as soon as bridge devices are marked as bridged in the interface definition process for the firewall host. However, we have also seen that we need to bind IP addresses to certain bridge ports. This in turn requires knowledge about a predictable IP-to-port-configuration. Such a requirement may be an obstacle for using iptables in scenarios with many virtual guests on one or several Linux bridges of a virtualization host as it reduces flexibility for automated IP address assignment.

Before we discuss administrative aspects in a further article, let us expand our iptables rules to a more complex situation:
In this article we discuss a scenario with 2 linked Linux bridges “virbr4” and “virbr6” plus the host attached to “virbr4”. This provides us with a virtual infrastructure for which we need to construct a more complex, but more general set of rules in comparison to what we discussed in the last article. We will look at the required rules and their order. Testing of the rules will be done in a forthcoming article.

Two coupled bridges and the host attached via veth devices

You see my virtual bridge setup in the following drawing.

(Note for those who read the article before: I have exchanged the picture to make it consistent with a forthcoming article. The port for kali2 has been renamed to “vk42”).


The small blue rectangles inside the bridges symbolize standard Linux tap devices – whereas the RJ45 like rectangles symbolize veth devices. veth pairs deliver a convenient way on a Linux system to link bridges and to attach the host to them in a controllable way. As a side effect one can avoid to assign the bridge itself an IP address. See:
Fun with veth devices, Linux virtual bridges, KVM, VMware – attach the host and connect bridges via veth

In the drawing you recognize our bridge “virbr6” and its guests from the last article. The new bridge “virbr4” is only equipped with one guest (kali2); this is sufficient for our test case purposes. Of course, you could have many more guests there in realistic scenarios. Note that attaching certain groups of guests to distinct bridges also occurs in physical reality for a variety of reasons.

Two types of ports

For the rest of this article we call ports as vethb2 on virbr6, vethb1 and vmh1 on virbr4 “border ports” of their respective bridges. Such border ports

  • connect a bridge to another bridges,
  • connect a bridge to the virtualization host
  • or connect the bridge to hosts on external real Ethernet segments.

We remind the reader that it always is the perspective of the bridge that decides
about the INcoming or OUTgoing direction of an Ethernet packet via a specific port when we define IN/OUT iptables rules.

Therefore, packets crossing a border port in the IN direction always come from outside the bridge. Packets leaving the port OUTwards may however come from guests of the bridge itself AND from guests outside other border ports of the very same bridge.

In contrast to border ports we shall call a port of a bridge with just one defined guest behind it a “guest port”. [In our test case the bridge connection of guests is realized with tap devices because this is required by KVM. In the case of LXC and docker containers we would rather see veth-pairs.]

Multiple bridges on one host – how are the iptables rules probed?

Just from looking at the sketch we see a logical conundrum, which has a significant impact on the setup of iptables rules on a host with multiple bridges in place:

A packet created at one of the ports may leave the bridge where it has been created and travel in and through a neighboring bridge via border ports. But when and how are port related rules tested as the packet travels – lets e.g. say from kali5 to the guest at “vnet0” or to the host at “vmh2”?

  • Bridge for bridge – IN-Port-rules, then OUT-port-rules on the same first bridge => afterwards IN-port-rules/OUT-port-rules again – but this time for the ports of the next entered bridge?
  • Or: iptables rules are checked only once, but globally and for all bridges – with some knowledge of port-MAC-relations of the different bridges included?

If the latter were true just one passed ACCEPT rule on one single bridge port would lead to an overall acceptance of a packet despite the fact that the packet possibly will cross further bridges afterward. Such a behavior would seem unreasonable – but who knows …

So the basic question is:

After having been checked on a first bridge, having been accepted for leaving one border port of this first bridge and then having entered a second linked bridge via a corresponding border port – will the packet be checked again against all denial and acceptance rules of the second bridge? Will the packet with its transportation attributes be injected again into the whole set of iptables rules?

It is obvious that the answer would have an impact of how we need to define our rules. Especially during port flooding, which we already observed in the tests described in our first article.

Tests of the order of iptables rules probing for ports of multiple bridges on a packet’s path

As a first test we do something very simple: we define some iptables rules for ICMP pings formally in the following logical order: We first deny a passage through vethb1 on virbr4 before we allow the packet to pass vethb2 on virbr6:

bridge vibr4 rule 15:  src, dest - ICMP IN vethb1 => DENY
bridge vibr6 rule 16:  src, dest - ICMP OUT vethb2 => ALLOW

and then we test the order of how these rules are passed by logging them.

To avoid any wrong or missing ARP information on the involved guest/host systems and missing MAC-port-relations in the “forward databases” [FWB] of the bridges we first clear any iptables rules and try some pings before sending ping packets. Then we activate the rules and get for ping packets sent from kali4 to the host:

2016-02-27T12:09:33.295145+01:00 mytux kernel: [ 
5127.067043] RULE 16 -- ACCEPT IN=virbr6 OUT=virbr6 PHYSIN=vk64 PHYSOUT=vethb2 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=22031 DF PROTO=ICMP TYPE=8 CODE=0 ID=1711 SEQ=1 
2016-02-27T12:09:33.295158+01:00 mytux kernel: [ 5127.067062] RULE 15 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vethb1 PHYSOUT=vmh1 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=22031 DF PROTO=ICMP TYPE=8 CODE=0 ID=1711 SEQ=1 
2016-02-27T12:09:34.302140+01:00 mytux kernel: [ 5128.075040] RULE 16 -- ACCEPT IN=virbr6 OUT=virbr6 PHYSIN=vk64 PHYSOUT=vethb2 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=22131 DF PROTO=ICMP TYPE=8 CODE=0 ID=1711 SEQ=2 
2016-02-27T12:09:34.302153+01:00 mytux kernel: [ 5128.075056] RULE 15 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vethb1 PHYSOUT=vmh1 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=22131 DF PROTO=ICMP TYPE=8 CODE=0 ID=1711 SEQ=2

Now we do a reverse test: We allow the incoming direction over port vk64 of virbr6 before we deny the incoming package over vethb1 on virbr4:

bridge vibr6 rule :  src, dest - IN vk64 => ALLOW
bridge vibr4 rule :  src, dest - IN vethb1 => DENY

We get

2016-02-27T14:02:32.821286+01:00 mytux kernel: [11913.962828] RULE 15 -- ACCEPT IN=virbr6 OUT=virbr6 PHYSIN=vk64 PHYSOUT=vethb2 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=21400 DF PROTO=ICMP TYPE=8 CODE=0 ID=2104 SEQ=1 
2016-02-27T14:02:32.821307+01:00 mytux kernel: [11913.962869] RULE 16 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vethb1 PHYSOUT=vmh1 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=21400 DF PROTO=ICMP TYPE=8 CODE=0 ID=2104 SEQ=1 
2016-02-27T14:02:33.820257+01:00 mytux kernel: [11914.962965] RULE 15 -- ACCEPT IN=virbr6 OUT=virbr6 PHYSIN=vk64 PHYSOUT=vethb2 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=21494 DF PROTO=ICMP TYPE=8 CODE=0 ID=2104 SEQ=2 
2016-02-27T14:02:33.820275+01:00 mytux kernel: [11914.962987] RULE 16 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vethb1 PHYSOUT=vmh1 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=21494 DF PROTO=ICMP TYPE=8 CODE=0 ID=2104 SEQ=2 

So to our last test:

bridge vibr6 rule :  src, dest - IN vk64 => ALLOW
bridge vibr6 rule :  src, dest - IN vethb2 => DENY
bridge vibr4 rule :  src, dest - IN vethb1 => DENY

We get:

2016-02-27T14:26:08.964616+01:00 mytux kernel: [13331.634200] RULE 15 -- ACCEPT IN=virbr6 OUT=virbr6 PHYSIN=vk64 PHYSOUT=vethb2 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=27122 DF PROTO=ICMP TYPE=8 CODE=0 ID=2218 SEQ=1 
2016-02-27T14:26:08.964633+01:00 mytux kernel: [13331.634232] RULE 17 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vethb1 PHYSOUT=vmh1 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=27122 DF PROTO=ICMP TYPE=8 CODE=0 ID=2218 
2016-02-27T14:26:09.972621+01:00 mytux kernel: [13332.643587] RULE 15 -- ACCEPT IN=virbr6 OUT=virbr6 PHYSIN=vk64 PHYSOUT=vethb2 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=27347 DF PROTO=ICMP TYPE=8 CODE=0 ID=2218 SEQ=2 
2016-02-27T14:26:09.972637+01:00 mytux kernel: [13332.643605] RULE 17 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vethb1 PHYSOUT=vmh1 MAC=96:b0:a9:7c:73:7d:52:54:00:74:60:4a:08:00 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=27347 DF PROTO=ICMP TYPE=8 CODE=0 ID=2218 SEQ=2 


Intermediate conclusions

We can conclude the following points :

  • A packet is probed per bridge – in the order of how multiple bridges of the host are passed by the packet.
  • An ALLOW rule for a port on one bridge does not overrule a DENY rule for a port on a second bridge which the package may pass on its way.
  • A packet is tested both for IN/OUT conditions of a FORWARD rule for each bridge it passes.
  • If we split IN and OUT rules on a bridge (as we need to do within some tools as FWbuilder) than we must probe the OUT rules first to guarantee the prevention of illegal packet transport.

For the rest of the article we shall follow the same rule we already used as a guide line in the previous article: Our general iptables policy is that a packet will be denied if it is not explicitly accepted by one of the tested rule.

Blocking of border ports in port flooding situations

During our tests in the last article we have seen that port flooding situations may occur – depending among other things on the “setaging” parameter of the bridge and the resulting deletion of stale entries in the “Forward Database” [FWD] of a bridge. Flooding of veth based border ports may be critical for packet transmission and may have to be blocked in some cases:

E.g. it would be unreasonable to transfer packets logically meant for hosts beyond port vmh1 of virbr4 over vethb1/2 to virbr6. We would stop such packets already via OUT DENY rules for vethb1:

bridge vibr4 rule :  src "guest of virbr4", dest "no guest of virbr6" - OUT vethb1 => DENY


Rules regarding packets just crossing and passing a bridge

Think about a bridge “virbrx” linked on its both sides to two other bridges “virbr_left” and “virbr_right”. In such a scenario packets could arrive at virbrx from bridge virbr_right, enter the intermediate bridge virbrx and leave it at once again for the third bridge virbr_left – because it never was destined to any guest of bridge virbrx.

For such packets we need at least one ACCEPT rule om virbrx – either on the IN direction of the border port of virbrx against virbr_right or on the OUT direction at the border port to virbr_left.

Again, we cling to our policy of the last article:
We define DENY rules for outgoing packets at all ports – also for border ports – and put these DENY rules to the top of the iptables list; then we define DENY rules for ports which are passed in IN direction; only after that we define ACCEPT rules for incoming packets for all ports of a bridge – including border ports – and set these rules below/after the DENIAL rules. This should provide us with a consistent handling also of packets crossing and passing bridges.

Grouping of guests/hosts

From looking at the drawing above we also understand the following point: In order to handle packets at border ports connecting two bridges we have the choice to block packets at either border
port – i.e. before the OUTgoing port passage on the first bridge OR before the INcoming port passage on the second. We shall do the blocking at the port in the packets OUTgoing direction. Actually, there would be no harm in setting up reasonable DENY rules for both ports. Then we would safely cover all types of situations.

Anyway – we also find that the rules for border ports require a certain grouping of the guests and hosts:

  • Group 1: Guests attached to the bridge of the border port.
  • Group 2: Guests on the IN side of the border port of a bridge – i.e. the internal side of the bridge. This group includes Group 1 plus external guests of further bridges beyond other border ports of the very same bridge.
  • Group 3:Guests on the outgoing side of a border port – i.e. the side to the next connected bridge. This group contains hosts of Group 1 for the next connected bridge and/or groups of external hosts on the OUT side of all other border ports of the connected bridge.

These groups can easily be formed per bridge by tools like FWbuilder. Without going into details: Note that FWbuilder handles the overall logical OR/AND switching during a negation of multiple groups of hosts correctly when compiling iptables rules.

Overall rules order in case of multiple and connected bridges

Taking into account the results of the first article in this series I suggest the following order of iptables rules:

  • We first define OUT DENY rules for all guest ports of all bridges – with ports grouped by bridges just to keep the overview. These rules are the most important ones to prevent ARP spoofing and a resulting packet redirection.
  • We then define all OUT DENY rules for border ports of all bridges – first grouped by bridges and then per bridge and ports grouped by hosts for the OUTgoing direction. These rules cover also port flooding situations with respect to neighboring linked bridges.
  • We then define IN DENY rules for incoming packets over border ports. These rules may in addition to the previous rules prevent implausible packet transport.
  • Now we apply OUT DENY and IN DENY rules for Ethernet devices on the virtualization host. Such rules must must not be forgotten and can be placed herein the rules’ sequence.
  • We then define IN ACCEPT rules on individual guest ports – ports again grouped by bridges.
  • We eventually define IN ACCEPT rules on bridge border ports – note that such rules are required for packets just passing an intermediate bridge without being destined to a guest of the bridge.
  • IN ACCEPT rules for the virtualization hosts’s Ethernet interfaces must not be forgotten and can be placed at the end.

How does that look like in FWbuilder?

Before looking at the pics note that we have defined the host groups

  • br6_grp to contain kali3, kali4, kali5,
  • br4_grp to contain only kali2,
  • ext_grp to contain the host and some external web server “lamp“.

With this we get the following 7 groups of rules:








Despite the host grouping : this makes quite a bunch of rules! But yet not uncontrollable …

Enough for today. I hope that tests being performed in a third article of this series will not proof me wrong. I am confident …. See
Linux bridges – can iptables be used against MiM attacks based on ARP spoofing ? – III