KVM/Qemu VMs with a multi-screen Spice console – VIII – VM and user specific restrictions for remote-viewer connections – iptables and sudo

The Spice console allows users to access the graphical desktop of a Qemu based virtual machines [VM]. The performance with both data encryption and data compression is excellent, audio is no problem and the required data transfer rates to client-applications as "remote-viewer" are within reasonable limits for a (switched) LAN. In virtualization scenarios where you can organize tasks according to a scheme "one user per VM" Spice is actually an attractive tool - and even professional virtualization environments a "Proxmoxx" and "oVirt" make use of it.

In this series we look at basic setups in self-administered Intranet environments. So far we had some ups and downs regarding the tool remote-viewer. If you want to use it in a desktop virtualization environment it is an almost perfect tool. We also found it to be a very convenient and efficient remote client-tool in an Intranet when we combined it with SSH and internal data compression of the Spice protocol. See:

KVM/Qemu VMs with a multi-screen Spice console – V – remote access via remote-viewer, a network port and a SSH-tunnel
KVM/Qemu VMs with a multi-screen Spice console – IV – remote access via SSH, remote-viewer and a Unix socket
KVM/Qemu VMs with a multi-screen Spice console – III – local access with remote-viewer via a Unix socket

SSH gave us all options we needed to take care of various security issues in remote access scenarios. The KVM/Qemu server could control the interaction of remote-users with VMs by applying user-specific SSH restrictions to port-forwarding. We could establish rules to bind the Spice access to a specific VM to a specific user.

Regarding the last point the combination of remote-viewer, TLS and SASL came as a solid disappointment. TLS worked perfectly. But we had problems with SASL; see:

KVM/Qemu VMs with a multi-screen Spice console – VII – remote-viewer, qemu and SASL authentication
KVM/Qemu VMs with a multi-screen Spice console – VI – remote access with remote-viewer and TLS encryption

We got SASL authentication working in two different ways. The frustrating result of our efforts, however, was that we could not confine the access to the Spice-console of a specific VM to exactly one user.

In the end we found that it might even be better to use TLS and a VM-specific Spice password set in the VM's XML-domain file. The problem then still is that users could share the password for a specific VM. So, we still could not ensure a scenario "one VM - one UID, only".

In this article we, therefore, look at measures on remote client-systems to allow a TLS connection to a specific VM on the KVM/Qemu host for exactly one user. We use two different Linux tools - namely iptables and sudo - to achieve this. The recipes given below are interesting in themselves and can be applied to other scenarios where the admin wants to restrict outgoing TCP-connection on a user specific level.

Why do we care about user-specific control for Spice at all?

Assume a situation in a small Linux oriented office where you run a bunch of VMs for special purposes on a central KVM-server.
You provide e.g. a VM (VM1) with Windows 10 ( 🙁 ) for book-keeping with a program your tax advisor understands. A user "Charlie" with a defined UID should use the VM's Spice console, but basically only he (and his boss in the evening hours).
You have a second VM (VM2) with a Web-server for development and test purposes. Only user "Maggie" shall have access to the Spice console of VM2 on Mondays and Tuesdays and a user "Anne" on Wednesdays and Thursdays and no one else.
And then there is yet another VM (VM3) which only user "Ralph" shall access via Spice to create documents of a confidential analysis for a governmental organization.
There are multiple Linux client PCs available in the office; any of the user can use and login to any free Linux workstations. None of the user has root rights on the clients and the server.

We have thus defined a scenario of the type

One VM + Spice console    < = >    one user or a group of selected users.

Can you cover such a situation with remote-viewer, TLS and server-based SASL, only?
The answer of my last article was: NO. At least not by simple means. (By the way: It is not the fault of SASL. Remote-viewer, simply and unfortunately, does not provide any system-based data for the realm or for the remote-system which SASL could evaluate. The user has too much freedom ...)

Do we need measures on the client-systems?

It is interesting that even Opensuse's "Virtualization Guide" writes about some required measures on the client (restricting access to specific client-certificates for selected users) when discussing the security for libvirt and vnc as tools to access the graphical desktop of a VM. We have not come to libvirt-based tools in this series, yet, but this is already some indication that restrictions of desktop access to KVM/Qemu based VMs may sometimes require measures on the client. So, let us turn to the client-systems ... Of course, we assume that such clients have a Linux OS, in my case a Opensuse Leap 15.2.

Schematic drawing

The following schematic drawing reflects the situation we want to control:
We want to make it impossible user "mybro" to access the Spice console of a VM1. Only "myself" should be able to use VM1's Spice console via a TLS connection. The other way round: Only user "mybro" shall be allowed to connect to the the Spice console of VM2 with remote-viewer. Cross-connections are forbidden; each user gets access to one defined VM only.

We use the same systems as in the last articles: A KVM/Qemu server host "MySRV" (IP: 192.168.2.2) with a Leap 15.2 OS on it, a test-VM1 "debianx" with a Kali-OS on it and a client-system "MyLAP" (a laptop with a Leap 15.2 OS; IP: 192.168.2.22). On MyLap we have a user "myself" and a user "mybro". VM1 ("debianx") gets the (TLS-) port 20002 on the server, and VM2 ("leaptest") the (TLS-) port 20004.

We shall perform the experiments in this article with a Qemu set up without SASL, but each VM configured for an individual password. (As we already know it will only means a small difference in the form of the authentication dialog. We will not be asked for a username.) The scenario can easily be changed to SASL authentication.

Strategies to allow Spice access to a VM's desktop only for a specific user

There are three major strategies we can follow:

Strategy 1: We could think about a specific TLS client certificate for a connection to a VM and to make it accessible only to the user in question. Such a strategy would require that client certificates are supported by Qemu and Spice.

Strategy 2: We could use iptables and add user-specific rules for outgoing data to certain TLS-ports on the server. As user-related rules can only be set for the "output chain" we cannot set any such user-specific filters for our problem on the server.

Strategy 3: We take the choice of the target port on the server away from the user as well as any free access to the remote-viewer command namely by changing file permissions, setting up special sudo rules and/or enforcing him/her to run a shell script which starts remote-viewer with a pre-defined target port for him/her.

The strategies (if working) can be combined. Even if the firewall of strategy 2 failed the user would only get access to a specific VM by restrictions of strategy 3. But all these things are a bit complicated and they all depend on the user not being able to get root rights, the firewall being set up at every system-startup and the sudo rules being tight. This is crucial; our KVM/Qemu server with TLS and SASL could only block connections from certain IPs, but not really users.

We could extend strategy 3 to a network namespace - but it won't help too much as we then would again need to apply routing and related firewall rules. Or to sudo rules. We could also think about using different networks for different VMs and thus a port-IP-VM-relation - but we would still depend on user-specific firewall rules on the clients.

Strategy 1: What about Spice and TLS client certificates?

Here I come with bad news:
If you activate the option "default_tls_x509_verify = 1" in "/etc/livirt/qemu.conf" it is simply ignored. One can guess it from a lack of required parameters in the qemu-command created by virt-manager. Even if and when you provide client certificates in the required directories on the server and the client. In contrast to VNC-settings there is no option like "spice_tls_x509_verify = 1" available. If you set it by yourself, it is silently ignored.

So, client-certificates will be of no help with Spice and remote-viewer. The situation my be different for the libvirt-dependent "virt-viewer" tool. But this is the topic of a future blog post.

Strategy 2: netfilter/iptables to the rescue

I assume that you have some tool in place on your Linux systems to configure your own iptables-rules. Professionals may write their own scripts. Independent of your interface to netfilter/iptables, it is obvious that working with firewall rules in a productive environment is a risky business. Therefore:

Disclaimer: I take no responsibility whatever for the consequences of the approach describes below and its application to your computers. The iptables-rules have to be tested carefully before making them productive and their setup on Linux hosts must be supervised by an expert.

In my network environments I still cling to the tool "fwbuilder" because it gives you a good graphical overview for most purposes and it allows for more complex configurations than e.g. frontends to firewalld or ufw. So, let me show you how to set up a basic set of user-specific iptables rules with fwbuilder and then have a closer look at the contents of the created statements of the firewall script.

fwbuilder allows you to set up "users" with a defined UID in the object catalog for "services":

This may seem strange; but the meaning is the following:

  1. Using a "user" as a "service" will trigger the creation of a rule for the OUTPUT chain which matches the user of outgoing connection packets against an owner with a defined UID and then triggers a reaction.
  2. To cover the reaction in fwbuilder we "branch off" into a specific rule-set for the user. On the iptables level this corresponds to a user-defined rule-chain as a reaction target.
  3. The filter-rules for data packets in the user-defined chain themselves then lead to certain final reactions in the sense of "allow" or "deny".

In pseudo-code: IF a user matches an UID THEN apply a set of filter rules with their own final reactions.

In Fwbuilder a set of filter-rules is usually put into an object called "policy". The following image shows some rules of the main "policy" on "MyLap":

You see that rules 4 and 5 include users "myself" and "mybro"; the rules branch off to policies "Spice_VM_1" and "Spice_VM_2", respectively.

Rule 6 blocks a whole bunch of TCP-ports used for VMs on the KVM/Qemu server for any other user than "myself" and "mybro".

Policy "Spice_VM_1" contains one rule, only:

This rule specifies a certain interface and the IP of the target host "mysrv". We allow access to port 20002 - corresponding to the TLS port defined for our test VM "debianx" (VM1). Something analogous holds for our second user "mybro". He gets access to another virtual machine "VM2" which we have bound to port 20004 on the KVM-server for external TLS connections.

Now, let us look at the iptables statements generated by fwbuilder:


    # ================ Table 'filter', automatic rules
    # accept established sessions
    $IPTABLES -A INPUT   -m state --state ESTABLISHED,RELATED -j ACCEPT 
    $IPTABLES -A OUTPUT  -m state --state ESTABLISHED,RELATED -j ACCEPT 
    $IPTABLES -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT 
    # backup ssh access from a admin host 
    $IPTABLES -A INPUT  -p tcp -m tcp  -s 192.168.44.12/255.255.255.255  --dport 22  -m state --state NEW,ESTABLISHED -j  ACCEPT 
    $IPTABLES -A OUTPUT  -p tcp -m tcp  -d 192.168.44.12/255.255.255.255  --sport 22  -m state --state ESTABLISHED,RELATED -j ACCEPT


    # ================ Table 'filter', rule set Spice_VM_1
    # 
    # Rule Spice_VM_1 0 (eth0)
    # 
    echo "Rule Spice_VM_1 0 (eth0)"
    # 
    $IPTABLES -N Spice_VM_1
    $IPTABLES -N Cid40091X30615.0
    $IPTABLES -A Spice_VM_1 -o eth0  -p tcp -m tcp  -d 192.168.2.2   --dport 20002  -m state --state NEW  -j Cid40091X30615.0
    $IPTABLES -A Cid40091X30615.0  -s 192.168.2.22   -j ACCEPT
    $IPTABLES -A Cid40091X30615.0  -s 192.168.2.22   -j ACCEPT
    
    # ================ Table 'filter', rule set Spice_VM_2
    # 
    # Rule Spice_VM_2 0 (eth0)
    # 
    echo "Rule Spice_VM_2 0 (eth0)"
    # 
    $IPTABLES -N Spice_VM_2
    $IPTABLES -N Cid40661X30615.0
    $IPTABLES -A Spice_VM_2 -o eth0  -p tcp -m tcp  -d 192.168.2.2   --dport 20004  -m state --state NEW  -j Cid40661X30615.0
    $IPTABLES -A Cid40661X30615.0  -s 192.168.2.22   -j ACCEPT
    $IPTABLES -A Cid40661X30615.0  -s 192.168.2.22   -j ACCEPT
    
    # ================ Table 'filter', rule set Main_Policy
    # 
    # Rule Main_Policy 0 (eth0)
    # 
    echo "Rule Main_Policy 0 (eth0)"
    # 
    # Antispoofing
    $IPTABLES -N In_Main_Policy_0
    $IPTABLES -A INPUT -i eth0   -s 192.168.2.22   -j In_Main_Policy_0
    $IPTABLES -A INPUT -i eth0   -s 192.168.4.22   -j In_Main_Policy_0
    $IPTABLES -A FORWARD -i eth0   -s 192.168.2.22   -j In_Main_Policy_0
    $IPTABLES -A FORWARD -i eth0   -s 192.168.4.22   -j In_Main_Policy_0
    $IPTABLES -A In_Main_Policy_0  -j LOG  --log-level info --log-prefix "RULE 0 -- DENY "
    $IPTABLES -A In_Main_Policy_0  -j DROP
    # 
    # Rule Main_Policy 1 (lo)
    # 
    echo "Rule Main_Policy 1 (lo)"
    # 
    $IPTABLES -A INPUT -i lo   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -o lo   -m state --state NEW  -j ACCEPT
    # 
    # Rule Main_Policy 2 (global)
    # 
    echo "Rule Main_Policy 2 (global)"
    # 
    #  ICMP, DNS, DHCP 
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 3  -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 0/0   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 8/0   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 11/0   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p icmp  -m icmp  --icmp-type 11/1   -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p tcp -m tcp  --dport 53  -m state --state NEW  -j ACCEPT
    $IPTABLES -A OUTPUT -p udp -m udp  -m multiport  --dports 68,67,53  -m state --state NEW  -j ACCEPT
    # 
    # Rule Main_Policy 3 (global)
    # 
    echo "Rule Main_Policy 3 (global)"
    # 
    $IPTABLES -A OUTPUT -p udp -m udp  --dport 123  -m state --state NEW  -j ACCEPT
    # 
    # Rule Main_Policy 4 (global)
    # 
    echo "Rule Main_Policy 4 (global)"
    # 
    $IPTABLES -A OUTPUT -m owner --uid-owner 1021  -j Spice_VM_1
    # 
    # Rule Main_Policy 5 (global)
    # 
    echo "Rule Main_Policy 5 (global)"
    # 
    $IPTABLES -A OUTPUT -m owner --uid-owner 1022  -j Spice_VM_2
    # 
    # Rule Main_Policy 6 (global)
    # 
    echo "Rule Main_Policy 6 (global)"
    # 
    $IPTABLES -N Out_Main_Policy_6
    $IPTABLES -A OUTPUT -p tcp -m tcp  --dport 20001:20010  -j Out_Main_Policy_6
    $IPTABLES -A Out_Main_Policy_6  -j LOG  --log-level info --log-prefix "RULE 6 -- DENY "
    $IPTABLES -A Out_Main_Policy_6  -j DROP
    # 
...
...

 
These are simple rules which the experienced reader will have no difficulty to interpret. (The reader also sees that I have multiple IPs on eth0, but this does not affect our present topic).

I leave the test to you. You should see that user "myself" can access VM1 (debianx) whilst user "mybro" and any other users cannot. User "mybro" instead can access VM2 on the server.

Important note:

Do not forget to write a small systemd service which starts your firewall automatically during the startup of your client-system.

I have written about his topic somewhere else in this blog already. believe me its easy.

Strategy 3: Using sudo

Working with "sudo" and manipulating the file "/etc/sudoers" is somewhat risky. Therefore:

Disclaimer: I take no responsibility whatever for the consequences of the sudo approach describe below and its application to your computers. The sudoer rules have to be tested carefully before the are used in a production environment and their setup must be supervised by an expert.

The settings below work for an Opensuse Leap system - partially due to the default settings there.

A standard problem with sudoer rules for graphical applications is the handling of the access to the X11 display if you need to start programs as another user (e.g. root). (Things may be even worse with Wayland; but I have no experience with it). To keep things simple it is a worthwhile investment to think a bit about the precise nature of your sudo-objective.

In our case we want enforce a user-specific usage of the command remote-viewer. More precisely:

  1. We want to disallow the usage of remote-viewer for most users. And even selected users shall not be able to call or invoke remote-viewer directly and freely.
  2. We want to enforce user-specific arguments to the command "remote-viewer", if executed by certain selected users.
  3. If possible we do not want to run any remote-tool with root-rights at all.
  4. We want to keep our user-specific firewall rules in place and not to create new ones.
  5. X11-display access shall be possible.

The answer to this challenge is a bit tricky. It first looks like you need to write a separate script (accessible to root) which evaluates UIDs or SUD_UIDs and then calls remote-viewer with appropriate arguments.

But you will then be confronted with problems to access the X11-display of the user issuing the script. In addition rule 3 above forces you to start the required remote-viewer command in the end as a specific user via "sudo -u USER" or "su -c '...' USER". This in turn forces you to allow "USER" to execute the remote-viewer command anyway according to a specific sudo (!) rule for USER. Therefore, he/she must be able to read and execute the remote-viewer command. But then he/she could execute it outside freely without sudo - and again use any arguments he/she likes. It took me a while to find a way out.

The right approach is to find a rule working for the user in question, first, before you turn to a script as a mediator. On the other side: The user must NOT be allowed to use "remote-viewer" freely - neither by user or group access rights. This seeming contradiction is solved by the following steps on our client-system; we describe the rules below for user "myself". You must execute most of the commands as root:

  • Step 1: Create a special group, e.g. named "spicegrp". Do NOT make user "myself" a member of this group!
  • Step 2: Change the ownership and access rights of "/usr/bin/remote-viewer" according to
    • chown root.spicegrp /usr/bin/remote-viewer
    • chmod 750 /usr/bin/remote-viewer
  • Step 3: Check that remote-viewer can no longer be executed by user "myself".
  • Step 4: Start editing the file "/etc/sudoers" with visudo.
  • Step 5: Add the following lines and turn some existing lines into comments:
     
    ....
    Defaults env_reset
    Defaults env_keep = "LANG LC_ADDRESS LC_CTYPE LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_ATIME LC_ALL LANGUAGE LINGUAS XDG_SESSION_COOKIE"
    
    Defaults:myself env_keep += "DISPLAY"
    
    #Defaults targetpw   # ask for the password of the target user i.e. root
    #ALL   ALL=(ALL) ALL
    
    myself ALL=(myself:spicegrp) /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002
    # myself ALL=(myself:spicegrp) NOPASSWD: /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002
    # mybro ALL=(mybro:spicegrp) NOPASSWD: /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20004
    
    

This is all we basically need for the user myself. An extension to user "mybro" should be clear. See the commented line for mybro.

The sudoer statements - also the commented ones - deserve some short explanation:
We reset the environment, but we keep some language settings for sudoer users. More important: We keep the "DISPLAY" variable of the environment of sudoer user "myself". Meaning, it will be available when commands are executed with his/her UID. Hereby, we avoid major X11 trouble. The two commented lines in the middle correspond to the the objective that sudoer users should provide their own passwords instead of the root password to execute the commands in the prescribed form.

And then comes some sudo vodoo:

We only define a rule for the user "myself": We allow him/her to execute the command remote-viewer with his own UID, but with a different group. This gives him access to "/usr/bin/remote-viewer" under "sudo" conditions ( only !). But, in no way else on a shell's command line, as he/her no longer has access rights to "/usr/bin/remote-viewer" and is no member of "spicegrp"!

The other part of the magic is that the sudoer-mechanism checks the precise form by which the user in question (here: myself) executes the command. It compares the command exactly to the form given in the rule line including the command's arguments!

myself ALL=(myself:spicegrp) /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002

The command and the arguments to it are together handled as one string for comparison!
Thus we have a user "myself" who can only use remote-viewer with "sudo" and he/she is forced to provide a specific argument. And when he issues the command the firewall rules defined in the first part of this post should open doors to the VM as the execution is done with his/her UID!

What will the required sudo command for a successful access to the VM1 (debianx) at TLS-port 20002 on server MySRV look like:

sudo -u myself -g spicegrp remote-viewer -- spice://mysrv.anraconc.de?tls-port=20002

You took notice of the argument for the group?

Test for a sudoer user

Let us test our theory (sorry for the German system messages, tried to translate them):

myself@mylap:~> remote-viewer -- spice://mysrv.anraconc.de?tls-port=20002
-bash: /usr/bin/remote-viewer: Keine Berechtigung
myself@mylap:~> # Keine Berechtigung = No access right
myself@mylap:~> # Now with sudo - but wrong port 20004
myself@mylap:~> sudo -u myself -g spicegrp remote-viewer -- spice://mysrv.anraconc.de?tls-port=20004
[sudo] Passwort für myself: 
Leider darf der Benutzer myself »/usr/bin/remote-viewer -- spice://mysrv.anraconc.de?tls-port=20004« als myself:spicegrp auf mylap.anraconc.de nicht ausführen.
myself@mylap:~> # Translation: Sorry, but user myself is not allowed to execute »/usr/bin/remote-viewer -- spice://beta.rux.anraconb.de?tls-port=20004« as myself:spicegrp on mylap.anraconc.de.
myself@mylap:~> # Now with sudo - but the right port 20002
myself@mylap:~> sudo -u myself -g spicegrp remote-viewer -- spice://mysrv.anraconc.de?tls-port=20002

And now we get the familiar dialog to authenticate :

Our firewall obviously has let us through. And - after filling in the VM-specific password (we work without SASL here; see above), we get:

Good!
I leave it to the reader extend the whole thing to two users and to test all combinations out.

Making things a bit easier for the users

An authorized user has to type a lot of things to make the sudo command work. One thing, you can think about, is to not require a password for the sudo command. (Personally, I would not do this; a password is a last security barrier in case of configuration mistakes. But, in principle it is possible to work without a password in this specific case - after you have checked out and tested all security implications). The entry in the "/etc/sudoers" file then would look like

....
myself ALL=(myself:spicegrp) NOPASSWD: /usr/bin/remote-viewer -- spice\://mysrv.anraconc.de?tls-port=20002
...

Another reduction of typing work comes through a script which readers can read and execute, but not change. The script could make all the relevant decisions for a user. A very simplified version would in my test scenario contain statements like:

#!/bin/bash
host="myserv.anraconc.de"
myself_port="20002"
mybro_port="20004"
Myself_UID="1021"
Mybro_UID="1022"
if  test $SUDO_UID
then
        if test  $UID -ne $SUDO_UID
        then
                echo "error - you are not allowed to run this command as another user"
                exit
        fi
fi
if  test $UID -eq $Myself_UID
then
        echo "Hello Ralph"
        msg="You are entering the Spice console of VM \"debianx\" - happy working"
        echo $msg
        sudo -u "#$UID" -g spicegrp /usr/bin/remote-viewer -- spice://$host?tls-port=$myself_port
elif  test $UID -eq $Mybro_UID
then
        echo "Hello Brother"
        msg4="You are entering the Spice console of VM \"leaptest\" - happy working"
        echo $msg4
        sudo -u "#$UID" -g spicegrp /usr/bin/remote-viewer -- spice://$host?tls-port=$mybro_port
else
        echo "Sorry, you are not allowed to access VMs"
fi

Note that the script would ask for a password before executing the sudo statement enclosed - if you had defined a password request in the sudoer file.
If you absolutely wanted to obfuscate the information contained in this script you could again use the trick with sudo and the special group spicegrp. You would the add lines

myself ALL=(myself:spicegrp) /usr/bin/rviewer
myself ALL=(mybro:spicegrp) /usr/bin/rviewer

to the "/etc/sudoers"-file

Important note:

You must not forget to check the fie permission of the file "/usr/bin/remote-viewer" after SW-updates or upgrades of your system.

I would recommend to start a small scheduled job or a service to check the rights settings frequently.

Conclusion

Remote-viewer connections to VMs cannot be controlled on a user level by the KVM/Qemu server if we just used TLS and SASL. We can set up a VM specific password. But external connections to a VM specific TLS port can only be blocked for external systems and IPs on the server.
However, on Linux client-systems iptables helps us to allow access to the Spice console of a specific VM for a selected user, only. This can be achieved by setting up user specific iptables rules on client-systems. This post has shown you how to create such rules for the OUTPUT chain with fwbuilder. We must set up a systemd service to implement these rules automatically at system (and network) startup.

To restrict users on Linux client systems even more we applied the sudo mechanism in a very specific way: we enforced the usage of certain arguments to the remote-viewer command for specific users. I think the method I discussed is safe; if you find a caveat please send me a mail.

Both strategies can and should be applied in Intranets where we want to provide remote-viewer and the Spice consoles of Qemu VMs as real working instruments.

ufw auf Strato-vServern mit Debian 8 – fehlende iptables Log-Meldungen im systemd-Journal – rsyslogd

Gestern hatte ich das Vergnügen, ein Debian-Server-System auf einer aktuellen vServer-Plattform bei Strato einzurichten. Ich bereite entsprechende Arbeiten in der Regel vor, indem ich elementare Konfigurationsschritte - im Besonderen solche, die sicherheitsrelevant sind - vorab auf einem ähnlichen KVM-Gast-System in unserem Hausnetz simuliere und teste.

Diese Art von vorbereitenden Tests hat jedoch ihre Grenzen; nicht alles ist vergleichbar. Gestern bin ich mal wieder auf einen Unterschied im Zusammenhang mit iptables, ufw und den zugehörigen LOG-Meldungen unter systemd gestoßen. Letztere fehlten nämlich im systemd-Journal des Strato-vServers völlig.

Da fragt man sich schon, wie man denn unter solchen Voraussetzungen das gehostete System bzgl. von Angriffsmustern monitoren soll. Ich finde, diese Frage ist so relevant, dass sie sich auch andere Strato-Kunden besser vor dem Mieten eines vServers beantworten sollten. Deshalb dieser Post. Die gute Nachricht ist: Es gibt unabhängig von den Ursachen für das Fehlen der LOG-Meldungen einen Workaround.

Die schlechte Nachricht ist: Die Ursache der fehlenden Kernel-Meldungen im systemd-Journal ist unklar; zumindest mir. Auf einem KVM-Host funktioniert alles wie erwartet. Unterschiede zu gehosteten Servern sind meist auf einen anderen Ansatz in der Virtualisierung zurückzuführen (Stichwort: Container-Technologie vs. Hypervisor für Full/Para-Virtualisierung).

In diesem Falle erscheint mir das aber als Erklärungsansatz nicht plausibel und hinreichend. Ich gehe nachfolgend auf die Gründe etwas genauer ein. Zudem ist bei Debian 8 (leider) eben auch systemd in den Logging-Prozess involviert. Defizite von "systemd" in der Interaktion mit bestimmten Virtualisierungsumgebungen halte ich für durchaus möglich. Der Irrwitz, dass ein Programm beim Systemstart die Umgebung analysieren und für jeden Fall die richtige Antwort ziehen muss, hat halt seinen Preis ...

ufw, netfilter/iptables und das Logging-Problem

Ich bin eigentlich ein Freund von Firewall-Builder (FWB). Für Debian-Systeme verwende ich aber auch "ufw", um initial die wichtigsten Paketfilter-Regeln, also iptables-Anweisungen, bequem und zeitsparend aufzusetzen. Die drehen sich zunächst um den SSH-Zugang von außen und die Erlaubnis, dass der gehostete Server DNS-Server, NTP-Server und bestimmte Update-Server kontaktieren darf. Auch "pings" und "traceroute" vom Server nach außen erlaube ich. Alles andere wird von mir anfänglich rigoros geblockt. Später wird dann für die angestrebten Services des Servers gezielt nachgearbeitet. (Off topic: Viele Dienste, die mein Kunde benötigt, tunnele ich auf dem Server über eine SSH-Verbindung; ein direkter SSH-Zugang des Users root wird sowieso unterbunden und der SSH-Port verschoben.)

Anfänglich ist hinsichtlich eines minimalen Regelsatzes gar nicht viel zu tun. Im Anschluss an das Etablieren der ersten Paketfilter-Regeln möchte ich gerne die Arbeit von "netfilter" testen und das zugehörige Logging mitverfolgen. Typischerweise lasse ich dann "nmap" von außen auf das gehostete System los. Für einen Test des Serverzugriffs auf externe DNS-Dienste und Zugriffe auf Update-Server tut es dagegen "apt-get". In beiden Fällen verfolge ich per SSH auf einem (Remote-) Terminal den Strom der Meldungen der (ufw-)"Firewall".

Das erhoffte Verfolgen der iptables-Log-Meldungen schlug auf dem Strato-vServer mit installiertem Debian 8 aber fehl.

Debian 8.x nutzt wie gesagt systemd. Ufw schreibt die iptables-Log-Daten mit eigenen Zusätzen in das Log-System des Servers - bei einem systemd-basierten Systemen also in das dortige binäre Journal. Das systemd-Journal fängt im Normalfall neben System-Meldungen und Meldungen aus dem Userspace auch Kernel-Messages auf. Da systemd den gesamten Mix aus Messages in ein binäres Datenformat in einer Datei überführt, muss man das Kommando "journalctl" mit geeigneten Filtern bzw. der Option "-f -nxxx" benutzen, um Log-Einträge auswerten bzw. direkt am Schirm mitverfolgen zu können.

Gesagt, getan. Leider tauchen auf einem Strato-vServer im Journal von "systemd" generell nur sehr wenig Informationen auf; hinsichtlich der Paketfilter-LOG-Meldungen findet man dort jedoch leider gar nichts.

Das iptables-Target "LOG" mündet auf einem mit "rsyslogd" ausgestatteten Log- und Warnsystem dagegen in Meldungen in der Datei "/var/log/kern.log" - schließlich handelt es sich ja um Kernel-Meldungen.

Die aus meiner Sicht schon immer kritikwürdige Idee, alle systemrelevanten Meldungen an einer Stelle in einem Binärformat zu sammeln, wird uns auf einem Strato vServer nun offenbar zum Verhängnis: Nur mit systemd können wir kleine und große externe Zugriffsversuche auf einen vServer offenbar nicht überwachen!

Ich bin übrigens nicht der Einzige, der dieses Problem hatte; siehe:
http://linux.debian.user.german.narkive.com/8AbyxJTP/keine-eintrage-von-dmesg-im-journal-systemd
Erstaunlich ist dennoch, dass man ansonsten im Internet fast nichts zu dieser Thematik findet.

Ob das Problem nun etwas mit systemd-Defiziten oder einer speziellen Konfiguration der systemd-Interaktion mit der Virtualisierungsumgebung bei Strato zu tun hat, muss man natürlich ein wenig austesten.

Firewall-Logging, Virtualisierung und Container

Tatsächlich erweist sich das Verhalten von Debian 8 mit "ufw" auf einem KVM-Gastsystem als gänzlich anders. Hier ein Auszug der ufw-Meldungen von einem KVM-Gast mit Debian 8, die mittels des Befehls

"journalctl -f -n20"

zur Anzeige gebracht wurden:

Apr 05 16:05:52 deb11 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:d5:4a:9b:52:54:00:fc:27:c5:08:00 SRC=192.168.10.1 DST=192.168.10.11 LEN=44 TOS=0x00 PREC=0x00 TTL=54 ID=25556 PROTO=TCP SPT=64358 DPT=110 WINDOW=1024 RES=0x00 SYN URGP=0 
Apr 05 16:05:52 deb11 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:d5:4a:9b:52:54:00:fc:27:c5:08:00 SRC=192.168.10.1 DST=192.168.10.11 LEN=44 TOS=0x00 PREC=0x00 TTL=59 ID=47868 PROTO=TCP SPT=64358 DPT=135 WINDOW=1024 RES=0x00 SYN URGP=0 
Apr 05 16:05:52 deb11 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:d5:4a:9b:52:54:00:fc:27:c5:08:00 SRC=192.168.10.1 DST=192.168.10.11 LEN=44 TOS=0x00 PREC=0x00 TTL=45 ID=41401 PROTO=TCP SPT=64358 DPT=53 WINDOW=1024 RES=0x00 SYN URGP=0 
Apr 05 16:05:52 deb11 kernel: [UFW ALLOW] IN=eth0 OUT= MAC=52:54:00:d5:4a:9b:52:54:00:fc:27:c5:08:00 SRC=192.168.10.1 DST=192.168.10.11 LEN=44 TOS=0x00 PREC=0x00 TTL=42 ID=10106 PROTO=TCP SPT=64358 DPT=22 WINDOW=1024 RES=0x00 SYN URGP=0 
Apr 05 16:05:52 deb11 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:d5:4a:9b:52:54:00:fc:27:c5:08:00 SRC=192.168.10.1 DST=192.168.10.11 LEN=44 TOS=0x00 PREC=0x00 TTL=54 ID=65381 PROTO=TCP SPT=64358 DPT=113 WINDOW=1024 RES=0x00 SYN URGP=0 
Apr 05 16:05:52 deb11 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:d5:4a:9b:52:54:00:fc:27:c5:08:00 SRC=192.168.10.1 DST=192.168.10.11 LEN=44 TOS=0x00 PREC=0x00 TTL=47 ID=1758 PROTO=TCP SPT=64358 DPT=587 WINDOW=1024 RES=0x00 SYN URGP=0 
Apr 05 16:05:52 deb11 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:d5:4a:9b:52:54:00:fc:27:c5:08:00 SRC=192.168.10.1 DST=192.168.10.11 LEN=44 TOS=0x00 PREC=0x00 TTL=47 ID=22236 PROTO=TCP SPT=64358 DPT=443 WINDOW=1024 RES=0x00 SYN URGP=0 
Apr 05 16:05:52 deb11 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:d5:4a:9b:52:54:00:fc:27:c5:08:00 SRC=192.168.10.1 DST=192.168.10.11 LEN=44 TOS=0x00 PREC=0x00 TTL=55 ID=60478 PROTO=TCP SPT=64358 DPT=23 WINDOW=1024 RES=0x00 SYN URGP=0 
Apr 05 16:05:52 deb11 kernel: [UFW BLOCK] IN=eth0 OUT= 

 
Offensichtlich führt im obigen Fall das System 192.168.10.1 einen Portscan auf dem betroffenen KVM-Host mit der IP 192.168.10.11 durch.

Ähnliche Meldungen erhält man bei einem Portscan auf einem vServer aber - wie gesagt - nicht.

Wie könnte man das erklären?
Ein naheliegender Erklärungsansatz wäre etwa folgender:
Das Logging von Kernel-Messages klappt auf einem KVM-Gast, also unter dem QEMU-Hypervisor (sog. Typ 2 Hypervisor), der über "virtio" auf dem Host nur partiell Paravirtualisierung und keine Container-Technologie einsetzt, natürlich problemfrei. Das Betriebssystem des KVM-Gastes und dessen Kernel arbeiten ja weitgehend autonom und greifen nur über Vermittlungsschichten auf den Kernel des Hosts und dessen HW-Unterstützung zu. Es besteht von Haus aus kein Problem bzgl. des Loggings von Kernel-Meldungen - sie beziehen sich immer auf den Kernel des Gastsystems.

Dagegen setzt Strato Container-Technologie ein - genauer VZ-Container unter Virtuozzo; selbiges basiert auf OpenVZ. Zu Grundeigenschaften siehe :
https://openvz.org/Main_Page und https://openvz.org/Features

Es handelt sich bei Strato wohl um Version 4.7 oder eine frühe 6er Version von "Virtuozzo Containers". Dafür gibt es Indizien (u.a, dass sich Docker nicht installieren lässt); um einen genauen Nachweis habe ich mich aber (noch) nicht gekümmert. Ist auch egal.

In einer Container-Lösung wird jedenfalls die Kapazität und Funktionalität des Host-Kernels zwischen den Containern, die keinen eigenen Kernel besitzen, geteilt (schlanker "Single-Kernel-Approach"). Der Zugriff auf Netze erfolgt über eine entsprechende Netzwerk- und Schnittstellen-Virtualisierung. Typischerweise werden virtuelle venet- oder veth-NICs eingesetzt; je nachdem, auf welcher Ebene OSI-Stacks man arbeiten will. (veth-NICs setze ich selbst vielfach auch in komplexeren KVM/Qemu-Umgebungen bei der Netzwerkvirtualisierung ein.)

Die notwendige Separation der Container und ihrer Netzwerk-Kommunikation gegeneinander und gegenüber dem Host muss vom Host-Kernel bzw. dessen Modulen auf der Basis von Konfigurationsvorgaben für unpriviligierte Container (in ihren separaten Namespaces und bei modernen Ansätzen ggf. in cGroups) gewährleistet werden. Man wird den Container-Systemen jedenfalls nicht erlauben, alles einzusehen, was auf dem für alle zuständigen Host-Kernel abläuft. Dies bedeutet u.a., dass Containersysteme nicht beliebige Kernel-Module (z.B. für Packettracking unter Wireshark) laden dürfen.

Wer "iptables" im Zusammenhang mit Virtualisierungshosts aber ein wenig genauer kennt, kann sich vorstellen, dass man eine Host-Firewall natürlich immer so konfigurieren kann, dass die einzelnen virtuellen Netzwerkschnittstellen der (Container-) Gäste gegeneinander geblockt werden, aber dass generelle Forward-Regeln für physikalische Interfaces des Hosts nicht in Konflikt mit speziellen Filter-Regeln für ein spezifisches (virtuelles) Gast-Interface geraten müssen.

OpenVZ kann man deshalb sehr wohl so einrichten, dass der Admin eines Container-Systems seine eigenen iptables-Regeln für seine gastspezifischen NICs definieren kann. Siehe hierzu z.B.:
https://openvz.org/Setting_up_an_iptables_firewall.

Wesentliche Teile der verschiedenen netfilter-Module - im Besonderen für die Schicht 3 - stehen also auch Gästen zur Verfügung. Voraussetzung ist in einer Container-Architektur natürlich, dass grundlegende "netfilter"-Module auf dem OpenVZ-Host selbst geladen wurden.

Aber: Es wäre fahrlässig, wenn ein Container-Host alle netzwerkspezifischen Kernel-Meldungen (darunter iptables-Meldungen) auch für die Einsichtnahme durch die root-User der Container preisgeben würde. Das würde u.a ein Ausspionieren der virtuellen Netzwerkumgebung und darauf aufbauend bestimmte Angriffsszenarien ermöglichen. Wenn wir überhaupt etwas im Container sehen, dann höchstens Meldungen zu selbst gesetzten Paketfilterregeln für die Container-spezifische NIC.

Zwischenfazit:

  • Wir dürfen uns in einer Container-Umgebung u.a. nicht darüber wundern, dass man bestimmte Kernel-Module vom Container aus erst gar nicht laden darf und z.B. lsmod eine vernünftige Antwort schuldig bleibt.
  • Wir dürfen uns nicht wundern, dass bestimmte sysctl-Befehle, die im Container abgesetzt werden, ggf. ignoriert werden.
  • Wir dürfen uns in einer Container-Umgebung nicht wundern, wenn man bestimmte Teile des systemd-Logs auf einem Container - und damit auf einem Strato-V-Server - nicht ggf. zu Gesicht bekommt. (Im Gegensatz zu einem KVM-Gast).

Der erste Punkt ist u.a. für den Betrieb der ufw relevant; s.u..

Bzgl. des zweiten Punktes ist zu beachten, dass OpenVZ, genauer der OpenVZ-Kernel, (network-) "Namespaces" nutzt. ("Namespaces" werden natürlich aber auch von aktuellen Linux-Kerneln unterstützt. Zu "Namespaces" siehe etwa
https://jvns.ca/blog/2016/10/10/what-even-is-a-container/
https://de.slideshare.net/jpetazzo/anatomy-of-a-container-namespaces-cgroups-some-filesystem-magic-linuxcon
https://openvz.org/WP/What_are_containers.

Deshalb lassen sich bestimmte Einstellung unterhalb von "/proc/sys" durchaus auch vom Container aus anpassen. Was in der jeweiligen OpenVZ-Umgebung erlaubt ist und was nicht, muss man ggf. durch Probieren herausfinden.

Den dritten Punkt werden wir in den nächsten Abschnitten für vServer kritisch hinterfragen.

Erste Konsequenzen für den Einsatz von "ufw"

Auch durch Probieren wird man herausfinden, dass "ufw" auch auf einem mit Debian 8 betriebenen vServer von Strato läuft - und man eigene iptables-Regeln problemfrei an den OpenVZ-Kernel weiterreichen kann.

Achtung:

Nach der Installation von ufw auf dem vServer NICHT unmittelbar "ufw enable" absetzen!! Zuerst den Port, auf dem man SSH betreibt, freischalten. Also etwa durch "ufw allow 22", wenn man den SSH-Standardport benutzt. Ihr wollt euch ja nicht durch Anschalten der Firewall selbst aussperren!

Wie man schnell (hier durch einen Blick in das systemd-Journal feststellt), ist der Start von ufw auf einem vServer - auch im Rahmen eines Systemstarts - mit ein paar Fehlermeldungen verbunden.

Apr 05 14:21:03 xxx.stratoserver.net ufw[147]: Starting firewall: ufw...modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not ope
Apr 05 14:21:03 xxx.stratoserver.net ufw[147]: modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file '/lib/modul
Apr 05 14:21:03 xxx.stratoserver.net ufw[147]: modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file '/lib/modul
Apr 05 14:21:03 xxx.stratoserver.net systemd-journal[114]: Permanent journal is using 24.0M (max allowed 4.0G, trying to leave 4.0G free of 499.5G a
Apr 05 14:21:03 xxx.stratoserver.net systemd-journal[114]: Time spent on flushing to /var is 1.770ms for 8 entries.
Apr 05 14:21:03 xxx.stratoserver.net ufw[147]: sysctl: permission denied on key 'net.ipv4.tcp_sack'
Apr 05 14:21:03 xxx.stratoserver.net ufw[147]: Setting kernel variables (/etc/ufw/sysctl.conf)...done.

 
Diese Meldungen rühren u.a. daher, dass ufw mit Hilfe von modprobe versucht, bestimmte "conntrack"-Sub-Module zu laden. Zudem versucht ufw per sysctl Kernel-Parameter abzuändern. Vorgegeben sind diese Schritte in den Dateien "/etc/default/ufw" und "/etc/ufw/sysctl.conf".

Die genannten Fehler blockieren den Start aktueller ufw-Versionen aber nicht; wer sich dennoch an den Meldungen stört, kann die über Modifikationen der genannten Dateien, nämlich durch Auskommentieren der fehlerträchtigen Statements, verhindern. Siehe auch
https://www.hosteurope.de/faq/server/virtual-server/besonderheiten-firewall-virtual-server/; im Unterschied zu den dortigen Tipps aber beachten, dass auf dem vServer nur einer der sysctl-Befehle problematisch ist.)

Übrigens: Über das Starten von ufw bei einem Reboot des vServers muss man sich nach einem Absetzen von

systemctl enable ufw

keine Gedanken mehr machen. Debian 8 beinhaltet für ufw einen passenden LSB-Service, der beim Hochfahren ausgeführt wird.

Monitoring von ufw-iptables-Meldungen auf dem vServer mit Hilfe von dmesg

Nachdem man in systemd-Journal nichts findet: Gibt es andere Möglichkeiten, die LOG-Messages von ufw-/iptables zu verfolgen?

Da es sich um Kernel-Messages handelt, liegt ein versuchsweiser Blick auf den "dmesg"-Output nahe. Und tatsächlich - auf meinem vServer:

    
root@xxx:~ # dmesg
[1240949.664984] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=45.55.2.201 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=40788 DPT=3306 WINDOW=65535 RES=0x00 SYN URGP=0 
[1240954.051018] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=93.174.93.136 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=250 ID=4710 PROTO=TCP SPT=43745 DPT=3128 WINDOW=1024 RES=0x00 SYN URGP=0 
[1240986.448807] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=45.55.1.72 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=46822 DPT=1900 WINDOW=65535 RES=0x00 SYN URGP=0 
[1241002.495868] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=88.100.184.82 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=50 ID=13380 PROTO=TCP SPT=35180 DPT=23 WINDOW=44554 RES=0x00 SYN URGP=0 
[1241015.141452] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=89.163.144.224 DST=xxx LEN=445 TOS=0x00 PREC=0x00 TTL=57 ID=32820 DF PROTO=UDP SPT=5180 DPT=5046 LEN=425 
[1241132.233004] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=197.44.69.222 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=239 ID=44476 PROTO=TCP SPT=53226 DPT=1433 WINDOW=1024 RES=0x00 SYN URGP=0 
[1241145.520318] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=208.100.26.228 DST=xxx LEN=40 TOS=0x08 PREC=0x00 TTL=242 ID=51210 PROTO=TCP SPT=47975 DPT=15672 WINDOW=1024 RES=0x00 SYN URGP=0 
[1241185.158299] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=89.163.144.224 DST=xxx LEN=443 TOS=0x00 PREC=0x00 TTL=57 ID=56812 DF PROTO=UDP SPT=5400 DPT=4000 LEN=423 
[1241297.661764] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=186.45.130.20 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=12134 PROTO=TCP SPT=63715 DPT=23 WINDOW=14600 RES=0x00 SYN URGP=0 
[1241350.742715] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=89.163.144.224 DST=xxx LEN=446 TOS=0x00 PREC=0x00 TTL=57 ID=14769 DF PROTO=UDP SPT=5320 DPT=5172 LEN=426 
[1241353.098569] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=5.53.113.195 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=51 ID=48667 PROTO=TCP SPT=46919 DPT=23 WINDOW=39535 RES=0x00 SYN URGP=0 
[1241377.620483] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=46.152.41.83 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=40038 PROTO=TCP SPT=12600 DPT=23 WINDOW=14600 RES=0x00 SYN URGP=0 
[1241386.187457] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=122.114.187.140 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=238 ID=8477 PROTO=TCP SPT=46170 DPT=23 WINDOW=1024 RES=0x00 SYN URGP=0 
[1241437.193431] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=218.91.210.142 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=48 ID=4557 PROTO=TCP SPT=45821 DPT=23 WINDOW=27541 RES=0x00 SYN URGP=0 
[1241512.054090] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=89.163.144.224 DST=xxx LEN=443 TOS=0x00 PREC=0x00 TTL=57 ID=37720 DF PROTO=UDP SPT=5179 DPT=1028 LEN=423 
[1241553.246515] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=49.4.143.59 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=104 ID=256 PROTO=TCP SPT=6000 DPT=135 WINDOW=16384 RES=0x00 SYN URGP=0 
[1241632.706391] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=39.71.216.3 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=48 ID=4841 PROTO=TCP SPT=57398 DPT=22 WINDOW=55725 RES=0x00 SYN URGP=0 
[1241643.559480] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=196.202.5.43 DST=xxx LEN=44 TOS=0x00 PREC=0x00 TTL=48 ID=53691 PROTO=TCP SPT=34866 DPT=23 WINDOW=33710 RES=0x00 SYN URGP=0 
[1241674.241572] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=89.163.144.224 DST=xxx LEN=446 TOS=0x00 PREC=0x00 TTL=57 ID=60393 DF PROTO=UDP SPT=5288 DPT=1029 LEN=426 
[1241683.411659] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=124.167.232.138 DST=xxx LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=40536 DF PROTO=TCP SPT=57844 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0 
[1241686.407572] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=124.167.232.138 DST=xxx LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=40537 DF PROTO=TCP SPT=57844 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0   

 
Ja, die Welt ist schlecht - und wir tun offenbar gut daran, den Zugang zum Server zu blocken bzw. die Logs auch mal auszuwerten und später Blacklists einzusetzen. "fail2ban" zu konfigurieren schadet nebenbei auch nichts.

Wer periodische Updates des Outputs von dmesg ähnlich zu "tail -f" verfolgen will, probiere mal das Kommando

watch -n 0,2 "dmesg | tail -n $((LINES-6))"

in einem Terminal aus. (Ggf. das Terminalfenster etwas vergößern. Auf englischsprachigen Systemen "0.2" statt wie hier "0,2" ! Auf neueren Kerneln als dem der aktuellen vServer gibt es übrigens auch die dmesg-Option "-w").

Aber das eigentlich Feststellenswerte ist ja, dass wir überhaupt was sehen!

Natürlich ist das, was man unter OpenVZ unter dmesg zu Gesicht bekommt, eingeschränkt (s. etwa https://bugs.openvz.org/browse/OVZ-5328).
Aber:
Der OpenVZ-Kernel liefert dem Container zulässige, relevante Informationen in den lokalen Message-Ringpuffer, die dort von root eingesehen werden können. Darunter auch die ersehnten iptables-Meldungen!

Nun stellt sich die große Frage, wie systemd mit diesen Kernelmeldungen interagiert und warum das, was unter dmesg erschient, nicht ins Journal der OpenVZ-Container-Umgebung eingestellt wird.

Das Problematische an systemd ist, wie immer, dass man das ohne seitenweises Lesen in systemd Blogs etc. und/oder gar Codestudium vermutlich nicht beantworten kann. Logisch erscheint mir das Ganze jedenfalls nicht. OpenVZ sorgt offenbar für eine Reduktion der Kernelinformationen auf das, was root im Container sehen sollte. iptables-Meldungen zur lokalen NIC des Containers werden dabei nicht ausgespart. Sie sollten daher eigentlich auch im systemd-Journal erscheinen!

Monitoring mittels rsyslogd ? !

Durch den dmesg-Test ermutigt, fragte ich mich, was wohl passieren würde, wenn rsyslog auf dem vServer-Debian-System installiert und aktiviert wäre. Das ist insofern interessant, als systemd ja externe Linux System-Logging-Services über Schnittstellen bedient und trotzdem sein eigenes Journal weiter versorgt. Man loggt dann im Normalfall sozusagen zweimal ...

Eigentlich würde man nun erwarten, dass die Meldungen von dmesg auch in den verschiedenen Dateien, die der rsyslog-Dämon bedient, nicht auftauchen sollten. Weil systemd ja schon den Transfer in die eigene Binärdatei verweigert. Also machen wir mal die Probe:

root@xxx:~# apt-get rsyslog
root@xxx:~# systemctl start rsyslog
root@xxx:~# systemctl enable rsyslog

Wenn nun doch etwas passieren sollte, so müssten iptables-Meldungen als Einträge unter "/var/log/kern.log" und/oder in der von ufw vorgesehenen Datei "/var/log/ufw.log" auftauchen.

Und tatsächlich finden wir (wider Erwarten) nach einer Weile in "kern.log" iptables-LOG-Meldungen:

  
Apr  6 13:19:57 xxx kernel: [1245761.082097] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=118.163.90.134 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=48 ID=14764 PROTO=TCP SPT=33202 DPT=23 WINDOW=44186 RES=0x00 SYN URGP=0 
Apr  6 13:21:03 xxx kernel: [1245827.018789] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=85.90.163.248 DST=xxx LEN=44 TOS=0x00 PREC=0x00 TTL=52 ID=358 PROTO=TCP SPT=21965 DPT=23 WINDOW=12453 RES=0x00 SYN URGP=0 
Apr  6 13:21:16 xxx kernel: [1245840.027789] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=89.163.144.224 DST=xxx LEN=445 TOS=0x00 PREC=0x00 TTL=57 ID=61654 DF PROTO=UDP SPT=5201 DPT=4011 LEN=425 
Apr  6 13:22:26 xxx kernel: [1245910.326051] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=91.98.36.115 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=46 ID=3739 PROTO=TCP SPT=31242 DPT=23 WINDOW=22306 RES=0x00 SYN URGP=0 
Apr  6 13:23:13 xxx kernel: [1245957.077099] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=117.193.182.117 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=48 ID=29137 PROTO=TCP SPT=12209 DPT=22 WINDOW=52845 RES=0x00 SYN URGP=0 
Apr  6 13:23:54 xxx kernel: [1245997.536218] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=45.55.1.114 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=43185 DPT=8123 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr  6 13:23:56 xxx kernel: [1246000.108495] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=192.151.169.29 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=51 ID=46742 PROTO=TCP SPT=54014 DPT=23 WINDOW=49390 RES=0x00 SYN URGP=0 
Apr  6 13:24:05 xxx kernel: [1246008.982092] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=89.163.144.224 DST=xxx LEN=445 TOS=0x00 PREC=0x00 TTL=57 ID=19866 DF PROTO=UDP SPT=5391 DPT=4012 LEN=425 
Apr  6 13:24:08 xxx kernel: [1246011.450635] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=163.172.204.214 DST=xxx LEN=447 TOS=0x00 PREC=0x00 TTL=58 ID=25411 DF PROTO=UDP SPT=5440 DPT=5060 LEN=427 
Apr  6 13:24:18 xxx kernel: [1246022.133966] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=181.193.99.26 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=48 ID=47683 PROTO=TCP SPT=36520 DPT=23 WINDOW=41856 RES=0x00 SYN URGP=0 
Apr  6 13:24:29 xxx kernel: [1246032.599018] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=89.248.166.146 DST=xxx LEN=48 TOS=0x00 PREC=0x00 TTL=123 ID=26103 PROTO=TCP SPT=11206 DPT=2083 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr  6 13:24:50 xxx kernel: [1246053.507827] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=71.165.26.106 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=50 ID=381 PROTO=TCP SPT=29725 DPT=23 WINDOW=3178 RES=0x00 SYN URGP=0 
Apr  6 13:25:44 xxx kernel: [1246108.065452] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=139.162.118.251 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=246 ID=54321 PROTO=TCP SPT=56219 DPT=6379 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr  6 13:26:16 xxx kernel: [1246139.839711] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=174.16.249.159 DST=xxx LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=54734 PROTO=TCP SPT=43074 DPT=23 WINDOW=14600 RES=0x00 SYN URGP=0 
Apr  6 13:26:28 xxx kernel: [1246151.805797] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=117.1.220.212 DST=xxx LEN=44 TOS=0x00 PREC=0x00 TTL=242 ID=2085 PROTO=TCP SPT=14216 DPT=5358 WINDOW=14600 RES=0x00 SYN URGP=0 
Apr  6 13:26:50 xxx kernel: [1246174.013725] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=123.151.42.61 DST=xxx LEN=135 TOS=0x00 PREC=0x00 TTL=47 ID=32696 PROTO=UDP SPT=9019 DPT=1701 LEN=115 
Apr  6 13:27:01 xxx kernel: [1246184.993843] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=89.163.144.224 DST=xxx LEN=445 TOS=0x00 PREC=0x00 TTL=57 ID=44551 DF PROTO=UDP SPT=5365 DPT=4013 LEN=425 
Apr  6 13:27:27 xxx kernel: [1246211.300971] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=89.120.177.89 DST=xxx LEN=44 TOS=0x00 PREC=0x00 TTL=55 ID=64605 PROTO=TCP SPT=59589 DPT=23 WINDOW=64213 RES=0x00 SYN URGP=0 

 
Die Welt ist inzwischen nicht besser geworden, aber nun können wir das Schlechte wenigstens mal verfolgen. (Dieselben Einträge erscheinen übrigens auch in ufw.log).

Tja, nicht alles im Leben mit systemd ist offenbar nachvollziehbar ....

Fazit

Das erwartete native Zusammenspiel zwischen einem Strato OpenVZ vServer und systemd unter Debian 8 funktioniert bzgl. der Protokollierung der LOG-Target-Meldungen von iptables nicht. Kernel-Meldungen zu iptables-Rules für die Container-NIC erschienen nicht im Journal von systemd.

Workarounds bestehen darin

  • den Output von dmesg kontinuierlich per Skript abzufragen und in eine Datei umzulenken.
  • rsyslog zu installieren und die Arbeit dem Zusammenspiel von systemd mit dem rsyslog-Dämon zu überlassen. Man protokolliert dann doppelt, aber man erhält wenigstens dauerhafte Log-Protokolle, die jeder sicherheitsbewußte Admin zur vorsorglichen Gefahrenabwehr benötigt.

Da die gewünschten Kernel-Meldungen bei gleicher Debian- und Systemd-Version in einem KVM-Gast erscheinen, ist das Problem mit dem systemd-Journal entweder

  • auf einen Fehler oder ein Sicherheitsfeature der OpenVZ-Umgebung,
  • oder auf ein seltsames (gewolltes oder fehlerhaftes) Zusammenspiel des OpenVZ-Kernels mit systemd in den Container-Umgebungen,
  • oder schlicht auf ein fehlerhaftes, bislang nicht erkanntes/bedachtes Verhalten von systemd in einem OpenVZ-Container

zurückzuführen.

Für letzteres spricht die Tatsache, dass der OpenVZ-Kernel iptables-Meldungen zur Container-NIC unter dmesg offenbart und dass systemd die Meldungen, die unter dmesg auftauchen wohl korrekt an weitere System-Logging-Services wie rsyslog weiterleitet.

Eine entsprechende Anfrage bei Strato, die hoffentlich Virtuozzo einschalten, läuft.

Das Fehlen von iptables-Log-Protokolle ist im Sinne der ISO 27000 (Strato hat da ein Zertifikat!) zudem als Sicherheitsproblem einzustufen, das Kunden kommentarlos zugemutet wird und von diesen Kunden selbst gelöst werden muss.

Weiterführende Links

Fehlende Einträge im systemd-Journal
http://linux.debian.user.german.narkive.com/8AbyxJTP/keine-eintrage-von-dmesg-im-journal-systemd

Container, Virtuozzo, OpenVZ und iptables, ufw
http://forum.openvirtuozzo.org/index.php?t=msg&goto=37264&&srch=container
https://openvz.org/Setting_up_an_iptables_firewall
http://askubuntu.com/questions/399624/ubuntu-server-12-04-and-ufw-failure-on-startup-and-several-module-not-found-err
https://www.hosteurope.de/faq/server/virtual-server/besonderheiten-firewall-virtual-server/
https://superuser.com/questions/659236/permission-denied-when-setting-values-in-sysctl-on-ubuntu-12-04
https://help.ubuntu.com/community/UFW
https://help.virtuozzo.com/customer/en/portal/articles/2509437?_ga=1.206607316.635252319.1491384754
ab S. 317 in folgender Referenz
http://www.odin.com/fileadmin/parallels/documents/hosting-cloud-enablement/pvc/Production_Documents/VzLinuxUG_03132013.pdf
https://bugs.openvz.org/browse/OVZ-5328

Namespaces
http://www.netdevconf.org/1.1/proceedings/slides/rosen-namespaces-cgroups-lxc.pdf

LXC vs. OpenVZ
https://www.janoszen.com/2013/01/22/lxc-vs-openvz/
https://en.wikipedia.org/wiki/LXC
https://openvz.org/Comparison

OpenVZ integrates KVM/Qemu
http://openvz.livejournal.com/tag/criu
https://www.heise.de/ix/meldung/Virtualisierungsplattform-OpenVZ-wird-eigenstaendige-Distribution-3278115.html
https://openvz.org/Virtuozzo
https://openvz.org/QEMU
https://www.heise.de/ix/meldung/Virtualisierungsplattform-OpenVZ-wird-eigenstaendige-Distribution-3278115.html
https://openvz.org/FAQ
https://virtuozzo.com/virtual-machines-in-virtuozzo-7/
https://openvz.org/WP/What_are_containers#Networking

Virtualisierungsangebote in D - Vergleich
https://timreeves.de/trip-content/uploads/dokumente/Internet-Mietserver-Typen_im-Vergleich.pdf

Watch dmesg Output
http://unix.stackexchange.com/questions/95842/how-can-i-see-dmesg-output-as-it-changes

Linux bridges – can iptables be used against MiM attacks based on ARP spoofing ? – III

This small series of blog contributions was written to understand a little better how to use iptables in the context of Linux bridges as a countermeasure against some of the effects of a man-in-the-middle [MiM] attack based on ARP spoofing. The attacking system as well as the attacked systems are in our scenarios attached to Linux bridge ports. My objective was to block redirected TCP/IP packets to the the attacking system.

In the first article
Linux bridges – can iptables be used against MiM attacks based on ARP spoofing ? – I
we had discussed how we have to set up iptables rules for ports of a single Linux bridge and their associated IP-addresses to get the desired blocking. We found that a certain order of DENY and ACCEPT rules is required.

In the second article
Linux bridges – can iptables be used against MiM attacks based on ARP spoofing ? – II
we investigated how iptables reacts to the existence of multiple and linked Linux bridges on one and the same hast. We defined "border ports" as ports that connect a Linux bridge to other bridges, to extended external network segments or to the virtualization host itself - but not to guests directly attached to the bridge via tap or veth-devices. "Border ports" may be passed by packets traveling to their destination across multiple bridges. We then extended our previous considerations on iptables rules to such "border ports" and found a general recipe for the order of the required DENY and ACCEPT rules for the ports of the multiple bridges.

In this article we shall test the required rules for the bridge test setup presented in the previous article. We consider some examples of attack variations with respect to the 2 bridges and ICMP packets. However, the tests would also work for any TCP based service. The reason is that the central DENY rules are very general and compiled without reference to any specific service type.

We assume that you have had a look at the screenshots of the logical rules displayed in a FWbuilder interface in the last article (II).

Setup and iptables rules created by FWbuilder

See the following drawing for the setup of our test scenario:

bridge3

The general DENY rules and the ICMP-related ACCEPT rules displayed in the last article are compiled by FWbuilder to create the following script commands:


    # Rule 2 (vk63)
    # 
    echo "Rule 2 (vk63)"
    # 
    # virbr6 guest port
    $IPTABLES -N Out_RULE_2
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vk63 !  -d 192.168.50.13   -j Out_RULE_2
    $IPTABLES -A Out_RULE_2  -j LOG  --log-level info --log-prefix "RULE 2 -- DENY "
    $IPTABLES -A Out_RULE_2  -j DROP
    # 
    # Rule 3 (vk64)
    # 
    echo "Rule 3 (vk64)"
    # 
    # virbr6 guest port
    $IPTABLES -N Out_RULE_3
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vk64 !  -d 192.168.50.14   -j Out_RULE_3
    $IPTABLES -A Out_RULE_3  -j LOG  --log-level info --log-prefix "RULE 3 -- DENY "
    $IPTABLES -A Out_RULE_3  -j DROP
    # 
    # Rule 4 (vk65)
    # 
    echo "Rule 4 (vk65)"
    # 
    # virbr6 guest port
    $IPTABLES -N Out_RULE_4
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vk65 !  -d 192.168.50.15   -j Out_RULE_4
    $IPTABLES -A Out_RULE_4  -j LOG  --log-level info --log-prefix "RULE 4 -- DENY "
    $IPTABLES -A Out_RULE_4  -j DROP
    # 
    # Rule 5 (vk42)
    # 
    echo "Rule 5 (vk42)"
    # 
    # virbr4 guest port
    $IPTABLES -N Out_RULE_5
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vk42 !  -d 192.168.50.12   -j Out_RULE_5
    $IPTABLES -A Out_RULE_5  -j LOG  --log-level info --log-prefix "RULE 5 -- DENY "
    $IPTABLES -A Out_RULE_5  -j DROP
    # 
    # Rule 6 (vk63)
    # 
    echo "Rule 6 (vk63)"
    # 
    # virbr6 guest port
    $IPTABLES -N Out_RULE_6
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vk63  -s 192.168.50.13   -j Out_RULE_6
    $IPTABLES -A Out_RULE_6  -j LOG  --log-level info --log-prefix "RULE 6 -- DENY "
    $IPTABLES -A Out_RULE_6  -j DROP
    # 
    # Rule 7 (vk64)
    # 
    echo "Rule 7 (vk64)"
    # 
    # virbr6 guest port
    $IPTABLES -N Out_RULE_7
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vk64  -s 192.168.50.14   -j Out_RULE_7
    $IPTABLES -A Out_RULE_7  -j LOG  --log-level info --log-prefix "RULE 7 -- DENY "
    $IPTABLES -A Out_RULE_7  -j DROP
    # 
    # Rule 8 (vk65)
    # 
    echo "Rule 8 (vk65)"
    # 
    # virbr6 guest port
    $IPTABLES -N Out_RULE_8
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vk65  -s 192.168.50.15   -j Out_RULE_8
    $IPTABLES -A Out_RULE_8  -j LOG  --log-level info --log-prefix "RULE 8 -- DENY "
    $IPTABLES -A Out_RULE_8  -j DROP
    # 
    # Rule 9 (vk42)
    # 
    echo "Rule 9 (vk42)"
    # 
    # virbr4 guest port
    $IPTABLES -N Out_RULE_9
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vk42  -s 192.168.50.12   -j Out_RULE_9
    $IPTABLES -A Out_RULE_9  -j LOG  --log-level info --log-prefix "RULE 9 -- DENY "
    $IPTABLES -A Out_RULE_9  -j DROP
    # 
    # Rule 10 (vk63)
    # 
    echo "Rule 10 (vk63)"
    # 
    # virbr6 guest port
    $IPTABLES -N In_RULE_10
    $IPTABLES -A INPUT -m physdev --physdev-in vk63 !  -s 192.168.50.13   -j In_RULE_10
    $IPTABLES -A FORWARD -m physdev --physdev-in vk63 !  -s 192.168.50.13   -j In_RULE_10
    $IPTABLES -A In_RULE_10  -j LOG  --log-level info --log-prefix "RULE 10 -- DENY "
    $IPTABLES -A In_RULE_10  -j DROP
    # 
    # Rule 11 (vk64)
    # 
    echo "Rule 11 (vk64)"
    # 
    # virbr6 guest port
    $IPTABLES -N In_RULE_11
    $IPTABLES -A INPUT -m physdev --physdev-in vk64 !  -s 192.168.50.14   -j In_RULE_11
    $IPTABLES -A FORWARD -m physdev --physdev-in vk64 !  -s 192.168.50.14   -j In_RULE_11
    $IPTABLES -A In_RULE_11  -j LOG  --log-level info --log-prefix "RULE 11 -- DENY "
    $IPTABLES -A In_RULE_11  -j DROP
    # 
    # Rule 12 (vk65)
    # 
    echo "Rule 12 (vk65)"
    # 
    # virbr6 guest port
    $IPTABLES -N In_RULE_12
    $IPTABLES -A INPUT -m physdev --physdev-in vk65 !  -s 192.168.50.15   -j In_RULE_12
    $IPTABLES -A FORWARD -m physdev --physdev-in vk65 !  -s 192.168.50.15   -j In_RULE_12
    $IPTABLES -A In_RULE_12  -j LOG  --log-level info --log-prefix "RULE 12 -- DENY "
    $IPTABLES -A In_RULE_12  -j DROP
    # 
    # Rule 13 (vk42)
    # 
    echo "Rule 13 (vk42)"
    # 
    # virbr4 guest port
    $IPTABLES -N In_RULE_13
    $IPTABLES -A INPUT -m physdev --physdev-in vk42 !  -s 192.168.50.12   -j In_RULE_13
    $IPTABLES -A FORWARD -m physdev --physdev-in vk42 !  -s 192.168.50.12   -j In_RULE_13
    $IPTABLES -A In_RULE_13  -j LOG  --log-level info --log-prefix "RULE 13 -- DENY "
    $IPTABLES -A In_RULE_13  -j DROP
    # 
    # Rule 15 (vethb2)
    # 
    echo "Rule 15 (vethb2)"
    # 
    # br6 border out
    $IPTABLES -N Cid7404X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vethb2  -j Cid7404X2034.0
    $IPTABLES -A Cid7404X2034.0  -s 192.168.50.13   -j RETURN
    $IPTABLES -A Cid7404X2034.0  -s 192.168.50.14   -j RETURN
    $IPTABLES -A Cid7404X2034.0  -s 192.168.50.15   -j RETURN
    $IPTABLES -N Out_RULE_15_3
    $IPTABLES -A Cid7404X2034.0  -j Out_RULE_15_3
    $IPTABLES -A Out_RULE_15_3  -j LOG  --log-level info --log-prefix "RULE 15 -- DENY "
    $IPTABLES -A Out_RULE_15_3  -j DROP
    # 
    # Rule 16 (vethb2)
    # 
    echo "Rule 16 (vethb2)"
    # 
    # br6 border out
    $IPTABLES -N Cid7478X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vethb2  -j Cid7478X2034.0
    $IPTABLES -A Cid7478X2034.0  -d 192.168.50.1   -j RETURN
    $IPTABLES -A Cid7478X2034.0  -d 192.168.0.37   -j RETURN
    $IPTABLES -A Cid7478X2034.0  -d 192.168.50.12   -j RETURN
    $IPTABLES -N Out_RULE_16_3
    $IPTABLES -A Cid7478X2034.0  -j Out_RULE_16_3
    $IPTABLES -A Out_RULE_16_3  -j LOG  --log-level info --log-prefix "RULE 16 -- DENY "
    $IPTABLES -A Out_RULE_16_3  -j DROP
    # 
    # Rule 17 (vethb1)
    # 
    echo "Rule 17 (vethb1)"
    # 
    # br4 border out
    $IPTABLES -N Cid8637X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vethb1  -j Cid8637X2034.0
    $IPTABLES -A Cid8637X2034.0  -s 192.168.50.1   -j RETURN
    $IPTABLES -A Cid8637X2034.0  -s 192.168.0.37   -j RETURN
    $IPTABLES -A Cid8637X2034.0  -s 192.168.50.12   -j RETURN
    $IPTABLES -N Out_RULE_17_3
    $IPTABLES -A Cid8637X2034.0  -j Out_RULE_17_3
    $IPTABLES -A Out_RULE_17_3  -j LOG  --log-level info --log-prefix "RULE 17 -- DENY "
    $IPTABLES -A Out_RULE_17_3  -j DROP
    # 
    # Rule 18 (vethb1)
    # 
    echo "Rule 18 (vethb1)"
    # 
    # br4 border out
    $IPTABLES -N Cid8753X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vethb1  -j Cid8753X2034.0
    $IPTABLES -A Cid8753X2034.0  -d 192.168.50.13   -j RETURN
    $IPTABLES -A Cid8753X2034.0  -d 192.168.50.14   -j RETURN
    $IPTABLES -A Cid8753X2034.0  -d 192.168.50.15   -j RETURN
    $IPTABLES -N Out_RULE_18_3
    $IPTABLES -A Cid8753X2034.0  -j Out_RULE_18_3
    $IPTABLES -A Out_RULE_18_3  -j LOG  --log-level info --log-prefix "RULE 18 -- DENY "
    $IPTABLES -A Out_RULE_18_3  -j DROP
    # 
    # Rule 19 (vmh1)
    # 
    echo "Rule 19 (vmh1)"
    # 
    # br4 border out
    $IPTABLES -N Cid8374X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vmh1  -j Cid8374X2034.0
    $IPTABLES -A Cid8374X2034.0  -s 192.168.50.12   -j RETURN
    $IPTABLES -A Cid8374X2034.0  -s 192.168.50.13   -j RETURN
    $IPTABLES -A Cid8374X2034.0  -s 192.168.50.14   -j RETURN
    $IPTABLES -A Cid8374X2034.0  -s 192.168.50.15   -j RETURN
    $IPTABLES -N Out_RULE_19_3
    $IPTABLES -A Cid8374X2034.0  -j Out_RULE_19_3
    $IPTABLES -A Out_RULE_19_3  -j LOG  --log-level info --log-prefix "RULE 19 -- DENY "
    $IPTABLES -A Out_RULE_19_3  -j DROP
    # 
    # Rule 20 (vmh1)
    # 
    echo "Rule 20 (vmh1)"
    # 
    # br4 border out
    $IPTABLES -N Cid8530X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-is-bridged --physdev-out vmh1  -j Cid8530X2034.0
    $IPTABLES -A Cid8530X2034.0  -d 192.168.50.1   -j RETURN
    $IPTABLES -A Cid8530X2034.0  -d 192.168.0.37   -j RETURN
    $IPTABLES -N Out_RULE_20_3
    $IPTABLES -A Cid8530X2034.0  -j Out_RULE_20_3
    $IPTABLES -A Out_RULE_20_3  -j LOG  --log-level info --log-prefix "RULE 20 -- DENY "
    $IPTABLES -A Out_RULE_20_3  -j DROP
    # 
    # Rule 22 (vethb2)
    # 
    echo "Rule 22 (vethb2)"
    # 
    # br6 border IN
    $IPTABLES -N Cid7917X2034.0
    $IPTABLES -A INPUT -m physdev --physdev-in vethb2  -j Cid7917X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-in vethb2  -j Cid7917X2034.0
    $IPTABLES -A Cid7917X2034.0  -d 192.168.50.13   -j RETURN
    $IPTABLES -A Cid7917X2034.0  -d 192.168.50.14   -j RETURN
    $IPTABLES -A Cid7917X2034.0  -d 192.168.50.15   -j RETURN
    $IPTABLES -N In_RULE_22_3
    $IPTABLES -A Cid7917X2034.0  -j In_RULE_22_3
    $IPTABLES -A In_RULE_22_3  -j LOG  --log-level info --log-prefix "RULE 22 -- DENY "
    $IPTABLES -A In_RULE_22_3  -j DROP
    # 
    # Rule 23 (vethb2)
    # 
    echo "Rule 23 (vethb2)"
    # 
    # br6 border IN
    $IPTABLES -N Cid8013X2034.0
    $IPTABLES -A INPUT -m physdev --physdev-in vethb2  -j Cid8013X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-in vethb2  -j Cid8013X2034.0
    $IPTABLES -A Cid8013X2034.0  -s 192.168.50.1   -j RETURN
    $IPTABLES -A Cid8013X2034.0  -s 192.168.0.37   -j RETURN
    $IPTABLES -A Cid8013X2034.0  -s 192.168.50.12   -j RETURN
    $IPTABLES -N In_RULE_23_3
    $IPTABLES -A Cid8013X2034.0  -j In_RULE_23_3
    $IPTABLES -A In_RULE_23_3  -j LOG  --log-level info --log-prefix "RULE 23 -- DENY "
    $IPTABLES -A In_RULE_23_3  -j DROP
    # 
    # Rule 24 (vethb1)
    # 
    echo "Rule 24 (vethb1)"
    # 
    # br4 border IN
    $IPTABLES -N Cid7639X2034.0
    $IPTABLES -A INPUT -m physdev --physdev-in vethb1  -j Cid7639X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-in vethb1  -j Cid7639X2034.0
    $IPTABLES -A Cid7639X2034.0  -s 192.168.50.13   -j RETURN
    $IPTABLES -A Cid7639X2034.0  -s 192.168.50.14   -j RETURN
    $IPTABLES -A Cid7639X2034.0  -s 192.168.50.15   -j RETURN
    $IPTABLES -N In_RULE_24_3
    $IPTABLES -A Cid7639X2034.0  -j In_RULE_24_3
    $IPTABLES -A In_RULE_24_3  -j LOG  --log-level info --log-prefix "RULE 24 -- DENY "
    $IPTABLES -A In_RULE_24_3  -j DROP
    # 
    # Rule 25 (vethb1)
    # 
    echo "Rule 25 (vethb1)"
    # 
    # br4 border IN
    $IPTABLES -N Cid7736X2034.0
    $IPTABLES -A INPUT -m physdev --physdev-in vethb1  -j Cid7736X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-in vethb1  -j Cid7736X2034.0
    $IPTABLES -A Cid7736X2034.0  -d 192.168.50.1   -j RETURN
    $IPTABLES -A Cid7736X2034.0  -d 192.168.0.37   -j RETURN
    $IPTABLES -A Cid7736X2034.0  -d 192.168.50.12   -j RETURN
    $IPTABLES -N In_RULE_25_3
    $IPTABLES -A Cid7736X2034.0  -j In_RULE_25_3
    $IPTABLES -A In_RULE_25_3  -j LOG  --log-level info --log-prefix "RULE 25 -- DENY "
    $IPTABLES -A In_RULE_25_3  -j DROP
    # 
    # Rule 26 (vmh1)
    # 
    echo "Rule 26 (vmh1)"
    # 
    # br4 border IN
    $IPTABLES -N Cid8881X2034.0
    $IPTABLES -A INPUT -m physdev --physdev-in vmh1  -j Cid8881X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-in vmh1  -j Cid8881X2034.0
    $IPTABLES -A Cid8881X2034.0  -s 192.168.50.1   -j RETURN
    $IPTABLES -A Cid8881X2034.0  -s 192.168.0.37   -j RETURN
    $IPTABLES -N In_RULE_26_3
    $IPTABLES -A Cid8881X2034.0  -j In_RULE_26_3
    $IPTABLES -A In_RULE_26_3  -j LOG  --log-level info --log-prefix "RULE 26 -- DENY "
    $IPTABLES -A In_RULE_26_3  -j DROP
    # 
    # Rule 27 (vmh1)
    # 
    echo "Rule 27 (vmh1)"
    # 
    # br4 border IN
    $IPTABLES -N Cid9010X2034.0
    $IPTABLES -A INPUT -m physdev --physdev-in vmh1  -j Cid9010X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-in vmh1  -j Cid9010X2034.0
    $IPTABLES -A Cid9010X2034.0  -d 192.168.50.12   -j RETURN
    $IPTABLES -A Cid9010X2034.0  -d 192.168.50.13   -j RETURN
    $IPTABLES -A Cid9010X2034.0  -d 192.168.50.14   -j RETURN
    $IPTABLES -A Cid9010X2034.0  -d 192.168.50.15   -j RETURN
    $IPTABLES -N In_RULE_27_3
    $IPTABLES -A Cid9010X2034.0  -j In_RULE_27_3
    $IPTABLES -A In_RULE_27_3  -j LOG  --log-level info --log-prefix "RULE 27 -- DENY "
    $IPTABLES -A In_RULE_27_3  -j DROP
    # 
    # Rule 29 (vmh2)
    # 
    echo "Rule 29 (vmh2)"
    # 
    # host OUT
    $IPTABLES -N Cid10297X2034.0
    $IPTABLES -A OUTPUT -o vmh2   -s 192.168.0.19   -j Cid10297X2034.0
    $IPTABLES -A OUTPUT -o vmh2   -s 192.168.50.1   -j Cid10297X2034.0
    $IPTABLES -A Cid10297X2034.0  -d 192.168.50.12   -j RETURN
    $IPTABLES -A Cid10297X2034.0  -d 192.168.50.13   -j RETURN
    $IPTABLES -A Cid10297X2034.0  -d 192.168.50.14   -j RETURN
    $IPTABLES -A Cid10297X2034.0  -d 192.168.50.15   -j RETURN
    $IPTABLES -N Out_RULE_29_3
    $IPTABLES -A Cid10297X2034.0  -j Out_RULE_29_3
    $IPTABLES -A Out_RULE_29_3  -j LOG  --log-level info --log-prefix "RULE 29 -- DENY "
    $IPTABLES -A Out_RULE_29_3  -j DROP
    # 
    # Rule 30 (vmh2)
    # 
    echo "Rule 30 (vmh2)"
    # 
    # host IN
    $IPTABLES -N Cid10437X2034.0
    $IPTABLES -A INPUT -i vmh2   -d 192.168.0.19   -j Cid10437X2034.0
    $IPTABLES -A INPUT -i vmh2   -d 192.168.50.1   -j Cid10437X2034.0
    $IPTABLES -A FORWARD -i vmh2   -d 192.168.0.37   -j Cid10437X2034.0
    $IPTABLES -A Cid10437X2034.0  -s 192.168.50.12   -j RETURN
    $IPTABLES -A Cid10437X2034.0  -s 192.168.50.13   -j RETURN
    $IPTABLES -A Cid10437X2034.0  -s 192.168.50.14   -j RETURN
    $IPTABLES -A Cid10437X2034.0  -s 192.168.50.15   -j RETURN
    $IPTABLES -N In_RULE_30_3
    $IPTABLES -A Cid10437X2034.0  -j In_RULE_30_3
    $IPTABLES -A In_RULE_30_3  -j LOG  --log-level info --log-prefix "RULE 30 -- DENY "
    $IPTABLES -A In_RULE_30_3  -j DROP
    # 
    # Rule 32 (vk63)
    # 
    echo "Rule 32 (vk63)"
    # 
    # br6 IN
    $IPTABLES -N In_RULE_32
    $IPTABLES -A INPUT -m physdev --physdev-in vk63 -p icmp  -m icmp  -s 192.168.50.13   --icmp-type any  -m state --state NEW  -j In_RULE_32
    $IPTABLES -A FORWARD -m physdev --physdev-in vk63 -p icmp  -m icmp  -s 192.168.50.13   --icmp-type any  -m state --state NEW  -j In_RULE_32
    $IPTABLES -A In_RULE_32  -j LOG  --log-level info --log-prefix "RULE 32 -- ACCEPT "
    $IPTABLES -A In_RULE_32  -j ACCEPT
    # 
    # Rule 33 (vk64)
    # 
    echo "Rule 33 (vk64)"
    # 
    # br6 IN
    $IPTABLES -N In_RULE_33
    $IPTABLES -A INPUT -m physdev --physdev-in vk64 -p icmp  -m icmp  -s 192.168.50.14   --icmp-type any  -m state --state NEW  -j In_RULE_33
    $IPTABLES -A FORWARD -m physdev --physdev-in vk64 -p icmp  -m icmp  -s 192.168.50.14   --icmp-type any  -m state --state NEW  -j In_RULE_33
    $IPTABLES -A In_RULE_33  -j LOG  --log-level info --log-prefix "RULE 33 -- ACCEPT "
    $IPTABLES -A In_RULE_33  -j ACCEPT
    # 
    # Rule 34 (vk64)
    # 
    echo "Rule 34 (vk64)"
    # 
    # br6 IN HTTP
    $IPTABLES -N In_RULE_34
    $IPTABLES -A FORWARD -m physdev --physdev-in vk64 -p tcp -m tcp  -m multiport  -s 192.168.50.14   -d 192.168.0.37   --dports 80,443  -m state --state NEW  -j In_RULE_34
    $IPTABLES -A In_RULE_34  -j LOG  --log-level info --log-prefix "RULE 34 -- ACCEPT "
    $IPTABLES -A In_RULE_34  -j ACCEPT
    # 
    # Rule 35 (vk65)
    # 
    echo "Rule 35 (vk65)"
    # 
    # br6 IN
    $IPTABLES -N In_RULE_35
    $IPTABLES -A INPUT -m physdev --physdev-in vk65 -p icmp  -m icmp  -s 192.168.50.15   --icmp-type any  -m state --state NEW  -j In_RULE_35
    $IPTABLES -A FORWARD -m physdev --physdev-in vk65 -p icmp  -m icmp  -s 192.168.50.15   --icmp-type any  -m state --state NEW  -j In_RULE_35
    $IPTABLES -A In_RULE_35  -j LOG  --log-level info --log-prefix "RULE 35 -- ACCEPT "
    $IPTABLES -A In_RULE_35  -j ACCEPT
    # 
    # Rule 36 (vethb2)
    # 
    echo "Rule 36 (vethb2)"
    # 
    # br6 border IN
    $IPTABLES -N Cid9698X2034.0
    $IPTABLES -A FORWARD -m physdev --physdev-in vethb2 -p icmp  -m icmp  -s 192.168.50.1   --icmp-type any  -m state --state NEW  -j Cid9698X2034.0
    $IPTABLES -N In_RULE_36
    $IPTABLES -A Cid9698X2034.0  -d 192.168.50.13   -j In_RULE_36
    $IPTABLES -A Cid9698X2034.0  -d 192.168.50.14   -j In_RULE_36
    $IPTABLES -A Cid9698X2034.0  -d 192.168.50.15   -j In_RULE_36
    $IPTABLES -N Cid9698X2034.1
    $IPTABLES -A FORWARD -m physdev --physdev-in vethb2 -p icmp  -m icmp  -s 192.168.50.1   --icmp-type any  -m state --state NEW  -j Cid9698X2034.1
    $IPTABLES -A Cid9698X2034.1  -d 192.168.50.13   -j In_RULE_36
    $IPTABLES -A Cid9698X2034.1  -d 192.168.50.14   -j In_RULE_36
    $IPTABLES -A Cid9698X2034.1  -d 192.168.50.15   -j In_RULE_36
    $IPTABLES -N Cid9698X2034.2
    $IPTABLES -A FORWARD -m physdev --physdev-in vethb2 -p icmp  -m icmp  --icmp-type any  -m state --state NEW  -j Cid9698X2034.2
    $IPTABLES -N Cid9698X2034.3
    $IPTABLES -A Cid9698X2034.2  -s 192.168.0.37   -j Cid9698X2034.3
    $IPTABLES -A Cid9698X2034.2  -s 192.168.50.12   -j Cid9698X2034.3
    $IPTABLES -A Cid9698X2034.3  -d 192.168.50.13   -j In_RULE_36
    $IPTABLES -A Cid9698X2034.3  -d 192.168.50.14   -j In_RULE_36
    $IPTABLES -A Cid9698X2034.3  -d 192.168.50.15   -j In_RULE_36
    $IPTABLES -A In_RULE_36  -j LOG  --log-level info --log-prefix "RULE 36 -- ACCEPT "
    $IPTABLES -A In_RULE_36  -j ACCEPT
    # 
    # Rule 38 (vk42)
    # 
    echo "Rule 38 (vk42)"
    # 
    # br4 IN
    $IPTABLES -N In_RULE_38
    $IPTABLES -A INPUT -m physdev --physdev-in vk42 -p icmp  -m icmp  -s 192.168.50.12   --icmp-type any  -m state --state NEW  -j In_RULE_38
    $IPTABLES -A FORWARD -m physdev --physdev-in vk42 -p icmp  -m icmp  -s 192.168.50.12   --icmp-type any  -m state --state NEW  -j In_RULE_38
    $IPTABLES -A In_RULE_38  -j LOG  --log-level info --log-prefix "RULE 38 -- ACCEPT "
    $IPTABLES -A In_RULE_38  -j ACCEPT
    # 
    # Rule 39 (vethb1)
    # 
    echo "Rule 39 (vethb1)"
    # 
    # br4 border IN
    $IPTABLES -N Cid15566X9203.0
    $IPTABLES -A FORWARD -m physdev --physdev-in vethb1 -p icmp  -m icmp  -d 192.168.50.1   --icmp-type any  -m state --state NEW  -j Cid15566X9203.0
    $IPTABLES -N In_RULE_39
    $IPTABLES -A Cid15566X9203.0  -s 192.168.50.13   -j In_RULE_39
    $IPTABLES -A Cid15566X9203.0  -s 192.168.50.14   -j In_RULE_39
    $IPTABLES -A Cid15566X9203.0  -s 192.168.50.15   -j In_RULE_39
    $IPTABLES -N Cid15566X9203.1
    $IPTABLES -A INPUT -m physdev --physdev-in vethb1 -p icmp  -m icmp  -d 192.168.50.1   --icmp-type any  -m state --state NEW  -j Cid15566X9203.1
    $IPTABLES -A Cid15566X9203.1  -s 192.168.50.13   -j In_RULE_39
    $IPTABLES -A Cid15566X9203.1  -s 192.168.50.14   -j In_RULE_39
    $IPTABLES -A Cid15566X9203.1  -s 192.168.50.15   -j In_RULE_39
    $IPTABLES -N Cid15566X9203.2
    $IPTABLES -A FORWARD -m physdev --physdev-in vethb1 -p icmp  -m icmp  --icmp-type any  -m state --state NEW  -j Cid15566X9203.2
    $IPTABLES -N Cid15566X9203.3
    $IPTABLES -A Cid15566X9203.2  -d 192.168.0.37   -j Cid15566X9203.3
    $IPTABLES -A Cid15566X9203.2  -d 192.168.50.12   -j Cid15566X9203.3
    $IPTABLES -A Cid15566X9203.3  -s 192.168.50.13   -j In_RULE_39
    $IPTABLES -A Cid15566X9203.3  -s 192.168.50.14   -j In_RULE_39
    $IPTABLES -A Cid15566X9203.3  -s 192.168.50.15   -j In_RULE_39
    $IPTABLES -A In_RULE_39  -j LOG  --log-level info --log-prefix "RULE 39 -- ACCEPT "
    $IPTABLES -A In_RULE_39  -j ACCEPT
    # 
    # Rule 40 (vmh1)
    # 
    echo "Rule 40 (vmh1)"
    # 
    # br4 border IN
    $IPTABLES -N Cid16232X9203.0
    $IPTABLES -A FORWARD -m physdev --physdev-in vmh1 -p icmp  -m icmp  -s 192.168.50.1   --icmp-type any  -m state --state NEW  -j Cid16232X9203.0
    $IPTABLES -N In_RULE_40
    $IPTABLES -A Cid16232X9203.0  -d 192.168.50.12   -j In_RULE_40
    $IPTABLES -A Cid16232X9203.0  -d 192.168.50.13   -j In_RULE_40
    $IPTABLES -A Cid16232X9203.0  -d 192.168.50.14   -j In_RULE_40
    $IPTABLES -A Cid16232X9203.0  -d 192.168.50.15   -j In_RULE_40
    $IPTABLES -N Cid16232X9203.1
    $IPTABLES -A FORWARD -m physdev --physdev-in vmh1 -p icmp  -m icmp  -s 192.168.50.1   --icmp-type any  -m state --state NEW  -j Cid16232X9203.1
    $IPTABLES -A Cid16232X9203.1  -d 192.168.50.12   -j In_RULE_40
    $IPTABLES -A Cid16232X9203.1  -d 192.168.50.13   -j In_RULE_40
    $IPTABLES -A Cid16232X9203.1  -d 192.168.50.14   -j In_RULE_40
    $IPTABLES -A Cid16232X9203.1  -d 192.168.50.15   -j In_RULE_40
    $IPTABLES -N Cid16232X9203.2
    $IPTABLES -A FORWARD -m physdev --physdev-in vmh1 -p icmp  -m icmp  -s 192.168.0.37   --icmp-type any  -m state --state NEW  -j Cid16232X9203.2
    $IPTABLES -A Cid16232X9203.2  -d 192.168.50.12   -j In_RULE_40
    $IPTABLES -A Cid16232X9203.2  -d 192.168.50.13   -j In_RULE_40
    $IPTABLES -A Cid16232X9203.2  -d 192.168.50.14   -j In_RULE_40
    $IPTABLES -A Cid16232X9203.2  -d 192.168.50.15   -j In_RULE_40
    $IPTABLES -A In_RULE_40  -j LOG  --log-level info --log-prefix "RULE 40 -- ACCEPT "
    $IPTABLES -A In_RULE_40  -j ACCEPT
    # 
    # Rule 42 (vmh2)
    # 
    echo "Rule 42 (vmh2)"
    # 
    # external are IN
    $IPTABLES -N Cid16691X6788.0
    $IPTABLES -A INPUT -i vmh2  -p icmp  -m icmp  -d 192.168.50.1   --icmp-type any  -m state --state NEW  -j Cid16691X6788.0
    $IPTABLES -N In_RULE_42
    $IPTABLES -A Cid16691X6788.0  -s 192.168.50.12   -j In_RULE_42
    $IPTABLES -A Cid16691X6788.0  -s 192.168.50.13   -j In_RULE_42
    $IPTABLES -A Cid16691X6788.0  -s 192.168.50.14   -j In_RULE_42
    $IPTABLES -A Cid16691X6788.0  -s 192.168.50.15   -j In_RULE_42
    $IPTABLES -N Cid16691X6788.1
    $IPTABLES -A FORWARD -i vmh2  -p icmp  -m icmp  -d 192.168.0.37   --icmp-type any  -m state --state NEW  -j Cid16691X6788.1
    $IPTABLES -A Cid16691X6788.1  -s 192.168.50.12   -j In_RULE_42
    $IPTABLES -A Cid16691X6788.1  -s 192.168.50.13   -j In_RULE_42
    $IPTABLES -A Cid16691X6788.1  -s 192.168.50.14   -j In_RULE_42
    $IPTABLES -A Cid16691X6788.1  -s 192.168.50.15   -j In_RULE_42
    $IPTABLES -A In_RULE_42  -j LOG  --log-level info --log-prefix "RULE 42 -- ACCEPT "
    $IPTABLES -A In_RULE_42  -j ACCEPT
    # 
    # Rule 43 (vmh2)
    # 
    echo "Rule 43 (vmh2)"
    # 
    # host border OUT
    $IPTABLES -N Cid16236X6788.0
    $IPTABLES -A OUTPUT -o vmh2  -p icmp  -m icmp  -s 192.168.50.1   --icmp-type any  -m state --state NEW  -j Cid16236X6788.0
    $IPTABLES -N Out_RULE_43
    $IPTABLES -A Cid16236X6788.0  -d 192.168.50.12   -j Out_RULE_43
    $IPTABLES -A Cid16236X6788.0  -d 192.168.50.13   -j Out_RULE_43
    $IPTABLES -A Cid16236X6788.0  -d 192.168.50.14   -j Out_RULE_43
    $IPTABLES -A Cid16236X6788.0  -d 192.168.50.15   -j Out_RULE_43
    $IPTABLES -N Cid16236X6788.1
    $IPTABLES -A FORWARD -o vmh2  -p icmp  -m icmp  -s 192.168.0.37   --icmp-type any  -m state --state NEW  -j Cid16236X6788.1
    $IPTABLES -A Cid16236X6788.1  -d 192.168.50.12   -j Out_RULE_43
    $IPTABLES -A Cid16236X6788.1  -d 192.168.50.13   -j Out_RULE_43
    $IPTABLES -A Cid16236X6788.1  -d 192.168.50.14   -j Out_RULE_43
    $IPTABLES -A Cid16236X6788.1  -d 192.168.50.15   -j Out_RULE_43
    $IPTABLES -A Out_RULE_43  -j LOG  --log-level info --log-prefix "RULE 43 -- ACCEPT "
    $IPTABLES -A Out_RULE_43  -j ACCEPT
    # 
    # Rule 45 (br0)
    # 
    echo "Rule 45 (br0)"
    # 
    # external
    $IPTABLES -A OUTPUT -o br0   -m state --state NEW  -j ACCEPT
    # 
    # Rule 46 (br0)
    # 
    echo "Rule 46 (br0)"
    # 
    # external
    $IPTABLES -A FORWARD -i br0   -s 192.168.0.10   -d 192.168.0.255   -m state --state NEW  -j ACCEPT
    $IPTABLES -A INPUT -i br0   -s 192.168.0.10   -d 192.168.0.255   -m state --state NEW  -j ACCEPT
    $IPTABLES -A INPUT -i br0   -s 192.168.0.10   -m state --state NEW  -j ACCEPT
    # 
    # Rule 47 (global)
    # 
    echo "Rule 47 (global)"
    # 
    $IPTABLES -N RULE_47
    $IPTABLES -A OUTPUT  -j RULE_47
    $IPTABLES -A INPUT  -j RULE_47
    $IPTABLES -A FORWARD  -j RULE_47
    $IPTABLES -A RULE_47  -j LOG  --log-level info --log-prefix "RULE 47 -- DENY "
    $IPTABLES -A RULE_47  -j DROP
}

 
The variable "$IPTABLES" identifies the local iptables command. As already discussed in the last articles we arranged our (virtual) guest systems, the virtualization host and external systems in 3 defined host groups in FWbuilder (see the last article):

  • br6_grp: kali3, kali4, kali5,
  • br4_grp: kali2,
  • ext_grp: the host and some external web server "lamp".

Remember that rules for bridge-ports are investigated separately and independently as a packet moves from one bridge to another. Note that the host and further systems attached to "virbr4" via a veth device "vmh2" are recognized as members of a distinct logical host area for which iptables rules again are reinvestigated separatly by the kernel during packet transport. Therefore we need ACCEPT rules to allow for incoming and outgoing packets at the host's interface "vmh2".

Examples of spoofing scenarios

With 2 bridges in place we can define already a variety of ARP spoofing scenarios with a subsequent MiM-attack. We only test some selected, but typical scenarios. Note again that we cannot prevent the act of spoofing itself with iptables - however, we can prevent that redirected packets arrive at the MiM system.

Example 1: kali2 of virbr4 attacks the communication between kali3 and kali5 within virbr6

Which rule do we expect to prevent this? Actually as kali2 tries to redirect the intended communication from bridge virbr6 into bridge virbr4 we would already expect a DENY rule at the border port "vethb2" to stop redirected packets. In our rules list this would be rule 16.

So let us see. We start ARP spoofing on kali2:

root@kali2: ~# echo 1 > /proc/sys/net/ipv4/ip_forward
root@kali2: ~# iptables -A OUTPUT -p icmp --icmp-type redirect -j REJECT
root@kali2: ~# arpspoof -i eth3 -t 192.168.50.13 192.168.50.14 & 2> /dev/null
root@kali2: ~# arpspoof -i eth3 -t 192.168.50.14 192.168.50.13 & 2> /dev/null

 
eth3 is the relevant Ethernet interface to net 192.168.50.0/24 on guest kali2.
After some time we get the following ARP information on e.g. kali3:

bridges_1

Consequently, after a "journalctl -f" on the virtualization host we find: :

Mar 17 13:21:53 mytux kernel: RULE 16 -- DENY IN=virbr6 OUT=virbr6 PHYSIN=vk63 PHYSOUT=vethb2 MAC=52:54:00:f2:a4:8d:52:54:00:b1:5d:1f:08:00 SRC=192.168.50.13 DST=192.168.50.14 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=16140 DF PROTO=ICMP TYPE=8 CODE=0 ID=1756 SEQ=1 
Mar 17 13:21:54 mytux kernel: RULE 16 -- DENY IN=virbr6 OUT=virbr6 PHYSIN=vk63 PHYSOUT=vethb2 MAC=52:54:00:f2:a4:8d:52:54:00:b1:5d:1f:08:00 SRC=192.168.50.13 DST=192.168.50.14 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=16252 DF PROTO=ICMP TYPE=8 CODE=0 ID=1756 SEQ=2

 
Our test example shows that rules for border ports help to isolate bridges against misguided packets.

Rule 16 deserves a closer look as it contains a logical negation of 2 separately defined groups of hosts. We see that FWbuilder compiles the negation internally correctly: The related subchain definition contains all required hosts.

As described in the previous articles we stop the attack by the command "killall arpspoof" on kali2. Remember that due to time limits for ARP and port caching information on the guests and the bridge, respectively, it may take some time until normal operation is possible again. See the first article of this series for more information on this topic.

Example 2: kali2 of virbr4 attacks the communication between kali3 and the virtualization host

In this scenario a regular (!) packet would propagate from virbr6 through virbr4 and then to the host. Therefore, neither border port rules for virbr6 nor for virbr4 can block the traffic. We must, instead, rely on the analysis of redirected packets following an OUT direction to port vk42 - this is rule 5.

Therefore, this example is just a repetition of what we learned in the first article of this series
Linux bridges – can iptables be used against MiM attacks based on ARP spoofing ? – I

Actually, after another spoofing attack by kali2

root@kali2: ~# arpspoof -i eth3 -t 192.168.50.1 192.168.50.13 & 2> /dev/null
root@kali2: ~# arpspoof -i eth3 -t 192.168.50.13 192.168.50.1 & 2> /dev/null

 
and sending pings from kali3 to the host we get:

Mar 17 18:44:51 mytux kernel: RULE 32 -- ACCEPT IN=virbr6 OUT=virbr6 PHYSIN=vk63 PHYSOUT=vethb2 MAC=54:00:f2:a4:8d:52:54:00:b1:5d:1f:08:00 SRC=192.168.50.13 DST=192.168.50.1 LEN=84 TOS=0x00 PREC=0x00 =64 ID=48428 DF PROTO=ICMP TYPE=8 CODE=0 ID=2872 SEQ=2 
Mar 17 18:44:51 mytux kernel: RULE 5 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vethb1 PHYSOUT=vk42 MAC=52:50:f2:a4:8d:52:54:00:b1:5d:1f:08:00 SRC=192.168.50.13 DST=192.168.50.1 LEN=84 TOS=0x00 PREC=0x00 TTL ID=48428 DF PROTO=ICMP TYPE=8 CODE=0 ID=2872 SEQ=2 

 
We see that the transition from bridge virbr6 to virbr4 works as planned - however the packets redirected to the MiM kali2 are stopped at vk42. Good!

Example 3: kali3 of virbr6 attacks the communication between kali4 of virbr4 and the virtualization host

We look at pings issued from the host to kali4 after an attack of kali3. In this case the border port rules again must not block. Instead, we rely on local port rules at port vk63, .i.e. rule 2. Indeed:

Mar 17 19:00:39 mytux kernel: RULE 43 -- ACCEPT IN= OUT=vmh2 SRC=192.168.50.1 DST=192.168.50.14 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53576 DF PROTO=ICMP TYPE=8 CODE=0 ID=2981 SEQ=1 
Mar 17 19:00:39 mytux kernel: RULE 40 -- ACCEPT IN=virbr4 OUT=virbr4 PHYSIN=vmh1 PHYSOUT=vethb1 MAC=52:54:00:b1:5d:1f:7a:ff:fc:bd:68:b6:08:00 SRC=192.168.50.1 DST=192.168.50.14 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53576 DF PROTO=ICMP TYPE=8 CODE=0 ID=2981 SEQ=1 
Mar 17 19:00:39 mytux kernel: RULE 2 -- DENY IN=virbr6 OUT=virbr6 PHYSIN=vethb2 PHYSOUT=vk63 MAC=52:54:00:b1:5d:1f:7a:ff:fc:bd:68:b6:08:00 SRC=192.168.50.1 DST=192.168.50.14 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53576 DF PROTO=ICMP TYPE=8 CODE=0 ID=2981 SEQ=1 

 

Example 4: kali3 of virbr6 attacks the communication between kali2 and the virtualization host

In this case border rules should stop redirected packets. For our test case this would in particular be rule 18.

And - after the initialization of the attack by kali3 and the trial to send pings from kali2 to the host, we get:

Mar 18 16:47:30 mytux kernel: RULE 18 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vk42 PHYSOUT=vethb1 MAC=52:54:00:b1:5d:1f:52:54:00:f2:a4:8d:08:00 SRC=192.168.50.12 DST=192.168.50.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=4405 DF PROTO=ICMP TYPE=8 CODE=0 ID=2420 SEQ=1 

 
And vice versa :

Mar 18 16:48:42 mytux kernel: RULE 43 -- ACCEPT IN= OUT=vmh2 SRC=192.168.50.1 DST=192.168.50.12 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=63251 DF PROTO=ICMP TYPE=8 CODE=0 ID=21778 SEQ=1 
Mar 18 16:48:42 mytux kernel: RULE 18 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vmh1 PHYSOUT=vethb1 MAC=52:54:00:b1:5d:1f:f2:be:a1:5a:cd:6e:08:00 SRC=192.168.50.1 DST=192.168.50.12 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=63251 DF PROTO=ICMP TYPE=8 CODE=0 ID=21778 SEQ=1 

 
As expected!

Example 5: The host attacks communication between guests attached to an inner bridge

One may think such an example is just academic. Actually, in my opinion it is not. Although the administrator of a virtualization host has in principle a variety of means available to follow any communication across a bridge, ARP spoofing should NOT be such a measure under normal operation conditions. In addition, there may be legal aspects in a professional hosting situation.

But more important: From the perspective of the involved bridges, in our setup the host is attached to bridge virbr4 as an external guest over a border port. Rules for the virtualization host are, therefore, only an example for similar rules applied to other external hosts which may have the allowance to communicate with bridge guests - via forwarding and a respective route defined on the virtualization host.

We expect rule 20 to stop packages redirected by the MiM:

         
Mar 18 17:38:38 rux kernel: RULE 32 -- ACCEPT IN=virbr6 OUT=virbr6 PHYSIN=vk63 PHYSOUT=vethb2 MAC=f2:be:a1:5a:cd:6e:52:54:00:b1:5d:1f:08:00 SRC=192.168.50.13 DST=192.168.50.12 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=36173 DF PROTO=ICMP TYPE=8 CODE=0 ID=2218 SEQ=1 
Mar 18 17:38:38 rux kernel: RULE 20 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vethb1 PHYSOUT=vmh1 MAC=f2:be:a1:5a:cd:6e:52:54:00:b1:5d:1f:08:00 SRC=192.168.50.13 DST=192.168.50.12 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=36173 DF PROTO=ICMP TYPE=8 CODE=0 ID=2218 SEQ=1 

 
And vice versa

       
Mar 18 17:39:39 rux kernel: RULE 20 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vk42 PHYSOUT=vmh1 MAC=f2:be:a1:5a:cd:6e:52:54:00:f2:a4:8d:08:00 SRC=192.168.50.12 DST=192.168.50.13 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=10910 DF PROTO=ICMP TYPE=8 CODE=0 ID=2730 SEQ=1 
Mar 18 17:39:40 rux kernel: RULE 20 -- DENY IN=virbr4 OUT=virbr4 PHYSIN=vk42 PHYSOUT=vmh1 MAC=f2:be:a1:5a:cd:6e:52:54:00:f2:a4:8d:08:00 SRC=192.168.50.12 DST=192.168.50.13 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=11113 DF PROTO=ICMP TYPE=8 CODE=0 ID=2730 SEQ=2 

 

Summary

So, all in all, for our few examples we could verify that our recipe for setting up iptables rules in case of several linked Linux bridges with guests on one [KVM] virtualization host guided us correctly. After associating unique IP addresses with bridge ports we can define rules that block the transport of packets redirected to a MiM system - even when multiple bridges are present on the virtualization host. Additional and special rules for the bridges' border ports help to prevent irregular traffic between defined groups of guests and/or external hosts.

Note that we only demonstrated this for specific allowance rules for the ICMP protocol. Yet, it is easy to understand that the very same principles should work for any protocol on level 4.

In the next article of this series we will look at an example with an external host and eventually discuss what our test results mean for administrators of virtualization hosts.