Some simple questions after the Bundeswehr Leak ….

I am following the present discussion about the leak of a communication between some generals of the German Air Defense from abroad. It really feels extremely embarrassing that the Russians could overhear a communication of high-ranked German generals. And I do not know how to answer questions of retired Norwegian IT-specialists (of my age) here in the north of Europe … With German troops participating right now in huge Nato maneuvers in northern Norway. And believe me I have often criticized my Norwegian friends for a lack of IT-security awareness, in particular regarding the use of Microsoft and Zoom. And now this disaster in my home country …

From my point of view establishing a formal Untersuchungsausschuss (investigation committee) of the German parliament is only one of the necessary steps required to clarify what has happened. But it is an important one, together with internal and technical investigations within the Bundeswehr.

In this post I want to ask some simple questions which, in my opinion, should be answered. Any IT-guy having worked with or for public governmental institutions in Europe could ask them. None of the links I give below is classified.

Why WebEx at all?

The first point in all the TV and newspaper discussions is that there seems to be no doubt that the communication is authentic and that WebEx was used. Now, anyone with a basic idea about information security would ask the following question:

Why for heavens sake did the German generals use Webex at all? A product of Cisco, i.e. of a commercial US company? With published security problems during the Corona years? Why does the German military not have its own security protocols and measures in place?

Well, we and the US are allies, but … Yeah, but, … and Trump on the horizon.

It is a German company, “Secunet”, who has developed mobile systems (SINA workstations) compliant with for NATO certifications on different levels of security. See e.g.:
https://www.secunet.com/ loesungen/ sina-workstation-s
https://www.ia.nato.int/ niapc/ Product/ SINA-Workstation-S_730
https://www.bundeswehr-journal.de/2020/mehr-als-6000-geraete-sina-workstation-s-fuer-die-bundeswehr/
https://www.secunet.com/ueber-uns/presse/artikel/sicheres-mobiles-arbeiten-bwi-beschafft-sina-workstations-s-fuer-die-bundeswehr
https://www.secunet.com/ueber-uns/presse/artikel/secunet-beliefert-die-bundeswehr-mit-sina-sicherheitstechnologie-fuer-schnelle-eingreiftruppe-der-nato

These systems were bought by the Bundeswehr in relatively large quantities.

So, my first two open question are:

  • Why was WebEx used at all? Did the generals not have SINA-workstations available? If they had not why not?
  • Under what conditions is the usage of WebEx regarded secure by the the responsible IT-specialists of the Bundeswehr?

Actually, the answer to the second question may not be independent of the answer to the first one …

Addendum, 05.05.2024: Another point which comes up during TV discussions is that some “experts” say there is reason to believe that the WebEx-session was set up without any end-to-end-encryption. This is really implausible. What is more plausible is that the session was set up with standard encryption and Cisco’s management of encryption keys. And maybe – due to mistakes – with allowance for some participants to use a telephone. Making the session insecure again …

A success message of the BWI of 2021 regarding the rollout of Cisco’s WebEx in the Bundeswehr

What did we read in 2021 as a success message regarding the digitalization of the German defense and public institutions?

” … Zusammen und in Echtzeit an Dokumenten arbeiten, E-Mails austauschen, gemeinsame Termine planen, Webkonferenzen mit und ohne Video abhalten, chatten und telefonieren – wenn alle Services komplett ausgerollt sind, ermöglicht Groupware Bw dies auf einer einheitlichen technischen Plattform und über alle zivilen und militärischen Bereiche hinweg. Die BWI startete am 1. Oktober den Rollout mit den Produkten Cisco Jabber und Webex. In einem zweiten Schritt werden im kommenden Jahr auch die Services E-Mail und Enterprise Content Management mit Microsoft Outlook und SharePoint ausgerollt.”

See: https://www.bwi.de/ magazin /artikel/ groupware-bw-bei-der-bundeswehr-perfekt-vernetzt-von-chat-bis-videokonferenz

Even without translation the combination of tools praised here would make a security aware person more than nervous. And one can only hope that the introduction of these systems only concerns the civil sector of the Bundeswehr. If not, well, … just ask the CCC what they would think of it.

We non-hackers could ask: Why does the BWI regard notoriously insecure tools to be safe within a BWI-based environment? For SINA an outsider can understand this in parts because any communication would (hopefully always and enforced by the system) occur VPN-based (with a special protocol). But for the BWI-environment I did not find a quick answer …

Addendum, 05.03.2024: What makes me personally nervous is that a standard approach to security these days is to assume that the enemy already sits somewhere inside your network. Then a lot of effort has to be put in the prevention of the so called “lateral move” of an attacker inside your network. Using Outlook and a Sharepoint infrastructure appears to be a very questionable, if not counterproductive decision in this respect. In general, encapsulation of an insecure infrastructure in some networks or system layers with perimeter protection, intrusion detection and communication encryption could be regarded as insufficient for top security requirements.

But let us be optimistic and assume that the BWI developed some very special secure variants of the named products within some very special and controlled environments with all communication encapsulated in layers of especially hardened encryption, and, and…
And that their solution has been tested extensively, also concerning potential internal attacks. But let us go back to WebEx.

Restrictions regarding the use of video conferences at the university of the Bundeswehr

The following publicly accessible web-page makes it clear what the Bundeswehr university thinks about the use of standard (encrypted) WebEx and BWI-based WebEx. See:
https://publicwiki.unibw.de/ display/ RZ/ Videokonferenzen#

The first thing which becomes clear is that normal use of WebEx has been forbidden since 26.01.2023. One would assume that the rules which a student at the Bundeswehr university must follow is obligatory on the level of generals, too.

Then the table on the named web-page makes a fine distinction between WebEx and WebEx [BWI], accompanied by a comment concerning WebEx BWI: “Freigabe BMVg für “VS – NfD. Nur mit SINA-Gerät oder lokaler BWI-Ausstattung möglich.”

For my English-speaking readers: “VS NfD” means something like “confidential information – to be used only for professional purposes within official institutions”.

So, the usage of a WebEx-variant on a BWI- or SINA-equipped device actually is allowed for members of the BMVg (German ministry of defense) to exchange confidential information. But the use of standard WebEx (and this probably includes WebEx with end-to-end-encryption with Cisco keys) is forbidden.

These are, in my opinion, very clear and precise rules. In contrast to some speculations in German newspapers about unclear orders and rules within the military …

If we take the rules published by the Bundeswehr university seriously then we would think that German generals use WebEx (in a special variant?) on BWI-systems or SINA-systems, only.

So the next two questions would be:

  • Did our generals use WebEx within the limits of BWI- or SINA-based equipment?
  • Is it completely impossible to start a non-encrypted or only standard encrypted WebEx-session from such an equipment?

If both questions were answered by a Yes, then the Bundeswehr, the BWI and also Secunet may have a much bigger problem than presently discussed in the German and international press.

Standard WebEx from a standard device?

The other possibility is that some of the generals used WebEx from a standard device (like a personal Windows laptops) or even from a phone via a standard telephone line.

Maybe one of the generals may just have had a problem with establishing a SINA based VPN connection. Which may have been blocked by some intelligent web-firewalls in his surroundings. So, maybe, due to time pressure a simple access to the conference was used? I was witness to this kind of “quick solution” behavior quite often in business contexts. I hope that generals are more security aware, but …

If normal windows clients were used, this was a major security risk by itself. I have regularly found that even professional IT-people just comment the following hint by a shrug of their shoulders:: End-to-end-encryption does not help you if your (Windows-) device (mobile, laptop, PC) is already compromised. Which, by the way, is relatively easy to achieve … The standard answer you get is: “But the communication is encrypted nevertheless …” Well, needless to comment such stupidness …

Windows 10 was rightfully considered not to be a reliable base for inter-governmental communication by German state institutions. Intermediate solutions for a high security environments were based on SINA-systems from Secunet for very good reasons. So we are back to question 2 again …

  • Did our generals use other equipment than the required ones according to the mentioned rules which even students must follow? Or did they use standard laptops?

I hope the answers will turn out to be NO.

Weak points of standard WebEx with end-to-end-encryption

I myself had to use WebEx multiple times in a professional context. On a Windows system which I was forced to use by the involved partners. One thing that always made me nervous was that WebEx never worked correctly with Firefox. Especially not when FF’s security measures were moved into the direction of strict.

Also error messages came up during sessions when you watched the traffic via FF’s own developer tools. But it worked seemingly flawless with Chrome. Such a kind of browser dependency makes one nervous because it is a sign that something has not be programmed as it should be. And this in turn is a sign for a weakness which can potentially be exploited. And there were reports on multiple security problems with WebEx during the Covid pandemic.

Another weak point is the transmission of login data for a scheduled WebEx session to the participants. I often got these data via email and often without end-to-end encryption regarding the email contents. This, of course, renders all other security measures at least in parts useless …

A very big problem is that often some participants log in to a standard encrypted WebEx-session via normal telephone devices or the standard telephone functionality of their mobiles. Such sessions can, of course, not be regarded as safe, because the encryption mechanism does not work across standard telephone lines. So, a standard telephone connection opens up a big open hole for security in the whole arrangement. If somebody listens on the telephone line somewhere the security of the whole session is compromised. I once dared to ask an administrator about it. Answer: Yeah, we know, but there are sometimes priorities regarding mobility of our managers …

Another point is that it depends on your local control of the WebEx interface settings whether you see really all participants of your session all the time. Also the information about end-to-end encryption is limited when you use a standard end-to-end encrypted WebEx session, which uses the Key Management System [KMS] of Cisco.

Something most people are not aware of is that with standard encrypted WebEx-sessions Cisco owns the encryption keys for all phases of the communication (initiation with asymmetric encryption, key exchange, symmetric encryption …). And most of the convenience features of standard encrypted WebEx require this knowledge of the encryption keys on Ciscos’ servers. Cisco makes no secret of this. So, there is no privacy of the communication during a standard end-to-end-encrypted WebEx session regarding Cisco servers. And these servers may pose a security risk by themselves – even if we assume that Cisco does not cooperate with US governmental institutions. This, probably, is the reason why the use of standard encrypted WebEx sessions is forbidden at the Bundeswehr university.

All in all, I have very often experienced a combination of Windows (hacker’ preferred dreamland), Chrome and standard WebEx encryption with some participants on a telephone line – either due to their mobility situation or due to browser problems. Not a combination I, personally, would choose if I wanted to keep a conversation really secret. But as an employee or freelancer one does as one is told to do.

Anyway, here are some more questions to ask:

  • Was a telephone access used by one of the participants of the WebEx session?
  • Was the WebEx session encrypted on the standard level, only? I.e., by using Cisco’s own key management?
  • Was the WebEx session scheduled and were respective access data distributed by (unencrypted) email?
  • Was the list of participants checked all the time during the session? Can it be excluded that someone took part in the conversation who was not invited?

Trivial security problems

Another danger which is independent of WebEx is the following: If the room of one of the communication partners was compromised and laptop speakers were used instead of headsets, no hacking was required to get a copy of all the communication. Standard recording devices would have been enough … Now, we would of course expect that a high-rank general is well equipped to check his private and hotel rooms for bugs and wiretaps. But what do we normal citizens know?

Another problem are headsets using Bluetooth. They can be hacked and electronic, wireless eavesdropping is possible with relatively cheap equipment from a neighboring room.

Further questions, therefore, are:

  • Were the rooms of the participants checked for bugs?
  • Where dammed and cable-based headsets used?

Two encryption levels for Cisco WebEx

WebEx offers two levels of end-to-end encryption handling. The only one that should be used for real confidentiality is “Zero-Trust End-to-End Encryption” based on MLS (see https://www.rfc-editor.org/rfc/rfc9420.html).
Telefon based communication is not allowed in such sessions for good and elementary reasons. And most of the standard convenience tools of WebEx can, of course, not be used.

So, we come to the next question:

  • Was the WebEx session set up as a zero-trust MLS session?

By the way, there is an additional question: What happened with the efforts to use “Matrix” as a supersystem for communication? And why was the Matrix-based BMWI messenger not used by our generals?

https://t3n.de/news/matrix-neuer-messenger-bundeswehr-behoerden-1282086/
https://www.bwi.de/magazin/artikel/bwi-veroeffentlicht-erstes-release-des-bundesmessengers

Conclusion

The leak of the communication of German generals by the Russians should make German citizens really nervous. We older ones saw all the deficits with respect to digitalization of German governmental institutions pile up during the decades both of the Kohl, the Schröder and the Merkel governments, i.e. throughout the last 40 years. In addition came the deficits with respect to education, active research und usage of AI in Germany during the last 15 years. And a notorious neglect of IT-security issues throughout the whole society – including in particular business companies. Now, the consequences of this total failure of leading politicians, which were and are responsible for the named fields, obviously have an impact on our German and European security. Our politicians have to wake up.

A thorough investigation of the leak is a minimum we concerned citizens expect. And consequences on the technical side – not only regarding our military.

And, my dear democratic parties: Keep the forefront organization for Russia’s disinformation, namely the AfD, out of this investigation, if possible.

 

More fun with veth and Linux network namespace – IV – L2-segments, same IP-subnet, ARP and routes

In the course of my present series on veths, network namespaces and virtual VLANs

More fun with veth and Linux network namespace – III – L2-segments of the same IP-subnet and routes in coupling network namespaces

More fun with veth and Linux network namespace – II – two L2-segments attached to a common network namespace

More fun with veth and Linux network namespace – I – open questions

we have started to study a somewhat academic network configuration:

We have attached two L2-segments to a common network namespace, with all IPs of both segments belonging to one and the same IP-subnet class.

The IP-setup makes the scenario a bit peculiar. It is not a situation an admin would normally create. Instead the IPs of each segment would be members of one of two distinct and different IP-subnets.

However, our scenario already taught us some important lessons to keep in mind when we approach our eventual objective:
Sooner or later we want to answer the question how to configure virtual VLAN configurations, in which veth-subdevices for tagged VLAN lines terminate in a common and routing namespace.

By our academic scenario we found out that we need to set up much more specific routes than the standard ones which are automatically created in the wake of “ip addr add” commands. Despite the fact that forwarding was disabled in the coupling network namespace! Only with clear and fitting routes we got a reasonable behavior of our artificial network with respect to ICMP requests in the last post (III).

Well, this is, in a way, a platitude. Whenever you have a special network you must adapt the routes to your network layout. This, of course, includes the resolution of ambiguities which our scenario introduced. The automatically defined routes caused an asymmetry in the ICMP replies in the otherwise very symmetric configuration (see the drawing below).

Nevertheless, the last post and also older posts on veths and virtual VLANs triggered an intense discussion of two of my readers with me concerning ARP. In this post I, therefore, want to look at the impact of routes on ARP requests and replies between the namespaces in more detail.

The key question is whether and to what extend the creation and emission of ARP-packets via a particular network interface may become route-dependent in namespaces or on hosts with multiple network interfaces.

While we saw a clear and route dependent asymmetry in the reaction of the network to ICMP-requests, we did not fully analyze how this asymmetry showed up on the ARP-level.

In my opinion we have already seen that ARP-requests triggered within ICMP-requests showed some asymmetry regarding the NIC used and the segment addressed – depending on the defined routes. But the experimental data of the last post did not show the flow of ARP-replies in detail. And we only regarded ARP-requests triggered by ICMP-requests; i.e. ARP-requests created by the Linux-kernel to support the execution of ICMP-requests.

Therefore, it is still unclear

  • whether routes do have an impact on ARP-replies;
  • whether routes have an impact on pure standalone ARP-requests, i.e. ARP-requests not caused by ICMP-requests (or by other requests of higher layer protocols).

The experiments below will deliver detailed information to clarify these points.

When I speak of ARP-tables below, I am referring to the ARP-caches of the various network namespaces in our scenario.

Continue reading

More fun with veth and Linux network namespace – III – L2-segments of the same IP-subnet and routes in coupling network namespaces

Linux network namespaces allow for experiments in virtual networks without setting up (virtual) hosts. This includes the study of packet propagation through virtual devices as bridges/switches and between network namespaces. With or without packet filters or firewall rules in place. In the last precedent post of this series

More fun with veth and Linux network namespace – II – two L2-segments attached to a common network namespace

I have posed a scenario with two L2-segments connected to a common network namespace. Such a scenario appears to be nothing special; we all are used to situations that require routing and forwarding because LAN-segments belong to different IP-subnets.

However, in my scenario all IPs are members of one and the same IP-subnet. An other interesting point is that the assignment of IPs to (virtual) NICs and bridges by the command “ip addr add” causes an automatic setup of elementary routes. We will see that such routes are insufficient to resolve the ambiguities of packet transport from the segment-coupling namespace to the attached segments.

To keep the conditions as plain and simple as possible in our scenario we do not mix in VLANs or packet filters or firewalls.

In the previous post we have already come to some conclusions about the posed scenario and its elements. However, these conclusions were based on theoretical considerations. In this post and an additional one we will verify or falsify these conclusions by performing concrete experiments on a Linux host.

The host I used for the experiments below was a laptop with an Opensuse Leap 15.5 OS, based on packets of SuSE’s enterprise server SLES. The namespaces are set up as unnamed network namespaces. This requires some special commands.

The detailed analysis of the scenario will help us to better understand more complex configurations including virtual VLANs in forthcoming posts.

If you notice varying MACs in some images below: You may ignore this safely. It is due to repeating the setup in the course of the experiments.

The scenario

The scenario is visualized by the following drawing:

How to setup the network namespaces and required devices?

I have discussed all commands required to configure such scenarios with unnamed Linux network namespaces in the first post of another older post series. See

Fun with veth-devices, Linux bridges and VLANs in unnamed Linux network namespaces – I

If you are new to experiments with unnamed Linux network namespaces please read this post first. Of particular importance are the commands

  • unshare …
  • nsenter …
  • ip link …
  • ip address …

I will not discuss details of the chain of commands required for setting up the elements of our scenario. At the end of this post you will find three PDFs with contents you can copy to bash shell scripts. All commands in their given form are meant to be executed by the user root in a safe test environment. I am very confident that readers interested in this post will understand the structure of the commands without further explanation.

Due to the fact that the scripts set and export intermediate variables as environment variables for sub-shells you should run the scripts via the “source“-command if you want these variables to be available in your original shell (probably in a terminal window), too.

The first script (create_netns) will then execute the following commands (you may need to scroll to see all):

Continue reading