Tag Archives: server certificate

KVM/Qemu VMs with a multi-screen Spice console – VI – remote access with remote-viewer and TLS encryption

Posted on by

In my series on various methods to access a Spice console of a VM I have already covered two remote scenarios in an Intranet based on remote-viewer and SSH:

  • Scenario 1: remote-viewer is run on the KVM/Qemu server and accesses the Qemu-hypervisor over a Unix socket. The user at the remote client-system opens a “ssh -X” or “ssh -XC” session to the server, starts remote-viewer there and uses the graphical output data via the client’s X-server. Audio requires a reverse SSH tunnel for Pulseaudio ( “ssh -X -R 44713:localhost:4713” ).
  • Scenario 2: remote-viewer is started on the remote client-system. A Spice and VM-specific TCP-port (TCP socket) on the server is used for the transfer of video + audio data. Security can be achieved by establishing a SSH tunnel with port-forwarding and further user-related SSH-restrictions.

See
KVM/Qemu VMs with a multi-screen Spice console – V – remote access via remote-viewer, a network port and a SSH-tunnel
KVM/Qemu VMs with a multi-screen Spice console – IV – remote access via SSH, remote-viewer and a Unix socket
KVM/Qemu VMs with a multi-screen Spice console – III – local access with remote-viewer via a Unix socket
KVM/Qemu VMs with a multi-screen Spice console – II – local access with remote-viewer via a network port
KVM/Qemu VMs with a multi-screen Spice console – I – Overview over local and remote access methods

Security regarding an encrypted data transfer, user authentication and port or socket access was achieved via SSH in both scenarios, plus ACLs in case of the first scenario. A critical point in both scenarios was data compression. SSH compression (=gzip) had a palpable negative impact on the responsiveness of the VM’s desktop in the Spice windows. However, data compression offered via Spice options did not diminish the performance – at least it could not be felt.

In this article we have a look at yet another scenario for remote-viewer: Spice is this time combined with TLS encryption. However, in this post, we use TLS for data encryption, only, and not yet for client-authentication. Client authentication methods in combination with TLS and remote-viewer will be the topic of the next article.

We use the same systems as in the last articles: A KVM/Qemu server host “MySRV” with a Leap 15.2 OS on it, a test-VM “debianx” with a Kali-OS on it and a client-system “MyLAP” (a laptop with a Leap 15.2 OS). On the KVM/Qemu host our meanwhile familiar user “uvma” is used to start the VM “debianx” for us with the help of virt-manager. On the client-system “MyLAP”, instead, a user “myself” will start remote-viewer.

Schematic drawing

The following sketch shows what we want to achieve:

The
Qemu-hypervisor shall use a TLS server-key and a X509-server-certificate to encrypt all application data transferred in our Intranet between the server and the remote-viewer application on the client-system. This should include all data channels of the Spice protocol.

TLS CA, server certificates and RSA-keys

A sound TLS setup requires at least a CA, a CA-certificate, a server-certificate for the KVM/Qemu-server and a file with the private key of an asymmetric (RSA) key-pair. This brings us to the question: What tools can we use on a Leap-system to create such certificates for our private network?

On a Linux system there are, of course the OpenSSL libraries together with the so called “certtool“, a CLI-tool. You get the latter on Leap 15.2 by installing the “gnutls” RPM from the Leap15.2-Update repository. A documentation for the creation of CA, server and client TLS certificates and related key-pairs with certtool is given on the following web-pages:
https://libvirt.org/tlscerts.html
https://qemu-project.gitlab.io/qemu/system/tls.html.

Personally, I prefer a graphical tool to keep an overview about my own CAs and related server-certificates. As Opensuse never replaced their YaST CA-tool after they changed to Ruby as the programming platform for YaST we have to look elsewhere.

A very good tool which provides a lot of options is “XCA“. You find it in the package “xca” on a Leap 15.2 system. It is intuitive to use, offers a lot of options and its “help” documentation is very good – if you already know something about the differences and requirements of certificates. It offers suitable templates for CA-, server- and client-certificates. For a step-by-step description of how to create certificates see section 14 in the help functionality. Other tutorials for certificate creation with “xca” can be found in th following PDFs
http://help.mguard.com/ pdf /en / mguard8/ AppNotes/ AH EN X.509 CERT XCA 108396 en 00
http://evardsson.github.io/ s3c3/ Generating, signing and exporting keys and certificates with XCA

Another older tool, which I still use, is TinyCA. The following image shows a test example (I have no such net nor server as displayed).

Its templates are simpler than those of “xca”; the options are also a bit more limited, but sufficient for private purposes. Opensuse provides a RPM “tinyca2” in its standard repositories. I have written some blog posts about it; see: TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – I and two later posts.
Please, be aware of the fact that you need to apply additional patches to get SHA-256 and SHA-512 capabilities. See the named post about it. You should be able with my descriptions to create a server certificate for your KVM/Qemu-host. Also see TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – II for a description into which directory the CA-certificate files should be placed on a Leap 15.x OS.

For the rest of the present article I assume
the following:
You have created a CA, a CA-certificate, a X509-compatible server-certificate and a RSA based private key for the server. You do this on whatever system you use for the administration of your CA. You have also exported the certificate data and the key into files with the “pem“-format and copied them (scp) to a save place under the control of root on your KVM/Qemu server host. In the end you should have the following files there:

  • ca-cert.pem
  • server-cert.pem
  • server-key.pem

Note that these specific file-names are important for the later Qemu-configuration. You should rename your files accordingly or make copies with these names. Note that the server-key file contains a private key – this file must be protected against unauthorized access during all steps of the configuration process and after.

Note also that the server-certificate should be set up with the FQDN of our KVM/Qemu-server: This is “mysrv.anraconc.de” for our test situation.

You must also place a copy of the CA-cert file onto the client-system(s) from which the Spice user later connects to the KVM-server. The “CA-cert”-file must in addition be known there as coming from a trustworthy CA. See a separate section below for the required measures on an Opensuse Leap system.

If you already had a CA and had already issued certificates in the past – can we reuse them? The answer is: Yes, you could. Personally, however, I prefer to issue a special dedicated server-certificate and keys for Qemu. One of the reasons is that the Qemu process must be able to read the server-key-file and it is run for a special user “qemu” and not root. But I do not want “qemu” to be able to read other server-keys used by completely other processes as e.g. for a web- or mysql-server.

Configuration of Qemu on the KVM-host for TLS connections

You remember from previous articles of this series that remote-viewer talks directly to the Qemu-hypervisor; it does not involve any libvirt layer. It is therefore no surprise that we need to configure Qemu itself such that it uses TLS. But which is the right configuration file to take? And where do we place our certificates and keys?

On a Leap 15.2 system we normally use the libvirt machinery to create and start Qemu-based VMs and the related Qemu-processes. Then we need to tell libvirt how to start a Qemu-process with the required correct options. The config-file to take in this case is “/etc/libvirt/qemu.conf” (see the 2nd article of this series).

[If you, by the way, are interested in the qemu-options and in particular the TLS options which can be used if and when you start a qemu-process manually see the qemu documentation, e.g. here and here.]

The named “qemu.conf”-file on our Leap based KVM/Qemu-server has multiple sections regarding TLS. The first main section covers the directories used for certificate and key-files. There you also find the file-names mentioned above. Later on you find an option to change the directory for Spice related certificates. We use this option.

For the time being we set the following parameters and un-comment the related lines:

# The following tells Qemu to use TLS for the encr<yption of Spice cahnnels 
spice_tls = 1  

# We need to specify a directory where we place the certifactes and key to be used  
spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice"

That is all we need for this article. (As mentioned in a previous post: On systems with apparmor active one should also activate “security_default_confined = 1”. But this no special TLS option).

Afterwards we have to copy our certificate- and key-files to “/etc/pki/libvirt-spice/“.

Note
again: It is obligatory to use the filenames given above. A VM would not start otherwise and Qemu would complain non-existing or unusable files.

What file access rights are required?
The certificates should only include public keys, so here we could grant others the “r”-right. The situation is different with the file “server-key.pem“. It contains the server’s private key – and probably in unencrypted form if we did not protect it by password.

Therefore, we need to restrict the read rights for this file. And here we are confronted with a small glitch in the Opensuse configuration – the apparmor settings allow the privileged libvirt-user who is allowed to start virsh or virt-manager to read the files – but not “qemu”. A simple solution is

mysrv:~ # cd /etc/pki/libvirt-spice/
mysrv:/etc/pki/libvirt-spice # chmod 440 *
mysrv:/etc/pki/libvirt-spice # chown root.qemu * 
mysrv:/etc/pki/libvirt-spice # la
total 20
drwxr-xr-x 2 root root 4096 Apr 11 11:44 .
drwxr-xr-x 8 root root 4096 Feb 27 15:11 ..
-r--r----- 1 root qemu 2504 Feb 27 15:14 ca-cert.pem
-r--r----- 1 root qemu 2504 Apr 11 09:49 server-cert.pem
-r--r----- 1 root qemu 3243 Apr 11 09:49 server-key.pem
mysrv:/etc/pki/libvirt-spice #

But from now on you should be careful during your experiments and check what members the group “qemu” has – on a Leap system it should only contain the user “qemu”, nobody else!

The question remains whether you need the CA-cert-file for any other purposes on the KVM server. If so, please follow the advice given in the section for using the ca-cert-file on the client-system and apply them in an analogous way on the server.

You should restart the libvirtd-daemon to activate the changed options for the start of Qemu-based VMs via virsh or virt-manager in the future.

Configuring Qemu to use TLS is NOT the same as configuring libvirt to use TLS!

Just a warning:
Readers who work totally libvirt-centered and use virt-viewer instead remote-viewer should at this point of reading become very clear about the fact that configuring Qemu for using TLS with Spice is something else than configuring libvirtd to use TLS for external connections. The Opensuse documentation at
doc.opensuse.org/ documentation/ leap/ virtualization/ html/ book-virt/ cha-libvirt-connect.html
refers to the latter. The settings to activate TLS for libvirtd and libvirt-tools use a different directory scheme and different file-names in comparison to what we did above. Also the required file access to certificates and private keys can be limited to root for pure libvitr-based tools – but not in our scenario.

Configuring the VM to use TLS with Spice

So far Qemu is prepared to support TLS on the KVM-server – if and when the use of TLS is requested. We, therefore, still have to define that TLS should be used for connections to the Spice console of our specific test-VM “debianx”.

Actually, we have to define a special network port for this purpose. In the XML-configuration file for the VM we change the Spice settings :

    
    <graphics type='spice' port='20001' tlsPort='20002' autoport='no' keymap='de' defaultMode='any' >
      <listen type='address' address='0.0.0.0'/>
      <image compression='auto_glz'/>
      <gl enable='no'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='virtio' heads='2' primary='yes'>
        <acceleration accel3d='yes'/>
      </model>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'
/>
    </video>

The first change in comparison to previous settings consists of a new attribute “tlsPort”
(Off topic: I use the same audio and video settings as before.)

Note:

The “tlsPort” – just as the the standard “port” – is specific for the VM.
For another VM you must define another “tlsPort”.

The second change is the removal of the attribute “defaultMode”. This allows us later to switch between TLS-secure and “insecure” access methods as we like. Remember that we always can make a insecure” connection (regrding a lack of TLS) secure at any time by building a SSH tunnel.

Some reader may ask why we did not set the default mode to “secure” and deleted the “port”-attribute. Well, if we were in a position in which we had tested the TLS configuration already and wanted to use TLS only in the future (and not SSH, for example) then we could make these changes, yes. But I keep the option of using a SSH-tunnel open. TLS is anyway the preferred option if the associated port is specified by the Spice client (here remote-viewer).

The reader certainly has noticed that I activated Spice image data compression. We made a good experience with it the other day.

How to deal with the CA-certificate on the Leap 15.2 client-system? Where to place it?

OpenSSL will validate the whole CA-chain when confronted with a server-certificate. It needs the CA-certificate for it. In addition it should trust the related public key – as the CA-root-certificate is self-signed. This means that we have to make the certificate of our private CA known to the client-system(s) – here MyLAP – on which we want to start a Spice client requesting a TLS encrypted connection from the KVM/Qemu-server.

Opensuse Leap systems are a bit picky about were to place the CA-certificates. To say it clearly: “/etc/ssl/certs” is the wrong place!

Any certificates unknown to Leap 15.2 will not survive a reboot there. They won’t even survive a call of the “update-ca-certificate“-program, which would make the CA-certificate known to other programs as a usable and trustworthy one. So, placing the “ca-cert.pem”-file of our private CA into “/etc/ssl/certs” will lead to severe problems: It won’t be found or won’t be accepted during the start of remote-viewer on a client-system as MyLAP.

Note:

The right place for the certificate of our private CA on a Leap-system is “/etc/pki/trust/anchors/“.

You should copy it there. By the way: You should use the name given to your CA initially – not necessarily the special file name used for Qemu on the KVM/Qmu server. In my case it is something like “anraconc-CA”, thus “anraconc-CA.pem”.

So, we perfom the following copy-process

CaSRV:~ # scp /root/.TinyCA/anraconc-CA/cacert.pem root@MyLAP:/etc/pki/trust/anchors/anraconc-CA.pem
Password:

On MyLAP we then enter

mylap:/etc/pki/trust/anchors # la 
-r--r--r-- 1 root root  2540 Apr 11 11:11 anraconc-CA.pem
mylap:/etc/pki/trust/anchors # update-ca-certificates
mylap:/etc/pki/trust/anchors # la /var/lib/ca-certificates/openssl | grep anrac
lrwxrwxrwx 1 root root    15 Apr 11 12:13 610c65bd.0 -> anraconc-CA.pem
-r--r--r-- 1 root root  2540 Apr 11 12:13 anraconc-CA.pem
lrwxrwxrwx 1 root root    15 Apr 11 12:13 bfb4c341.0 -> anraconb-CA.pem

“update-ca-certificates” makes the CA-certificate system-wide. The look into “/var/lib/ca-certificates/openssl” just was a check for this.
That is all we have to do regarding certificates on the client.

Test of the TLS encrypted remote connection to the Spice console

We first open our firewalls for connection from MyLAP to port 20002 on MySRV. On our KVM/Qemu-host we then start the libvirtd-daemon again to cover all
changes to Qemu and the VM. Afterward our privileged user “uvma” starts the VM “debianx” for us on the server. This should work without any problems – there should not be any errors regarding the TLS options and TLS files.

Then we have a brief look at https://libvirt.org/uri.html and https://www.spice-space.org/spice-user-manual.html to get an idea how we have to formulate our remote-viewer parameters for TLS:

myself@mylap:~> remote-viewer spice://mysrv.anraconc.de?tls-port=20002

(remote-viewer:29853): GSpice-WARNING **: 15:29:34.201: Warning no automount-inhibiting implementation available

with results that look very similar to what we have done in the last article. I omit the proof by screenshots as we would get no new information from them.

But did you notice a major difference besides the special way of specifying the TLS-port to use ?
We have to use the FQDN of the server!
Exactly in the form it was filled as the “common name” into the server-certificate! If we just used “mysrv” this would lead to an error message of OpenSSL – despite the fact that the short name can be resolved by a DNS server.
Note also that something like “remote-viewer spice://mysrv.anraconc.de:20002” will NOT work.

The required form of the command for remote-viewer with TLS is :

remote-viewer spice://FQDN_OF_KVM_HOST?tls-port=VM_SPECIFIC_PORT_NR

You should also test that the variant

myself@mylap:~> remote-viewer --spice-ca-file=/etc/pki/trust/anchors/anraconc-CA.pem  spice://mysrv.anraconc.de?tls-port=20002

works flawlessly.

Checking for encryption

A look at netstat on the server proves a connection to port 20002:

mysrv:~ # netstat -an | grep 20002
tcp        0      0 0.0.0.0:20002           0.0.0.0:*               LISTEN     
tcp        0      0 192.168.2.4:20002      192.168.2.22:36650      ESTABLISHED
tcp        0      0 192.168.2.4:20002      192.168.2.22:36646      ESTABLISHED
tcp        0      0 192.168.2.4:20002      192.168.2.22:36642      ESTABLISHED
tcp        0      0 192.168.2.4:20002      192.168.2.22:36648      ESTABLISHED
tcp        0      0 192.168.2.4:20002      192.168.2.22:36658      ESTABLISHED
tcp        0      0 192.168.2.4:20002      192.168.2.22:36656      ESTABLISHED
tcp        0      0 192.168.2.4:20002      192.168.2.22:36640      ESTABLISHED
tcp        0      0 192.168.2.4:20002      192.168.2.22:36644      ESTABLISHED
tcp        0      0 192.168.2.4:20002      192.168.2.22:36652      ESTABLISHED
tcp        0      0 192.168.2.4:20002      192.168.2.22:36654      ESTABLISHED
mysrv:~ #

While this only shows that a connection to the right port is established. However, an additional look with wireshark shows you the TLS version (which it gets from protocol headers):

We see that TLS V1.3 is indeed used to encrypt the application data crossing port 20002.

Data transfer rates and responsiveness

The data transfer rates for the present scenario overall are very similar to the ones measured for the SSH-scenarios with Spice data compression. Maybe a tiny bit higher for TLS. But there are too many impact factors to really say this. The responsiveness of the window manager and single applications are excellent – as with the SSH scenario of the last article. I leave it to the reader to test it out on his own.

Does TLS encryption to the VM work locally, too?

An interesting question is whether we can have
encryption locally on the KVM/Qemu-server, too. The answer is: Yes, but you still have to provide the FQDN of the server; the network request will nevertheless be handled over the “lo”-device. This at least enables you to test your VM settings locally. But local encryption could also be interesting in some multi-user scenarios.

Using different options for the defaultMode and the Spice data channels

Allowed values for the “defaultMode” attribute in the Spice configuration of the VM are “secure”, “insecure” and “any”. With “any” you can switch between encrypted and unencrypted access at any time by closing the Spice windows and opening them again with different settings for the remote-viewer. You should check which port is taken by looking at the output of e.g. netstat. Using “secure” will enforce TLS-encryption; if no sufficient certificates were in place the VM would not even start then.

An important feature regarding TLS is that you may define which Spice data channel should/must be encrypted. You find more information about this at https://libvirt.org/formatdomain.html#graphical-framebuffers. I quote from the documentation there:

When SPICE has both a normal and TLS secured TCP port configured, it can be desirable to restrict what channels can be run on each port. This is achieved by adding one or more elements inside the main element and setting the mode attribute to either secure or insecure. Setting the mode attribute overrides the default value as set by the defaultMode attribute. (Note that specifying any as mode discards the entry as the channel would inherit the default mode anyways.) Valid channel names include main, display, inputs, cursor, playback, record (all since 0.8.6 ); smartcard ( since 0.8.8 ); and usbredir ( since 0.9.12 ).

The example given in the libvirt documentation is: 

    
<graphics type='spice' port='-1' tlsPort='-1' autoport='yes'>
  <channel name='main' mode='secure'/>
  <channel name='record' mode='insecure'/>
  <image compression='auto_glz'/>
  <streaming mode='filter'/>
  <clipboard copypaste='no'/>
  <mouse mode='client'/>
  <filetransfer enable='no'/>
  <gl enable='yes' rendernode='/dev/dri/by-path/pci-0000:00:02.0-render'/>
</graphics>

Off topic: You see that there are more options for Spice – e.g. the streaming parameter, which can be used for video-streaming. In a quiet minute the interested reader should have a look into the Spice documentation at
https://www.spice-space.org/spice-user-manual.html
and
https://qemu.readthedocs.io/en/latest/system/invocation.html
to get more information.

What about other security elements?

It is nice that we now are able to use a TLS encrypted connection for remote-viewer. But a noticeable disadvantage in comparison to the SSH-tunnel scenario of my last article is that we must keep the Spice TLS-port open and accessible on the server. In addition: Basic security measures on a server should also include some form of user authentication – and the access to the Spice console of a VM should be restricted to selected users. With the TLS-elements discussed above alone we cannot achieve this. So, you have to wait for yet another blog article.

In the meantime you can and should at least set a password for the Spice console: 

    
<graphics type='spice' port='20001' tlsPort='20002' autoport='no' keymap='de' defaultMode='any' password='my_verysecret_pwd' >
.....

See the 2nd article of this series about it.

Conclusion

It is relatively easy to
configure a KVM/Qemu-server such that it encrypts Spice data with TLS. We tested this with a remote-viewer instance started on a client-system somewhere in our Intranet. A basic requirement is of course the creation of a private CA and certificates/keys. Tools like TinyCA or XCA help us with this.
As remote-viewer directly talks to the Qemu-emulator we had to configure libvirtd to enable dependent tools like “virsh” and “virt-manager” to start qemu-processes with the required parameters for TLS. This could be done via a few settings in the file “/etc/libvirt/qemu.conf”.
This type of qemu-configuration differs from activating TLS for a remote access to libvirt-based tools themselves. As a consequence the private server-key to be used by the Qemu process must be made readable for the “qemu”-user.
The TLS setup for remote-viewer and Spice neither allowed for user authentication on the server nor for user-specific restrictions so far. In the next article

KVM/Qemu VMs with a multi-screen Spice console – VII – remote-viewer, qemu and SASL authentication

we shall, therefore, have a look at methods for user authentication combined with TLS.

Links

Qmu, Spice und TLS
https://qemu-project.gitlab.io/qemu/system/tls.html

https://www.libvirt.org/tlscerts.html

https://ravada.readthedocs.io/en/latest/docs/spice_tls.html

Spice + defaultMode
https://libvirt.org/formatdomain.html#video-devices

OpenSSL to check for a valid TLS certificate on a network port
https://tenable.force.com/s/article/Using-OpenSSL-to-verify-certificate-information-on-a-port

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – III

Posted on by

I proceed with my article series on using TinyCA2 instead of YaST’s CA-admin tools, whic are simply missing since Opensuse Leap 15.0. Which I regard as a shame by the way …

In the first article

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – I

I briefly introduced the user interface of TinyCA2. We used it to create a new CA and a server certificate.

In the second article

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – II

we found out where to place the new CA’s root certificate on an Opensuse Leap system to make it available system wide and permanently. We also used the newly created server certificate for the server “myserv.anraconc.de” for all services on this server – and placed the necessary files into “/etc/ssl/servercerts”.

We then discussed the reconfiguration of an Apache2-, an OpenLDAP- and the SSSD-service as examples of how to adapt to the CA and certificate changes. All the required operations on “myserv” were local copy and file editing operations – as “myserv” was the server for administering our new CA, too.

In this third article we, however, look at a second independent server “mymail.anraconc.de” for IMAP/SMTP-services. This server needs its own TLS/SSL-server-certificate. In lack of other tools we shall use scp to deploy the necessary files. I shall further discuss how to reconfigure “cyrus” and “postfix”. Note that these services do not only offer their own TLS/SSL connections to their clients; at the same time they themselves will work as LDAP-clients, i.e. as clients for the OpenLDAP service on server “myserv”. Therefore we must adapt the SSSD-service on “mymail”, too.

Placing a TinyCA2 server certificate on a mail server with cyrus, saslautd and postfix

Our first step is to use TinyCA2 on our server “myserv” to create a second server certificate for the server “mymail” with the common name “mymail.anraconc.de”. This is done exactly as described in my first article for server “myserv”. We export the related certificate and the key files to “/etc/certs” on “myserv” and limit access to the private key file. We export the key file with its password – and strip it off afterwards: 

myserv:/etc/certs # openssl rsa -in /etc/certs/mymail-anraconc-key.pem -out /etc/certs/mymail-anraconc-key_new.pem
Enter pass phrase for /etc/certs/mymail-anraconc-key.pem:
writing RSA key
myserv:/etc/certs # cp /etc/certs/mymail-anraconc-key_new.pem /etc/certs/mymail-anraconc-key.pem

Now, we look for some export/import mechanism for deploying “server certificates” from our sytem with the CA administration tool TinyCA2 to other systems. Unfortunately, TinyCA2 does not offer such a client/server feature for certificate deployment within a network. We are forced to transfer our certificates manually 🙁 .

I assume that we have a ssh-connection form our sever “myserv” to our mail-server “mymail”. 

myserv:/etc # scp /etc/certs/mymail-anraconc-cert.pem root@mymail:/etc/certs
Password: 
mymail-anraconc-cert.pem                   100% 2147     1.5MB/s   00:00    
myserv:/etc # scp /etc/certs/mymail-anraconc-key.pem root@mymail:/etc/certs
Password: 
mymail-anraconc-key.pem                    100% 1679     1.4MB/s   00:00    
myserv:/etc # scp /etc/certs/anraconc-CA.pem root@mymail:/etc/certs
Password: 
anraconc-CA.pem                           100% 2504     1.7MB/s   00:00    
myserv:/etc #

Note that I transferred the CA certificate, too.

On “mymail”: 

mymail:~ # cp /etc/certs/anraconc-CA.pem /etc/pki/trust/anchors/
mymail:~ # cp -r /etc/ssl/servercerts/ /etc/ssl/servercerts.orig
mymail:~ # cp /etc/certs/mymail-anraconc-cert.pem /etc/ssl/servercerts/servercert.pem
mymail:~ # cp /etc/certs/mymail-anraconc-key.pem /etc/ssl/servercerts/serverkey.pem
mymail:~ # chmod 600 /etc/ssl/servercerts/serverkey.pem 
mymail:~ #

If you read my second article of this series these steps need no further explanation.

Reconfiguring sssd, cyrus and postfix for the new SSL/TLS certificates

First, the sssd-service configuration in the file “/etc/sssd/sssd.conf” has to be updated on mymail:

.....
[pam]
[domain/default]
ldap_uri = ldap://myserv.anraconc.de
ldap_search_base = dc=anraconc,dc=de
ldap_schema = rfc2307bis
id_provider = ldap
ldap_user_uuid = entryuuid
ldap_group_uuid = entryuuid
ldap_id_use_start_tls = False
enumerate = True
cache_credentials = False
ldap_tls_cacertdir = /etc/ssl/certs
ldap_tls_cacert = /etc/ssl/certs/anraconc-CA.pem
chpass_provider = ldap
auth_provider = ldap
...
....

Note that the ldap_uri is the same (!) as on “myserv”! See the last article!

We then change “/etc/ldap.conf” – this enables any clients : 

uri     ldap://myserv.anraconc.de
base    dc=anraconc,dc=de
nss_map_attribute       uniqueMember member
ssl     start_tls
tls_cacertdir   /etc/ssl/certs
tls_cacertfile  /etc/ssl/certs/anraconc-CA.pem              
pam_password    exop
pam_filter      objectClass=posixAccount
...

Afterwards we change the file “/etc/imapd.conf” – which is a central piece for a working cyrus-based IMAP-server: 

....
# LMTP
# ****
lmtp_overquota_perm_failure: no
lmtp_downcase_rcpt: yes

# SASL
# ****
allowplaintext: yes
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN LOGIN

# TLS
# ****
tls_key_file: /etc/ssl/servercerts/serverkey.pem
tls_cert_file: /etc/ssl/servercerts/servercert.pem
tls_ca_file: /etc/ssl/certs/anraconc-CA.pem
...
...

You see that we saslauthd as an intermediate link of imap to authenticate (cyrus) users via LDAP (on another server).

Important Note:
The system users “mail” and “cyrus” must get read access to “/etc/ssl/servercerts/serverkey.pem” in case we use exactly one server-wide server certificate/key for all services. Otherwise TLS connections (starttls) cannot be provided for the imap/cyrus-service.

One solution for this problem is to use ACL-settings analogously to what we have done above for user “ldap” on the server “myserv”. See my last article …

For SMTP we also need to change “/etc/postfix/main.cf“: 

...
....
 Client for the local SMTP-Server
#---------------------------------

# Changed by admin according to local sasl politics
#----------------------------------------------------
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = cyrus
# Postfix > 2.3
smtpd_sasl_path = smtpd
broken_sasl_auth_clients = yes


# Local SMTP server as a client for Relay servers 
#------------------------------------------------
# Inserted by admin - 06.01.2017
#-----------------------------------

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
...
....
############################################################
# TLS stuff
############################################################
relay_clientcerts =
#tls_random_source = dev:/dev/urandom

smtp_use_tls = yes
#smtp_tls_loglevel = 0
#smtp_enforce_tls = no
smtp_tls_policy_maps = hash:/etc/postfix/tls_policy
smtp_tls_CApath = /etc/ssl/certs/
smtp_tls_
CAfile = /etc/ssl/certs/anraconc-CA.pem

smtp_tls_session_cache_database =

smtpd_use_tls = yes
smtpd_tls_auth_only = yes
smtpd_tls_loglevel = 2
smtpd_tls_CAfile = /etc/ssl/certs/anraconc-CA.pem
smtpd_tls_cert_file = /etc/ssl/servercerts/servercert.pem
smtpd_tls_key_file = /etc/ssl/servercerts/serverkey.pem
smtpd_tls_ask_ccert = no
smtpd_tls_received_header = yes

...
...

You saw again that I use saslauth for authentication. But, actually there is nothing to be changed in the saslauthd configuration in comparison to a previously working solution.

Finally, we restart all our services: 

mymail:/etc/sasl2 # systemctl restart sssd.service
mymail:/etc/sasl2 # systemctl restart cyrus.service
mymail:/etc/sasl2 # systemctl restart postfix.service
mymail:/etc/sasl2 # systemctl restart saslauthd.service

This should work flawlessly, if we have done everything correctly.

Testing with Kmail and other mail clients from client systems

Before you can start testing with email-clients you need to transfer the new CA’s certificate to your client system’s directory “/etc/pki/trust/anchors/”. Do not forget to import “anraconc-CA.pem” on all other clients that build up TLS or SSL connections to your reconfigured servers, too. And put the CA certificate into “/etc/pki/trust/anchors/” – if your clients are Opensuse Leap 15.x systems.

If you need any direct LDAP client services form your client system to the LDAP service on “myserv” you should take care of the necessary changes in the local “/etc/ldap.conf” and the “/etc/sssd/sssd.conf” !

Otherwise you should have a look at Kmail’s or your email’s IMAP account configuration:

Note that you should give the name of the mail server in its FQDN-form to avoid messages on certificates not fitting the server. And:

Clicking on the button “Automatisch erkennen” (or whatever the button is named in English …”Automatic detection ?) should offer you STARTTLS now. If you get any messages about “no connection could be enabled” your configuration of the mail server still has deficits – either in offering TLS/SSL encryption or in its role as a client to the OpenLDAP server. You would have to analyze status and log information of all involved servers then.

You have to do similar things on other email-clients. I was lucky – things started to operate smoothly again on my mail clients.

Conclusion

It really is a problem that the SuSE people did not replace the YaST module for the administration of TLS/SSL CAs and server certificates by a Ruby counterpart with the start of Leap 15.0. Especially as Opensuse works as a basic platform for SLES 🙁 . Also SuSE previously often used a policy to allow for server wide certificates – i.e. one certificate for all services on a server instance. The deployment of such certificates issued by a central CA was supported by an export/import mechanism of YaST, which took care of password-protected keys by stripping of password and setting the right read access rights on the destination servers.

If you want to use TinyCA2 as a CA administration tool you are confronted with a series of challenges – among others the question of access rights of system users to certificate’s key files. In addition you have to place the CA’s root certificate at a proper location on Opensuse Leap systems (/etc/pki/trust/anchors) to
make it permanently available.

TinyCA2 – which I had a look at – is relatively old. The last Gitub change appeared in 2015. Nevertheless it does its job to create a CA, sub-CAs and server certificates (as well as client certificates) – and to revoke them if required. It also offers a simple local export mechanism for certifictaes and keys into standard file formats – as pem. However, this is it.

You need a patch to include SHA256 or SHA512 – which is a must these days. There are some flaws with the simple GUI, but these you can work around related problems – if you know about them. And: TinyCA2 will not help you to deploy the certificates/keys on your server or client systems. Not even a simple import mechanism for a server wide “server certificate plus key” issued by the CA is offered on target servers. So, if you do not want to invest time into your own deployment scripts you may come to the conclusion that TinyCA2 is not suited well for large networks with many servers. In my case I fount it applicable for a dozen servers – but not more. Too much handwork ….

If you consider to use TinyCA2 as a replacement for YaST’s CA administration tools most of your work will be to deploy the TLS/SSL certificates/keys onto your server systems and to reconfigure all services to use these certificates for TLS, StartTLS and SSL connections.

On Opensuse systems the deployment of the CA root certificate within the bunch of permanently trusted CA certificates requires some knowledge about the relevant directory to make the CA certificate available in a central directory as “/etc/ssl/certs” – which may be used by legacy applications. The access rights to a server certificates key file may pose some special problems – you may have to test which system user associated with your services requires read rights.

There are actually several ways how to handle the read access topic for the private certificate keys. I have shown how to use ACLs for this purpose. My recommendation, however, is to use separate certificates and related keys for all services – even if they reside on the same server and despite consequences for your network and DNS configuration due to unique FQGNs and common names. You never know what can happen to a service specific user during attacks. If one certificate/key gets corrupted the others may still survive. Actually, this could be a reason for encapsulating services even more in form of LXC containers. But this is a different story …

Good luck with Opensuse and TinyCA2 !

 

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – I

Posted on by

Today server services should offer network connectivity for clients with encryption. On Linux StartTLS based services are common – for LDAP, email/groupware servers as well as web servers. To set up SSL/TLS/StartTLS based services we need certificates and encryption keys issued by a central CA – which we trust. Administering your own local CA and server certificates can be a bit challenging without graphical tools – even in smaller networks with a dozen server instances.

In our networks with mainly Opensuse and Debian servers I had used YaST’s CA-module to create a CA and server certificates signed by this CA. The stupid thing is that the required “yast2-ca”-module and its RPM are missing since Opensuse Leap 15.0. This was not a major problem so far; the update processes respected existing certificates, of course. However, some days ago two of my central server certificates – namely the one for my LDAP-server and an Apache2-server – expired. This in turn lead to a breakdown of several other services on other (virtual) machines: SSSD, IMAP, Postfix (SMTP, because these services use the LDAP server among other things as a backend for user authentication. (SSSD itself provides a TLS connection to LDAP.)

The Opensuse documentation cha.security.yast_ca.html is really misleading because it claims to be valid for Opensuse Leap 15.1 – which it is not, as there still is no yast-ca-module available. For me this kind of policy of Opensuse is unbelievable; doesn’t Leap provide the basic platform for SLES? How shall SLES admins in smaller companies tackle the resulting problems? Buy a PKI tool? Everybody talks about secuirty …. but SuSE (???)

I wanted some cost free alternative for my own network – and as a first trial I went for “TinyCA2“.

This became more of an adventure than expected. Part of the hurdles were due to Opensuse specific settings – but also due to the very many different configuration files which had to be adapted for the certificate of my new CA – which came in addition to my old one. (I did not yet want to give up the old CA as some (virtual) servers still have valid server certificates from it.) Another obstacle appeared when Opensuse deleted any new files in “/etc/ssl/certs” after a system restart. Also the GUI of TinyCA2 has some strange “features” regarding default values, which I did not become aware of during my first trials. In addition it seemed to be necessary to replace SHA1 by SHA256. And in the end I got e.g. Apache running with its new server certificate, but not e.g. the slapd.service – due to a access rights problem which was difficult to see.

In this article I shall describe most of the required steps for switching to a TinyCA2 CA and adjusting server settings. I shall concentrate on some simple services as examples. But I hope the general pattern of how to proceed will become clear and help others, who work with Opensuse, to save a bit of time.

Installing and patching TinyCA2

Both Opensuse Leap 15.0 and Opensuse Leap 15.1 provide RPMs for TinyCA2 version 0.7.5. Which is from 2015. If you have a look at GitHub (see the link in the last section of this article) you may also find some (important) patches. On an Opensuse system You install the RPM easily with the help of YaST (yast2). After the installation you find the Perl files of TinyCA2 in the directory “/usr/share/TinyCA2/lib“.

When you start TinyCA2 via the command “tinyca2 &” the first thing you may stumble across is the fact that (among other digests) MD5 and SHA1 are offered as hashing algorithms. Look at the bottom part of the following screenshot:

(By the way: The layout – especially the icons – may look different on your system. It depends on your graphical desktop and your settings for GTK applications)

You see that we get a variety of hashing algorithms offered under the category “Digest”. Most of them are regarded insecure today. So, even in a semiprofessional environment you would like to see something better – e.g. SHA256. Fortunately, another guy (Bill Thorsteinson) had the same problem and he has created a patch for TinyCA2 which enables SHA256. You find the patch at
https://www.systemajik.com/tinyca-sha2/.

Let us try this out; on my ssh session to my central server “myserv” (this is the one with LDAP): 

myserv:~ # mkdir /extras/Updates/tinyca
myserv:~ # wget https://www.systemajik.com/wp-uploads/2014/10/tinyca_sha256.patch_.txt -O /central/Updates/tinyca/tinyca_sha256.patch_.txt
--2019-07-20 12:44:57--  https://www.systemajik.com/wp-uploads/2014/10/tinyca_sha256.patch_.txt
Resolving www.systemajik.com (www.systemajik.com)... 206.47.13.3, 2001:470:1f11:b22::8
Connecting to www.systemajik.com (www.systemajik.com)|206.47.13.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4863 (4.7K) [text/plain]
Saving to: ‘/central/Updates/tinyca/tinyca_sha256.patch_.txt’

/central/Updates/tinyca/tinyca_sha256.pat 100%[=================>]   4.75K  --.-KB/s    in 0s      

2019-07-20 12:44:58 (138 MB/s) - ‘/central/Updates/tinyca/tinyca_sha256.patch_.txt’ saved [4863/4863]

myserv:~ # 
myserv:~ # cd /usr/share/TinyCA2/lib
myserv:/usr/share/TinyCA2/lib # cp /extras/Updates/tinyca/tinyca_sha256.patch_.txt .
myserv:/usr/share/TinyCA2/lib # patch --verbose -p1 < tinyca_sha256.patch_.txt
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From e5e25e55f8da2b4d2bad584f2145ca0ff6b3a92a Mon Sep 17 00:00:00 2001
|From: Bill Thorsteinson <bill.git@systemajik.com>
|Date: Thu, 30 Oct 2014 22:26:47 -0400
|Subject: [PATCH] Apply changes
|
|---
...
...
|--- a/REQ.pm
|+++ b/REQ.pm
--------------------------
patching file REQ.pm
Using Plan A...
Hunk #1 succeeded at 59.
Hunk #2 succeeded at 426.
Hmm...  Ignoring the trailing garbage.
done

Note: The “-p1” in “patch –verbose -p1 < tinyca_sha256.patch_.txt” reads “-pONE” and not “-pL” with a small L-letter.

A “tinyca2” command now produces:

Much better !

Importing the old CA from Opensuse?

If you play around with the menus of TinyCA2 you find an option to import other CAs. Could this work with my old YaST-CA? To make a long story short – I did not succeed with this. The reasons are still unclear to me …. TinyCA could not read the relevant information.

So, I really was forced to set up a new CA – with all consequences as issuing and deploying new server certificates and the (trusted) CA certificate on my servers (and the CA cert also on my client machines). Which even in my little network (12 servers- thanks to virtualization) is painstaking …

Creating a new TinyCA2 based CA

Let us create a new CA with TinyCA2. We have some freedom regarding the “common name“. I choose a reference to my main internal domain “anraconc.de” – so my common name is: “anraconc-CA”. (It only looks like an official Internet domain; but actually it is an internal domain, only, and my DNS server is configured accordingly.)

Important hint:
Change the settings for Keylength and Digest by explicitly clicking first on other values and then the real choice again! If you do not change anything explicitly you may get a surprise regarding default values. They may not be what is indicated. Seems to be a bug. Do not disregard this hint if you want to save time ….

Now, a click on the “OK”-button gives us:

We set the keyUsage to “critical” (this certificate extension is used by some applications). And we eventually get all the information about our CA certificate:

The data – and especially the private key – can be found in the directory “/root/.TinyCA/anraconc-CA/”. TinyCA2 creates such a directory for every main CA. (If you use sub-CAs you will find respective directories below it). 

myserv:~/.TinyCA # cd anraconc-CA/
myserv:~/.TinyCA/anraconc-CA # la
total 44
drwx------ 7 root root 4096 Jul 20 13:13 .
drwx------ 5 root root 4096 Jul 20 13:12 ..
-rw------- 1 root root 3311 Jul 20 13:13 cacert.key
-rw------- 1 root root 2504 Jul 20 13:13 cacert.pem
drwx------ 2 root root 4096 Jul 20 13:12 certs
drwx------ 2 root root 4096 Jul 20 13:13 crl
-rw------- 1 root root    0 Jul 20 13:12 index.txt
drwx------ 2 root root 4096 Jul 20 13:12 keys
drwx------ 2 root root 4096 Jul 20 13:12 newcerts
-rw------- 1 root root 3872 Jul 20 13:13 openssl.cnf
drwx------ 2 root root 4096 Jul 20 13:12 req
-rw------- 1 root root    2 Jul 20 13:12 serial

Hint: You should make a backup of the CA directories on a periodic basis.

Now, you can export the CA certificate in form of a standard pem-file to some intermediate place where you gather your own certificates and keys – in my case this is a directory “/etc/certs” – which so far survived any Opensuse upgrades. Depending on what else you intend to save there (private keys?), you should make this place accessible to root only! We click on the second to last icon in the icon row of TinyCA2:

Note: In general you have any freedom here to give the exported file any kind of name – whatever you like. However, it is a good policy to use the “common name” which you gave to the CA certificate. See below for the reason.

Place the CA certificate at a central location for trusted CAs

We can now export this certificate file with public information to servers into directories where we gather the public certificates (keys) of all trusted CAs. Of course we need to do this on the server “myserv”, too, as some services may refer to it. In my age my first guess is “/etc/ssl/certs“; old habit form a decade ago where this directory was used more frequently. 

myserv:~/.TinyCA/anraconc-CA # cp /etc/certs/anraconc-CA.pem /etc/ssl/certs
myserv:~/.TinyCA/anraconc-CA # chmod 640 /etc/ssl/certs/anraconc-CA.pem 
myserv:~/.TinyCA/anraconc-CA #

A wrong decision in the end – see below. But for our present session this will work.

Note: If we would create Sub-CAs we would have to export all respective pem files to such a central location – the whole CA-chain must be reflected there for the verification of a service whose “server certificate” has been issued by a sub-CA. I do not use Sub-CAs in this article – but it my be necessary in your organization!

Create a server certificate

Now, we need to create “server certificates” or even service specific certificates. It depends on your policy of how far you want to discriminate services.

In this article I follow the path of a server wide central “server certificate” for all the services implemented there. As examples we shall later have a look at a local OpenLDAP service and a local Apache web server. My central server “myserv” with OpenLDAP has a FQDN of “myserv.anraconc.de”.

Important note: You must use the FQDN as a “common name” in server certificates – consistent with DNS settings. Otherwise you may risk warnings of security aware applications that the server certificate does not fit the server!

in our TinyCA2 window we click on the tab “Certificates” and then on the empty sheet icon:

We fill in the required data. As keys protected by a password may cause trouble for services during automated system startups we try to leave the password-fields empty:

But this approach is not accepted!

So we type in some lengthy password – and set the options for Digest and Algorithm again explicitly – by clicking a bit around first (see above). Then we click on “OK” and get:

Think a bit about the validity! Actually, the length of the validity period should be somewhat shorter than the period for your CA! E.g. 5 years. Otherwise you will get a warning. Eventually:

If you click on the tab “Keys” you will see a related (private) key, too.

Important Note:
We have taken a shortcut here. You could have started in a different way – namely via a certificate “request”. In a first step you would then have issued such a “request” under the tab “Requests” and filled out an initial form there. Afterwards, you explicitly need to sign the requested certificate with the CA’s signature. You get the option for signing by right-clicking on the request entry after its creation. This approach also leads to a valid certificate.

Exporting the server certificate and the key to a central location on the Opensuse system

We need to export the certificate and the (private) key to some
save location on our server “myserv” – only accessible to root (and maybe read accessible to some special system user). On an Opensuse Leap system the location for server certificates should be “/etc/ssl/servercerts“. For exporting the certificate we right-click on the entry:

Then we switch to the tab “Keys” and do the same there:

Important note:
At this point we get an option to export the key without a password. I have chosen this option. This implicates security risks – your exported private key is protected by nothing afterwards. So be very careful where you save it and with which access rights. On the other side such a key will allow for automated service starts – otherwise someone would have to provide the password during startup. I do not want to deepen the discussion here. But be careful with unprotected private keys!

You saw that I exported into my intermediate directory “/etc/certs”. There we change rights for security reasons to: 

myserv:/etc/ssl # chmod 600 /etc/certs/myserv-anraconc-key.pem

Note:
If you instead export your key with the password there is a way to get rid of it afterwards:

myserv:/etc/certs # openssl rsa -in /etc/certs/myserv-anraconc-key.pem -out /etc/certs/myserv-anraconc-key_new.pem
Enter pass phrase for /etc/certs/myserv-anraconc-key.pem:
writing RSA key
myserv:/etc/certs # cp /etc/certs/myserv-anraconc-key_new.pem /etc/certs/myserv-anraconc-key.pem

This creates a key without password! Actually I recommend to use this approach – we do not know details of TinyCA2’s export procedure, but “openssl” will always create the key in the format required for further processing.

Now, before any further steps, I make a backup of everything existing in the folder “/etc/ssl/servercerts“. This is important! If you loose your previous certificates and keys they are gone and you have no chance to get services running with them.

Then we overwrite the existing entries:

 
myserv:/etc/ssl # cp /etc/certs/myserv-anraconc-cert.pem /etc/ssl/servercerts/servercert.pem 
myserv:/etc/ssl # cp /etc/certs/myserv-anraconc-key.pem /etc/ssl/servercerts/serverkey.pem 
myserv:/etc/ssl # chmod 600 /etc/ssl/servercerts/serverkey.pem

Note: The last step is of fundamental importance due to security reasons! See the discussion below if this leads to trouble for some services and the related system user.

You may try “644” in the beginning to avoid any problems with system users running special services. But if you do this then DO NOT FORGET to restrict the read rights again in the end and after your tests.

Note that replacing the contents of /etc/ssl/severcerts” will probably lead to a breakdown of all services which base their TLS/SSL functionality on these files. In most cases the reason for this will be that the configuration will refer to a wrong CA-certificate. Therefore, you must reconfigure your local services step by step.

Conclusion

Enough for today. We have seen how TinyCA2 can be patched for SHA256, how it can be used to create a CA and server certificates. In the next article

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – II

we shall reconfigure an Apache and an LDAP service to work with the new server certificate. And I shall show how we can make the CA-certificate permanently available in “/etc/ssl/certs” – across any server reboot. Stay tuned !

Links

CAs and Certificates – general information
https://wiki.ubuntuusers.de/CA/
stackexchange.com: what-role-do-hashes-play-in-tls-ssl-certificate-validation
stackexchange.com how-does-ssl-tls-work
robpol86.com root certificate authority
https://roll.urown.net/ca/ca_cert.html

TinyCA2
https://github.com/glennie/tinyca2
linux-magazin 2008: eigene zertifikatsstelle mit tinyca

TinyCA2 patches
https://www.systemajik.com/tinyca-sha2/.

Critical
stackexchange.com: which-properties-of-a-x-509-certificate-should-be-critical-and-which-not