TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – III

I proceed with my article series on using TinyCA2 instead of YaST's CA-admin tools, whic are simply missing since Opensuse Leap 15.0. Which I regard as a shame by the way ...

In the first article

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – I

I briefly introduced the user interface of TinyCA2. We used it to create a new CA and a server certificate.

In the second article

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – II

we found out where to place the new CA's root certificate on an Opensuse Leap system to make it available system wide and permanently. We also used the newly created server certificate for the server "myserv.anraconc.de" for all services on this server - and placed the necessary files into "/etc/ssl/servercerts".

We then discussed the reconfiguration of an Apache2-, an OpenLDAP- and the SSSD-service as examples of how to adapt to the CA and certificate changes. All the required operations on "myserv" were local copy and file editing operations - as "myserv" was the server for administering our new CA, too.

In this third article we, however, look at a second independent server "mymail.anraconc.de" for IMAP/SMTP-services. This server needs its own TLS/SSL-server-certificate. In lack of other tools we shall use scp to deploy the necessary files. I shall further discuss how to reconfigure "cyrus" and "postfix". Note that these services do not only offer their own TLS/SSL connections to their clients; at the same time they themselves will work as LDAP-clients, i.e. as clients for the OpenLDAP service on server "myserv". Therefore we must adapt the SSSD-service on "mymail", too.

Placing a TinyCA2 server certificate on a mail server with cyrus, saslautd and postfix

Our first step is to use TinyCA2 on our server "myserv" to create a second server certificate for the server "mymail" with the common name "mymail.anraconc.de". This is done exactly as described in my first article for server "myserv". We export the related certificate and the key files to "/etc/certs" on "myserv" and limit access to the private key file. We export the key file with its password - and strip it off afterwards:

myserv:/etc/certs # openssl rsa -in /etc/certs/mymail-anraconc-key.pem -out /etc/certs/mymail-anraconc-key_new.pem
Enter pass phrase for /etc/certs/mymail-anraconc-key.pem:
writing RSA key
myserv:/etc/certs # cp /etc/certs/mymail-anraconc-key_new.pem /etc/certs/mymail-anraconc-key.pem

Now, we look for some export/import mechanism for deploying "server certificates" from our sytem with the CA administration tool TinyCA2 to other systems. Unfortunately, TinyCA2 does not offer such a client/server feature for certificate deployment within a network. We are forced to transfer our certificates manually 🙁 .

I assume that we have a ssh-connection form our sever "myserv" to our mail-server "mymail".

myserv:/etc # scp /etc/certs/mymail-anraconc-cert.pem root@mymail:/etc/certs
Password: 
mymail-anraconc-cert.pem                   100% 2147     1.5MB/s   00:00    
myserv:/etc # scp /etc/certs/mymail-anraconc-key.pem root@mymail:/etc/certs
Password: 
mymail-anraconc-key.pem                    100% 1679     1.4MB/s   00:00    
myserv:/etc # scp /etc/certs/anraconc-CA.pem root@mymail:/etc/certs
Password: 
anraconc-CA.pem                           100% 2504     1.7MB/s   00:00    
myserv:/etc # 

Note that I transferred the CA certificate, too.

On "mymail":

mymail:~ # cp /etc/certs/anraconc-CA.pem /etc/pki/trust/anchors/
mymail:~ # cp -r /etc/ssl/servercerts/ /etc/ssl/servercerts.orig
mymail:~ # cp /etc/certs/mymail-anraconc-cert.pem /etc/ssl/servercerts/servercert.pem
mymail:~ # cp /etc/certs/mymail-anraconc-key.pem /etc/ssl/servercerts/serverkey.pem
mymail:~ # chmod 600 /etc/ssl/servercerts/serverkey.pem 
mymail:~ # 

If you read my second article of this series these steps need no further explanation.

Reconfiguring sssd, cyrus and postfix for the new SSL/TLS certificates

First, the sssd-service configuration in the file "/etc/sssd/sssd.conf" has to be updated on mymail:

.....
[pam]
[domain/default]
ldap_uri = ldap://myserv.anraconc.de
ldap_search_base = dc=anraconc,dc=de
ldap_schema = rfc2307bis
id_provider = ldap
ldap_user_uuid = entryuuid
ldap_group_uuid = entryuuid
ldap_id_use_start_tls = False
enumerate = True
cache_credentials = False
ldap_tls_cacertdir = /etc/ssl/certs
ldap_tls_cacert = /etc/ssl/certs/anraconc-CA.pem
chpass_provider = ldap
auth_provider = ldap
...
....

Note that the ldap_uri is the same (!) as on "myserv"! See the last article!

We then change "/etc/ldap.conf" - this enables any clients :

uri     ldap://myserv.anraconc.de
base    dc=anraconc,dc=de
nss_map_attribute       uniqueMember member
ssl     start_tls
tls_cacertdir   /etc/ssl/certs
tls_cacertfile  /etc/ssl/certs/anraconc-CA.pem              
pam_password    exop
pam_filter      objectClass=posixAccount
...

Afterwards we change the file "/etc/imapd.conf" - which is a central piece for a working cyrus-based IMAP-server:

....
# LMTP
# ****
lmtp_overquota_perm_failure: no
lmtp_downcase_rcpt: yes

# SASL
# ****
allowplaintext: yes
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN LOGIN

# TLS
# ****
tls_key_file: /etc/ssl/servercerts/serverkey.pem
tls_cert_file: /etc/ssl/servercerts/servercert.pem
tls_ca_file: /etc/ssl/certs/anraconc-CA.pem
...
...

You see that we saslauthd as an intermediate link of imap to authenticate (cyrus) users via LDAP (on another server).

Important Note:
The system users "mail" and "cyrus" must get read access to "/etc/ssl/servercerts/serverkey.pem" in case we use exactly one server-wide server certificate/key for all services. Otherwise TLS connections (starttls) cannot be provided for the imap/cyrus-service.

One solution for this problem is to use ACL-settings analogously to what we have done above for user "ldap" on the server "myserv". See my last article ...

For SMTP we also need to change "/etc/postfix/main.cf":

...
....
 Client for the local SMTP-Server
#---------------------------------

# Changed by admin according to local sasl politics
#----------------------------------------------------
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = cyrus
# Postfix > 2.3
smtpd_sasl_path = smtpd
broken_sasl_auth_clients = yes


# Local SMTP server as a client for Relay servers 
#------------------------------------------------
# Inserted by admin - 06.01.2017
#-----------------------------------

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
...
....
############################################################
# TLS stuff
############################################################
relay_clientcerts =
#tls_random_source = dev:/dev/urandom

smtp_use_tls = yes
#smtp_tls_loglevel = 0
#smtp_enforce_tls = no
smtp_tls_policy_maps = hash:/etc/postfix/tls_policy
smtp_tls_CApath = /etc/ssl/certs/
smtp_tls_CAfile = /etc/ssl/certs/anraconc-CA.pem

smtp_tls_session_cache_database =

smtpd_use_tls = yes
smtpd_tls_auth_only = yes
smtpd_tls_loglevel = 2
smtpd_tls_CAfile = /etc/ssl/certs/anraconc-CA.pem
smtpd_tls_cert_file = /etc/ssl/servercerts/servercert.pem
smtpd_tls_key_file = /etc/ssl/servercerts/serverkey.pem
smtpd_tls_ask_ccert = no
smtpd_tls_received_header = yes

...
...

You saw again that I use saslauth for authentication. But, actually there is nothing to be changed in the saslauthd configuration in comparison to a previously working solution.

Finally, we restart all our services:

mymail:/etc/sasl2 # systemctl restart sssd.service
mymail:/etc/sasl2 # systemctl restart cyrus.service
mymail:/etc/sasl2 # systemctl restart postfix.service
mymail:/etc/sasl2 # systemctl restart saslauthd.service 

This should work flawlessly, if we have done everything correctly.

Testing with Kmail and other mail clients from client systems

Before you can start testing with email-clients you need to transfer the new CA's certificate to your client system's directory "/etc/pki/trust/anchors/". Do not forget to import "anraconc-CA.pem" on all other clients that build up TLS or SSL connections to your reconfigured servers, too. And put the CA certificate into "/etc/pki/trust/anchors/" - if your clients are Opensuse Leap 15.x systems.

If you need any direct LDAP client services form your client system to the LDAP service on "myserv" you should take care of the necessary cahnges in the local "/etc/ldap.conf" and the "/etc/sssd/sssd.conf" !

Otherwise you should have a look at Kmail's or your email's IMAP account configuration:

Note that you should give the name of the mail server in its FQDN-form to avoid messages on certificates not fitting the server. And:

Clicking on the button "Automatisch erkennen" (or whatever the button is named in English ..."Automatic detection ?) should offer you STARTTLS now. If you get any messages about "no connection could be enabled" your configuration of the mail server still has deficits - either in offering TLS/SSL encryption or in its role as a client to the OpenLDAP server. You would have to analyze status and log information of all involved servers then.

You have to do similar things on other email-clients. I was lucky - things started to operate smoothly again on my mail clients.

Conclusion

It really is a problem that the SuSE people did not replace the YaST module for the administration of TLS/SSL CAs and server certificates by a Ruby counterpart with the start of Leap 15.0. Especially as Opensuse works as a basic platform for SLES 🙁 . Also SuSE previously often used a policy to allow for server wide certificates - i.e. one certificate for all services on a server instance. The deployment of such certificates issued by a central CA was supported by an export/import mechanism of YaST, which took care of password-protected keys by stripping of password and setting the right read access rights on the destination servers.

If you want to use TinyCA2 as a CA administration tool you are confronted with a series of challenges - among others the question of access rights of system users to certificate's key files. In addition you have to place the CA's root certificate at a proper location on Opensuse Leap systems (/etc/pki/trust/anchors) to make it permanently available.

TinyCA2 - which I had a look at - is relatively old. The last Gitub change appeared in 2015. Nevertheless it does its job to create a CA, sub-CAs and server certificates (as well as client certificates) - and to revoke them if required. It also offers a simple local export mechanism for certifictaes and keys into standard file formats - as pem. However, this is it.

You need a patch to include SHA256 or SHA512 - which is a must these days. There are some flaws with the simple GUI, but these you can work around related problems - if you know about them. And: TinyCA2 will not help you to deploy the certificates/keys on your server or client systems. Not even a simple import mechanism for a server wide "server certificate plus key" issued by the CA is offered on target servers. So, if you do not want to invest time into your own deployment scripts you may come to the conclusion that TinyCA2 is not suited well for large networks with many servers. In my case I fount it applicable for a dozen servers - but not more. Too much handwork ....

If you consider to use TinyCA2 as a replacement for YaST's CA administration tools most of your work will be to deploy the TLS/SSL certificates/keys onto your server systems and to reconfigure all services to use these certificates for TLS, StartTLS and SSL connections.

On Opensuse systems the deployment of the CA root certificate within the bunch of permanently trusted CA certificates requires some knowledge about the relevant directory to make the CA certificate available in a central directory as "/etc/ssl/certs" - which may be used by legacy applications. The access rights to a server certificates key file may pose some special problems - you may have to test which system user associated with your services requires read rights.

There are actually several ways how to handle the read access topic for the private certificate keys. I have shown how to use ACLs for this purpose. My recommendation, however, is to use separate certificates and related keys for all services - even if they reside on the same server and despite consequences for your network and DNS configuration due to unique FQGNs and common names. You never know what can happen to a service specific user during attacks. If one certificate/key gets corrupted the others may still survive. Actually, this could be a reason for encapsulating services even more in form of LXC containers. But this is a different story ...

Good luck with Opensuse and TinyCA2 !

 

Mail-server-upgrade to Opensuse Leap 15 – and some hours with authentication trouble …

Some of my readers know that the mail server in my private LAN resides in an encrypted KVM/qemu guest. It provides IMAP- and SMTP-services to mail-clients - and receives emails from several hosted servers on the Internet via a fetchmail system in a DMZ. It was/is my policy that any access to a mail service account on my private mail server requires authentication against user entries on a separate LDAP-server. The accounts the IMAP/SMTP-service-daemons deal with are not to be confused with Linux user accounts: On the mail server NO Linux user accounts are required for mail handling. I regard the absence of standard Linux user accounts on the mail-server as a small, but effective contribution to security. No mail user can login into the mail server as a Linux user. Nevertheless authentication for the use of mail services is required.

Upgrade top Leap 15 resulted in unusable mail services

The manual upgrade of the mail server system from OS Leap 42.3 to Leap 15 seemed to work perfectly. I got no serious warnings. As root I could login to the virtual KVM guest via ssh - and systemd had started all required services: Fetchmail operated on external servers in the DMZ and transferred new mails to the upgraded mail server. There my postfix queues and secondary services for virus and spam checks seemed to work perfectly. Regarding IMAP I could also see that new entries for new mails appeared in various email accounts (more precisely: in related spool directories). Everything looked great ...

However, the big surprise came pretty soon: No mail user could login to the IMAP-service - neither with Kmail, nor Mutt, nor Thunderbird. Access to the SMTP service for sending email was not possible either. System messages appeared on the GUIs saying that there was an authentication error. A very unpleasant situation which required analysis ....

In the meantime I had to start a backup copy of the old mailserver guest installation. On this front
virtualization proved its strengths and simplicity - I had made a copy of the whole KVM guest before the upgrade and just had to include the copy as a virtualization domain into the KVM setup of the virtualization host. But, as the upgraded server had already processed some new mails, I needed to transfer these mails between the servers and reconstruct and reindex some IMAP accounts ... A simple, but a task which can be done by some scripts ....

Error analysis

It took me quite a while to figure out where the origin of the upgrade problem was located. I first checked my local firewall protocols both on the mail server and on the LDAP server. Nothing special there ... Then I checked the LDAP access protocols - there were successful data requests from various systems - but astonishingly not from the mail server. So it seemed that the mail services did not even try to authenticate their users ... strange ....
I then created a new IMAP test user based on a new local Linux user account, i.e. with an entry in "/etc/passwd" and "/etc/shadow". Guess what? This user could work without any problems. So, obviously the communication of the mail services with the LDAP service for authentication was not triggered or failed in a strange way. I, therefore, tried a manual data request to the LDAP server from the mail server - and got a perfect answer. So, there seemed to be no fundamental problem with the required sssd-daemon and client-configurations. What else was wrong? A bit of despair was in the air ...

The role of PAM in the game

Then I tried to remember what I had done in the past to separate the email accounts from the Linux user accounts. Over the last years I had actually forgotten how I had treated this problem. After some reading in old diaries my attention eventually turned to PAM ...

The PAM-layer offers Linux admins the possibility of fine-tuning access/authentication conditions for the usage of a service; among other things you can request certain PAM-modules to authenticate a user via given credentials against defined resources. A sequential series of required, sufficient or optional criteria can be set up. ( When a service needs additional Linux user account information during a session NSS can in addition request information from name resolution backends as "/etc/passwd". However, this is NOT required for mail services. )

Linux mail services often use the SASL-framework and the saslauthd daemon to perform authentication. SASL can communicate with a variety of authentication backends directly - but it can also use PAM as an intermediate layer. And, in fact, I had used PAM to access my independent LDAP-server with mail account credentials via the PAM-module for SSSD.

I have described my simple approach in a previous article
https://linux-blog.anracom.com/2014/03/15/cyrus-imap-mit-sasl-pam-sssd-und-ldap-opensuse-12-313-1-ii/
Unfortunately, only in German. I, therefore, summarize the basic ideas shortly (see also the graphics in the referenced article).

The things one has to do to force IMAP and/or a SMTP services to use LDAP, but other services to use local authentication resources is:

  • We direct local login-services, ssh-services, etc. on the server via PAM to local resources, only.
  • We eliminate any UIDs corresponding to email account names from local resources as "/etc/passwd" and "/etc/shadow" (and NIS if used).
  • We do not create any local home-directories for email account names.
  • We eliminate LDAP as a usable NSS resource from nsswitch.conf; i.e. we eliminate all LDAP references there.
  • We setup special PAM files in "/etc/pam.d" for the IMAP-service and the SMTP-service. These files will necessarily include pam-modules for the "sss"-service (pam_sss.so).
  • We select entries on the LDAP server for users which for any reasons shall NOT or NO LONGER get access to the mail service accounts, set the "host"-attribute for these entries and configure sssd to use the host-attribute. Or deny certain user names access to the IMAP-service by the help of IMAP -configuration files; cyrus e.g offers the maintenance of a list "userdeny_db".

This strategy worked very conveniently already on an Opensuse 12.3 platform. Examples for the required special files "/etc/pam.d/imap" and "/etc/pam.d/smtp" are given in the article named above. The mix and sequence of the statements there is due to the fact that locally defined system users as "cyrus" must get access to the IMAP-service, too.

What had happened during the Leap 15 upgrade?

My configuration had survived multiple upgrades of the virtualized server in the past. But not the one to Leap 15! What had happened?

After having remembered the importance of the PAM configuration, I eventually had a look into the directory "/etc/pam.d" on the upgraded system and compared it to the respective directory on the original system. Whereas during previous upgrades "personal" PAM configuration files for services had been respected, the upgrade to Leap 15 simply had removed my PAM configuration files for SMTP and IMAP - without a backup! Incredible, but true! Actually and astonishingly, the contents of some other standard PAM files had been transferred to a backup ....

A restoration of my PAM files for the mail services lead to an immediate success - authentication for the access of mail accounts on the upgraded server was possible again. Some hours in the hell for Linux admins came to an end.

Conclusion

Never (!) trust a standard upgrade of a whole Linux system procedure to keep configuration files in "/etc/pam.d" untouched. Keep an admin or system diary for unusual approaches. And - of course: Full backups of critical systems before backups are a MUST ....