The moons dataset and decision surface graphics in a Jupyter environment – V – a class for plots and some experiments

We proceed with our exercises on the moons dataset. This series of articles is intended for readers which - as me - are relatively new both to Python and Machine Learning. By working with examples we try to extend our knowledge about the tools "Juypter notebooks" and "Eclipse/PyDev" for setting up experiments which require plots for an interpretation.

We have so far used a Jupyter notebook to perform some initial experiments for creating and displaying a decision surface between the moons dataset clusters with an algorithm called "LinearSVC". If you followed everything I described in the last articles

The moons dataset and decision surface graphics in a Jupyter environment – I
The moons dataset and decision surface graphics in a Jupyter environment – II – contourplots
The moons dataset and decision surface graphics in a Jupyter environment – III – Scatter-plots and LinearSVC
The moons dataset and decision surface graphics in a Jupyter environment – IV – plotting the decision surface

you may now have gathered around 20 different cells with code. Part of the cells' code was used to learn some basics about contour and scatter plots. This code is now irrelevant for further experiments. Time to consolidate our plotting knowledge.

In the last article I promised to put plot-related code into a Python class. The class itself can become a part of a Python module - which we in turn can import into the code of Jupyter notebook. By doing this we can reduce the number of cells in a notebook drastically. The importing of external classes is thus helpful for concentrating on "real" data analysis experiments with different learning and predicting algorithms and/or a variation of their parameters.

I assume that you have some basic knowledge on how classes are build in Python. If not please see an introductory book on Python 3.

A class for plotting simple decision surfaces in a 2-dimensional space

In the articles

Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – I
Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – II
Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – III

I had shown how to set up Eclipse PyDev to be used in the context of a Python virtual environment. In our special environment "ml1" used by our Jupyter notebook "moons1.ipynb" we have the following directory structure:

"ml1" has a sub-directory "mynotebooks" which contains notebook files as our "moons1.ipynb". To provide a place for other general code there we open up a directory "mycode". In it we create a file "myplots.py" for a module "myplots", which shall comprise our own Python classes for plotting.

We distribute the code discussed in the last 2 articles of this series into methods of a class "MyDecisionPlot"; we put the following code into our file "myplots.py" with the Pydev editor.

'''
Created on 15.07.2019
Module to gather classes for plotting
@author: rmo
'''
import numpy as np
import sys
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
import matplotlib.patches as mpat 
#from matplotlib import ticker, cm
#from mpl_toolkits import mplot3d

class MyDecisionPlot():
    '''
    This class allows for 
    1) decision surfaces in 2D (x1,x2) planes 
    2) plotting scatter data of datapoints 
    '''


    def __init__(self, X, y, predictor = None, ax_x_delta=1.0, ax_y_delta=1.0, 
                 mesh_res=0.01, alpha=0.4, bcontour=1, bscatter=1, 
                 figs_x1=12.0, figs_x2=8.0, 
                 x1_lbl='x1', x2_lbl='x2',   
                 legend_loc='upper right'
                 ):
        '''
        Constructor of MyDecisionPlot
        Input: 
            X: Input array (2D) for learning- and predictor-algorithm as VSM 
            y: result data for learning- and predictor-algorithm
            ax_x_delta, ax_y_delta : delta for extension of both axis beyond the given X, y-data
            mesh_res: resolution of the mesh spanned in the (x1,x2)-plane (x_max-x_min) * mesh_res
            alpha:  transparency  of contours 
            bcontour: 0: Do not plot contour areas 1: plot contour areas 
            bscatter: 0: Do not plot scatter points of the input data sample 1: Plot scatter plot of the input data sample 
            figs_x1: plot size in x1 direction 
            figs_x2: plot size in x2 direction 
            x1_lbl, x2_lbl : axes lables 
            legend_loc : position of a legend
        Ouptut:
            Internal: self._mesh_points (mesh points created) 
            External: Plots - shoukd cone up automatically in Jupyter notebooks 
        '''
        
                
        # initiate some internal variables 
        self._x1_min = 0.0
        self._x1_max = 1.0
        self._x2_min = 0.0
        self._x2_max = 1.0
        
        # Alternatives to resize plots 
        # 1: just resize figure  2: resize plus create subplots() [figure + axes] 
        self._plot_resize_alternative = 2 
                
        # X (x1,x2)-Input array 
        self.__X = X
        self.__y = y 
        self._Z  = None
        
        # predictor = algorithm to create y-values for new (x1,x2)-data points 
        self._predictor = predictor 
        
        # meshdata
        self._resolution = mesh_res   # resolution of the mesh 
        self.__ax_x_delta = ax_x_delta 
        self.__ax_y_delta = ax_y_delta
        self._alpha = alpha
        self._bscatter = bscatter
        self._bcontour = bcontour 
        
        self._xm1 = None
        self._xm2 = None
        self._mesh_points = None 
        
        # set marker array and define colormap 
        self._markers = ('s', 'x', 'o', '^', 'v')
        self._colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
        self._cmap = ListedColormap(self._colors[:len(np.unique(y))])
        
        self._x1_lbl = x1_lbl
        self._x2_lbl = x2_lbl
        self._legend_loc = legend_loc
        
        # Plot-sizing
        self._figs_x1 = figs_x1
        self._figs_x2 = figs_x2
        self._fig = None
        self._ax  = None 
                # alternative 2 does resizing and (!) subplots() 
        self.initiate_and_resize_plot(self._plot_resize_alternative) 
               
        
        # create mesh in x1, x2 - direction with mesh_res resolution 
        # meshpoint-array will be creted with right dimension for plotting data 
        self.create_mesh()
        # Array meshpoints should exist now 
        
        if(self._bcontour == 1):
            try:
                if self._predictor == None:
                    raise ValueError
            except ValueError:
                print ("You must provide an algorithm = 'predictor' as parameter")
                #sys.exit(0)   
                sys.exit()   
            
            self.make_contourplot()
        else:
            if (self._bscatter == 1):
                self.make_scatter_plot() 


    # method to create a dense mesh in the (x1,x2)-plane 
    def create_mesh(self):
        '''
        Method to create a dense mesh in an (x1,x2) plane 
        Input: x1, x2-data are constructed from array self.__X
        Output: A suitable array of meshpoints is written to self._mesh_points()
        '''
        try:
            self._x1_min = self.__X[:, 0].min()
            self._x1_max = self.__X[:, 0].max()
        except ValueError: # as e: 
            print ("cannot determine x1_min = X[:,0].min() or x1_max = X[:,0].max()")
        try:
            self._x2_min = self.__X[:, 1].min()
            self._x2_max = self.__X[:, 1].max()
        except ValueError: # as e: 
            print ("cannot determine x2_min =X[:,1].min()) or x2_max = X[:,1].max()")
            
        self._x1_min, self._x1_max = self._x1_min - self.__ax_x_delta, self._x1_max + self.__ax_x_delta
        self._x2_min, self._x2_max = self._x2_min - self.__ax_x_delta, self._x2_max + self.__ax_x_delta
        
        #create mesh data (x1, x2) 
        self._xm1, self._xm2 = np.meshgrid( np.arange(self._x1_min, self._x1_max, self._resolution), 
                                            np.arange(self._x2_min, self._x2_max, self._resolution))
        
        #print (self._xm1)
        # for hasattr the variable cannot be provate ! 
        #print ("xm1 is there: ", hasattr(self,'_xm1' ) )
                          
        # ordering and transposing of the mesh-matrix   
        # for understanding the structure and transpose requirements see linux-blog.anracom.con          
        self._mesh_points = np.array([self._xm1.ravel(), self._xm2.ravel()]).T
        
        try:
            if( hasattr(self, '_mesh_points') == False ):
                raise ValueError
        except ValueError:
            print("The required array mesh_points has not been created!")
            exit

    # -------------
    # Some helper functions to change valus on the fly if necessary 
              
    def set_mesh_res(self, new_mesh_res): 
        self._resolution = new_mesh_res
        
    def change_predictor(self, new_predictor):
        self._predictor = new_predictor
        
    def change_alpha(self, new_alpha):    
        self._alpha = new_alpha
    
    def change_figs(self, new_figs_x1, new_figs_x2):    
        self._figs_x1 = new_figs_x1
        self._figs_x2 = new_figs_x2
    
    
    # -------------
    # method to get subplots and resize the figure       
    # -------------
    def initiate_and_resize_plot(self, alternative=2 ):
        
        # Alternative 1 to resize plots - works afte rimports to Jupyter notebooks, too
        if alternative == 1:
            self._fig_size = plt.rcParams["figure.figsize"]
            self._fig_size[0] = self._figs_x1
            self._fig_size[1] = self._figs_x2
            plt.rcParams["figure.figsize"] = self._fig_size
        
        # Not working for sizing if plain subplots() is used 
        #plt.figure(figsize=(self._figs_x1 , self._figs_x2))
        #self._fig, self._ax = plt.subplots()
        # instead you have to put the figsize-parameter into the subplots() function 
        
        # Alternative 2 for resizing plots and using subplots()
        # we use this alternative as we may need the axes later for 3D plots 
        if alternative == 2:
            self._fig, self._ax = plt.subplots(figsize=(self._figs_x1 , self._figs_x2))
    
    
    # ***********************************************
    
    # -------------
    # method to create contour plots      
    # -------------
    def make_contourplot(self):
        '''
        Method to create a contourplot based on a dense mesh of points in an (x1,x2) plane 
        and a predictor algorithm which allows for value calculations
        '''
        
        try:
            if( not hasattr(self, '_mesh_points') ):
                raise ValueError
        except ValueError:
            print("The required array mesh_points has not been created!")
            exit
        
        # Predict values for all meshpoints 
        try:
            self._Z = self._predictor.predict(self._mesh_points)
        except AttributeError: 
            print("method predictor.predict() does not exist") 
        
        #reshape     
        self._Z = self._Z.reshape(self._xm1.shape)
        #print (self._Z)
        
        # make the plot
        plt.contourf(self._xm1, self._xm2, self._Z, alpha=self._alpha, cmap=self._cmap)     
        
        # create a scatter-plot of data sample as well 
        if (self._bscatter == 1):
            self.make_scatter_plot()
            
        self.make_plot_legend()
        
           
                 
    # -------------
    # method to create a scatter plot of the data sample 
    # -------------
    def make_scatter_plot(self):
        alpha2 = self._alpha + 0.4 
        if (alpha2 > 1.0 ):
            alpha2 = 1.0
        for idx, yv in enumerate(np.unique(self.__y)): 
            plt.scatter(x=self.__X[self.__y==yv, 0], y=self.__X[self.__y==yv, 1], 
                        alpha=alpha2, c=[self._cmap(idx)], marker=self._markers[idx], label=yv)
        
        if self._bscatter == 0:
            self._bscatter = 1
        
        self.make_plot_legend()
          
        
    # -------------
    # method to add a legend  
    # -------------
    def make_plot_legend(self):
        plt.xlim(self._x1_min, self._x1_max)
        plt.ylim(self._x2_min, self._x2_max)
        plt.xlabel(self._x1_lbl)
        plt.ylabel(self._x2_lbl)
        
        # we have two cases 
        #     a) for a scatter plot we have array values where the legend is taken from automatically
        #     b) For apure contourplot we need to prepare a legend with "patches" (kind og labels) used by pyplot.legend() 
        if (self._bscatter == 1):
            plt.legend(loc=self._legend_loc)
        else:
            red_patch  = mpat.Patch(color='red',  label='0', alpha=0.4)
            blue_patch = mpat.Patch(color='blue', label='1', alpha=0.4)
            plt.legend(handles=[red_patch, blue_patch], loc=self._legend_loc)

    

 
This certainly is no masterpiece of superior code design; so you may change it. However, the code is good enough for our present purposes.

Note that we have to import basic Python modules into the namespace of this module. This is conventionally done at the top of the file.

Note also the 2 alternatives offered for resizing a plot! Both work also for "inline" plotting in a Jupyter environment; see the text below.

Using the module "myplots" in a Jupyter notebook

In a terminal we move to our example directory "/projekte/GIT/ai/ml1" and start our virtual Python environment:

myself@mytux: /projekte/GIT/ai/ml1> source bin/activate    
(ml1) myself@mytux:/projekte/GIT/ai/ml1> jupyter notebook
[I 11:46:14.687 NotebookApp] Writing notebook server cookie secret to /run/user/1999/jupyter/notebook_cookie_secret
[I 11:46:15.942 NotebookApp] Serving notebooks from local directory: /projekte/GIT/ai/ml1
[I 11:46:15.942 NotebookApp] The Jupyter Notebook is running at:
....
....

We then open our old notebook "moons1" and save it under the name "moons2":

We delete all old cells. Then we change the first cell of our new notebook to the following contents:

import imp
%matplotlib inline
from mycode import myplots

from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import PolynomialFeatures
from sklearn.svm import LinearSVC
from sklearn.svm import SVC

You see that we imported the "myplots" module from the "package" directory "mycode". The Jupyter notebook will find the path to "mycode" as long as we have opened the notebook on a higher directory level. Which we did ... see above.

Note the statement with a so called "magic command" for our notebook:

%matplotlib inline

There are many other "magic commands" and parameters which you can study at
Ipython magic commands

The command "%matplotlib inline" informs the notebook to show plots created by any imported modules "inline", i.e. in the visual context of the affected cell. This specific magic directive should be issued before matplotlib/pyplot is imported into any of the external python modules which we in turn import into the cell code.

A call of plt.show() in our class's method "make_contourplot()" is no longer necessary afterwards.

If we, however, want to resize the plots in comparison to Jupyter standard values we have to control this by parameters of our plot class. Such parameters are offered already in the interface of the class; but they can be changed by a method "change_figs(new_figs_x1, new_figs_x2)), too.

In a second cell of our new notebook we now prepare the moon data set

X, y = make_moons(noise=0.1, random_state=5)

Further cells will be used for quick individual experiments with the moons dataset. E.g.:

imp.reload(myplots)
X, y = make_moons(noise=0.1, random_state=5)

# contour plot of the moons data - with scatter plot / training with LinearSVC 
polynomial_degree = 3
max_iterations = 6000
polynomial_svm_clf = Pipeline([
  ("poly_features", PolynomialFeatures(degree=polynomial_degree)),
  ("scaler", StandardScaler()),
  ("svm_clf", LinearSVC(C=18, loss="hinge", max_iter=max_iterations))
])
#training
polynomial_svm_clf.fit(X, y)

#plotting 
MyPlt = myplots.MyDecisionPlot(X, y, polynomial_svm_clf, bcontour = 1, bscatter=1 )

The last type of cell just handles the setup and training of our specific algorithm "LinearSVC" and delegates plotting to our class.

Testing the new notebook

A test of the 3 cells in their order gives

All well! This is exactly what we hoped to achieve.

Three experiments with a varying polynomial degree

As we now have a simple and compact cell template for experiments we add three further cells where we vary the degree of the polynomials for LinearSVC. Below we show the results for degree 6, 7 and for comparison also for a degree of 2.

On a modern computer it should take almost no time to produce the results. (We shall learn how to measure CPU-time in the next article).

We understand that we at least need a polynomial of degree 3 to separate the data reasonably. However, polynomials with an even degree (>= 4) separate the 2 data regions very differently compared to polynomials with an uneven degree (>=3) in outer areas of the (x1,x2)-plane (where no training data were placed):

Depending on the polynomial degree our Linear SVC algorithm obviously extrapolates in different ways to regions without such data. And we have no clue which of the polynomials is the better choice ...

This poses a warning for the use of AI in general:

We should be extremely careful to trust predictions of any AI algorithm in parameter regions for which the algorithm must extrapolate - as long as we have no real data points available there to discriminate between multiple solutions that all work equally well in regions with given training data.

Would general modules be imported twice in a Jupyter cell - via the import of an external module, which itself includes import statements, and a direct import statement in a Jupyter cell?

The question posed in the headline is an interesting one for me as a Python beginner. Coming from other programming languages I get a bit nervous when I see the possibility for import statements referring to a specific module both in another already imported module and by a direct import statement afterwards in a Jupyter cell. E.g. we import numpy indirectly via our "myplots" module, but we could and sometimes must import it in addition directly in our Jupyter cell.

Note that we must make the general modules as numpy, matplotlib, etc. available in the namespace environment of our private module "myplots". Otherwise the functions cannot be used there. The Jupyter cell, however, corresponds to an independent namespace environment. So, an import may indeed be required there, too, if we plan to handle numpy arrays via code in such a cell.

Reading a bit about the Python import logic on the Internet reveals that a double import or overwriting should not take place; instead an already imported piece of code only gets multiple references in the various specific namespaces of different code environments.

We can test this with the following code in a Jupyter cell:

Note that numpy is also imported by our "myplots". However, the length of the list produced by sys.modules.keys(), which enumerates all possible module reference points (including imports) does not change.

Reloading modules in Jupyter cells

What if we in the course of or experiments need to change the code of our imported module? Then we need to reload the module in a Jupyter cell before we run it again. In our case (Python 3!) this can be done by the command

imp.reload(myplots)

As the code of our first cell reveals, the general package "imp" must have been imported before we can use its reload-function.

Conclusion

We saw that it is easy to use our own modules with Python class code, which we created in an Eclipse/PyDev environment, in a Jupyter notebook. We just use Python's standard import mechanism in Jupyter cells to get access to our classes. We applied this to a module with a simple class for preparing decision surface plots based on contour and scatter plot routines of matplotlib. We imported the module in a new Jupyter notebook.

Plots created by our imported class-methods were displayed correctly within the cell environment as soon as we used the magic directive "%matplotlib inline" within our notebook.

In addition we used our new notebook with its compact cell structure for some quick experiments: We set different values for the polynomial degree parameter of our LinearSVC algorithm. We saw that the results of algorithms should be interpreted with caution if the predictions refer to data points in regions of the representation or feature space which were not at all covered by the data sets for training.

The prediction results (= extrapolations) of even one and the same algorithm with different parameters may deviate strongly in such regions - and we may not have any reliable indications to decide which of the predictions and their basic parameter setting are better.

In the next article of this series

The moons dataset and decision surface graphics in a Jupyter environment – VI – Kernel-based SVC algorithms

we shall have a look at what kind of decision surface some other SVM algorithms than LinearSVC create for the moons dataset. In addition shall briefly discuss kernel based algorithms.

 

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – III

I proceed with my article series on using TinyCA2 instead of YaST's CA-admin tools, whic are simply missing since Opensuse Leap 15.0. Which I regard as a shame by the way ...

In the first article

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – I

I briefly introduced the user interface of TinyCA2. We used it to create a new CA and a server certificate.

In the second article

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – II

we found out where to place the new CA's root certificate on an Opensuse Leap system to make it available system wide and permanently. We also used the newly created server certificate for the server "myserv.anraconc.de" for all services on this server - and placed the necessary files into "/etc/ssl/servercerts".

We then discussed the reconfiguration of an Apache2-, an OpenLDAP- and the SSSD-service as examples of how to adapt to the CA and certificate changes. All the required operations on "myserv" were local copy and file editing operations - as "myserv" was the server for administering our new CA, too.

In this third article we, however, look at a second independent server "mymail.anraconc.de" for IMAP/SMTP-services. This server needs its own TLS/SSL-server-certificate. In lack of other tools we shall use scp to deploy the necessary files. I shall further discuss how to reconfigure "cyrus" and "postfix". Note that these services do not only offer their own TLS/SSL connections to their clients; at the same time they themselves will work as LDAP-clients, i.e. as clients for the OpenLDAP service on server "myserv". Therefore we must adapt the SSSD-service on "mymail", too.

Placing a TinyCA2 server certificate on a mail server with cyrus, saslautd and postfix

Our first step is to use TinyCA2 on our server "myserv" to create a second server certificate for the server "mymail" with the common name "mymail.anraconc.de". This is done exactly as described in my first article for server "myserv". We export the related certificate and the key files to "/etc/certs" on "myserv" and limit access to the private key file. We export the key file with its password - and strip it off afterwards:

myserv:/etc/certs # openssl rsa -in /etc/certs/mymail-anraconc-key.pem -out /etc/certs/mymail-anraconc-key_new.pem
Enter pass phrase for /etc/certs/mymail-anraconc-key.pem:
writing RSA key
myserv:/etc/certs # cp /etc/certs/mymail-anraconc-key_new.pem /etc/certs/mymail-anraconc-key.pem

Now, we look for some export/import mechanism for deploying "server certificates" from our sytem with the CA administration tool TinyCA2 to other systems. Unfortunately, TinyCA2 does not offer such a client/server feature for certificate deployment within a network. We are forced to transfer our certificates manually 🙁 .

I assume that we have a ssh-connection form our sever "myserv" to our mail-server "mymail".

myserv:/etc # scp /etc/certs/mymail-anraconc-cert.pem root@mymail:/etc/certs
Password: 
mymail-anraconc-cert.pem                   100% 2147     1.5MB/s   00:00    
myserv:/etc # scp /etc/certs/mymail-anraconc-key.pem root@mymail:/etc/certs
Password: 
mymail-anraconc-key.pem                    100% 1679     1.4MB/s   00:00    
myserv:/etc # scp /etc/certs/anraconc-CA.pem root@mymail:/etc/certs
Password: 
anraconc-CA.pem                           100% 2504     1.7MB/s   00:00    
myserv:/etc # 

Note that I transferred the CA certificate, too.

On "mymail":

mymail:~ # cp /etc/certs/anraconc-CA.pem /etc/pki/trust/anchors/
mymail:~ # cp -r /etc/ssl/servercerts/ /etc/ssl/servercerts.orig
mymail:~ # cp /etc/certs/mymail-anraconc-cert.pem /etc/ssl/servercerts/servercert.pem
mymail:~ # cp /etc/certs/mymail-anraconc-key.pem /etc/ssl/servercerts/serverkey.pem
mymail:~ # chmod 600 /etc/ssl/servercerts/serverkey.pem 
mymail:~ # 

If you read my second article of this series these steps need no further explanation.

Reconfiguring sssd, cyrus and postfix for the new SSL/TLS certificates

First, the sssd-service configuration in the file "/etc/sssd/sssd.conf" has to be updated on mymail:

.....
[pam]
[domain/default]
ldap_uri = ldap://myserv.anraconc.de
ldap_search_base = dc=anraconc,dc=de
ldap_schema = rfc2307bis
id_provider = ldap
ldap_user_uuid = entryuuid
ldap_group_uuid = entryuuid
ldap_id_use_start_tls = False
enumerate = True
cache_credentials = False
ldap_tls_cacertdir = /etc/ssl/certs
ldap_tls_cacert = /etc/ssl/certs/anraconc-CA.pem
chpass_provider = ldap
auth_provider = ldap
...
....

Note that the ldap_uri is the same (!) as on "myserv"! See the last article!

We then change "/etc/ldap.conf" - this enables any clients :

uri     ldap://myserv.anraconc.de
base    dc=anraconc,dc=de
nss_map_attribute       uniqueMember member
ssl     start_tls
tls_cacertdir   /etc/ssl/certs
tls_cacertfile  /etc/ssl/certs/anraconc-CA.pem              
pam_password    exop
pam_filter      objectClass=posixAccount
...

Afterwards we change the file "/etc/imapd.conf" - which is a central piece for a working cyrus-based IMAP-server:

....
# LMTP
# ****
lmtp_overquota_perm_failure: no
lmtp_downcase_rcpt: yes

# SASL
# ****
allowplaintext: yes
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN LOGIN

# TLS
# ****
tls_key_file: /etc/ssl/servercerts/serverkey.pem
tls_cert_file: /etc/ssl/servercerts/servercert.pem
tls_ca_file: /etc/ssl/certs/anraconc-CA.pem
...
...

You see that we saslauthd as an intermediate link of imap to authenticate (cyrus) users via LDAP (on another server).

Important Note:
The system users "mail" and "cyrus" must get read access to "/etc/ssl/servercerts/serverkey.pem" in case we use exactly one server-wide server certificate/key for all services. Otherwise TLS connections (starttls) cannot be provided for the imap/cyrus-service.

One solution for this problem is to use ACL-settings analogously to what we have done above for user "ldap" on the server "myserv". See my last article ...

For SMTP we also need to change "/etc/postfix/main.cf":

...
....
 Client for the local SMTP-Server
#---------------------------------

# Changed by admin according to local sasl politics
#----------------------------------------------------
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = cyrus
# Postfix > 2.3
smtpd_sasl_path = smtpd
broken_sasl_auth_clients = yes


# Local SMTP server as a client for Relay servers 
#------------------------------------------------
# Inserted by admin - 06.01.2017
#-----------------------------------

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
...
....
############################################################
# TLS stuff
############################################################
relay_clientcerts =
#tls_random_source = dev:/dev/urandom

smtp_use_tls = yes
#smtp_tls_loglevel = 0
#smtp_enforce_tls = no
smtp_tls_policy_maps = hash:/etc/postfix/tls_policy
smtp_tls_CApath = /etc/ssl/certs/
smtp_tls_CAfile = /etc/ssl/certs/anraconc-CA.pem

smtp_tls_session_cache_database =

smtpd_use_tls = yes
smtpd_tls_auth_only = yes
smtpd_tls_loglevel = 2
smtpd_tls_CAfile = /etc/ssl/certs/anraconc-CA.pem
smtpd_tls_cert_file = /etc/ssl/servercerts/servercert.pem
smtpd_tls_key_file = /etc/ssl/servercerts/serverkey.pem
smtpd_tls_ask_ccert = no
smtpd_tls_received_header = yes

...
...

You saw again that I use saslauth for authentication. But, actually there is nothing to be changed in the saslauthd configuration in comparison to a previously working solution.

Finally, we restart all our services:

mymail:/etc/sasl2 # systemctl restart sssd.service
mymail:/etc/sasl2 # systemctl restart cyrus.service
mymail:/etc/sasl2 # systemctl restart postfix.service
mymail:/etc/sasl2 # systemctl restart saslauthd.service 

This should work flawlessly, if we have done everything correctly.

Testing with Kmail and other mail clients from client systems

Before you can start testing with email-clients you need to transfer the new CA's certificate to your client system's directory "/etc/pki/trust/anchors/". Do not forget to import "anraconc-CA.pem" on all other clients that build up TLS or SSL connections to your reconfigured servers, too. And put the CA certificate into "/etc/pki/trust/anchors/" - if your clients are Opensuse Leap 15.x systems.

If you need any direct LDAP client services form your client system to the LDAP service on "myserv" you should take care of the necessary cahnges in the local "/etc/ldap.conf" and the "/etc/sssd/sssd.conf" !

Otherwise you should have a look at Kmail's or your email's IMAP account configuration:

Note that you should give the name of the mail server in its FQDN-form to avoid messages on certificates not fitting the server. And:

Clicking on the button "Automatisch erkennen" (or whatever the button is named in English ..."Automatic detection ?) should offer you STARTTLS now. If you get any messages about "no connection could be enabled" your configuration of the mail server still has deficits - either in offering TLS/SSL encryption or in its role as a client to the OpenLDAP server. You would have to analyze status and log information of all involved servers then.

You have to do similar things on other email-clients. I was lucky - things started to operate smoothly again on my mail clients.

Conclusion

It really is a problem that the SuSE people did not replace the YaST module for the administration of TLS/SSL CAs and server certificates by a Ruby counterpart with the start of Leap 15.0. Especially as Opensuse works as a basic platform for SLES 🙁 . Also SuSE previously often used a policy to allow for server wide certificates - i.e. one certificate for all services on a server instance. The deployment of such certificates issued by a central CA was supported by an export/import mechanism of YaST, which took care of password-protected keys by stripping of password and setting the right read access rights on the destination servers.

If you want to use TinyCA2 as a CA administration tool you are confronted with a series of challenges - among others the question of access rights of system users to certificate's key files. In addition you have to place the CA's root certificate at a proper location on Opensuse Leap systems (/etc/pki/trust/anchors) to make it permanently available.

TinyCA2 - which I had a look at - is relatively old. The last Gitub change appeared in 2015. Nevertheless it does its job to create a CA, sub-CAs and server certificates (as well as client certificates) - and to revoke them if required. It also offers a simple local export mechanism for certifictaes and keys into standard file formats - as pem. However, this is it.

You need a patch to include SHA256 or SHA512 - which is a must these days. There are some flaws with the simple GUI, but these you can work around related problems - if you know about them. And: TinyCA2 will not help you to deploy the certificates/keys on your server or client systems. Not even a simple import mechanism for a server wide "server certificate plus key" issued by the CA is offered on target servers. So, if you do not want to invest time into your own deployment scripts you may come to the conclusion that TinyCA2 is not suited well for large networks with many servers. In my case I fount it applicable for a dozen servers - but not more. Too much handwork ....

If you consider to use TinyCA2 as a replacement for YaST's CA administration tools most of your work will be to deploy the TLS/SSL certificates/keys onto your server systems and to reconfigure all services to use these certificates for TLS, StartTLS and SSL connections.

On Opensuse systems the deployment of the CA root certificate within the bunch of permanently trusted CA certificates requires some knowledge about the relevant directory to make the CA certificate available in a central directory as "/etc/ssl/certs" - which may be used by legacy applications. The access rights to a server certificates key file may pose some special problems - you may have to test which system user associated with your services requires read rights.

There are actually several ways how to handle the read access topic for the private certificate keys. I have shown how to use ACLs for this purpose. My recommendation, however, is to use separate certificates and related keys for all services - even if they reside on the same server and despite consequences for your network and DNS configuration due to unique FQGNs and common names. You never know what can happen to a service specific user during attacks. If one certificate/key gets corrupted the others may still survive. Actually, this could be a reason for encapsulating services even more in form of LXC containers. But this is a different story ...

Good luck with Opensuse and TinyCA2 !

 

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – II

In my last article on TinyCA2

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – I

I discussed how to open an new CA and how to create a server certificate (plus private key) with SHA256 as the preferred hashing algorithm. We saw that TinyCA2 offers an option to export the certificate's private key with or without password. I also gave an "openssl" command for stripping a key off its password later on - even if it was exported with a password in the beginning.

We have followed a policy of using a server wide TLS/SSL-certificate for all TLS/SSL-based services run on a server - in our case on a server "myserv.anraconc.de". Therefore we have copied the certificate's pem file and the key's pem file to the directory "/etc/ssl/servercerts" with file names "servercert.pem" and "serverkey.pem".

In addition we naively copied the CA certificate as file "anraconc-CA.pem" into the central directory "/etc/ssl/certs" which contains (links to) all trusted certificates on an Opensuse system.

If the private key was exported with a password we have a problem with automated system and service starts: As the private key of a server certificate must in some cases also by system users starting and running services we would be asked to provide a password interactively, whenever the service is started - e.g. during system startup. To avoid such a situation we need a private key without password - hence we have to deal carefully with read access rights. Actually, we need to find out which services - not started or run by root - need the access right for its special system user. For the time being we took away all read rights for "others" from the server key for the system "myserv".

In this second article I discuss the changes to be made to the configuration of two services - Apache2 and OpenLDAP on server "myserv". After a test we shall detect that Opensuse rebuilds the contents of "/etc/ssl/certs" during a reboot - and references of our services to the CA's certificate will fail. I shall show where to save our CA certificate to make it permanently available.

Re-configuring a local Apache service for the new CA and the changed server certificate

As I had used a server wide certificate on "myserv" already at times when YaST's CA-administration was still available, I expected that the required changes to the service configuration would be limited to providing the path to the new CA certificate. The example of the Apache service proved that this really was true.

In my case I use virtual domains with my Apache server for Intranet sites. The configuration of the SSL-setup then is to be found in either a central file as "/etc/apache2/ssl-global.conf" or in "/etc/apache2/vhosts.d/vhost-ssl.conf" for individual domains. I used the latter file. I shall only show the most simple case here - with just a main standard domain associated on an Opensuse system with the directory "/srv/www/htdocs". Other domains would get the same SSL settings (if you do not plan to issue different certificates for each intranet domain):

<VirtualHost *:443>

        #  General setup for the virtual host
        DocumentRoot "/srv/www/htdocs/"
        ServerName myserv
        #ServerAdmin webmaster@example.com
        ErrorLog /var/log/apache2/error_log
        TransferLog /var/log/apache2/access_log

        #   SSL Engine Switch:
        #   Enable/Disable SSL for this virtual host.
        SSLEngine on

        #   SSL Cipher Suite:
        #   List the ciphers that the client is permitted to negotiate.
        #   See the mod_ssl documentation for a complete list.
        SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

        #   Server Certificate:
        #   Point SSLCertificateFile at a PEM encoded certificate.  If
        #   the certificate is encrypted, then you will be prompted for a
        #   pass phrase.  
        SSLCertificateFile /etc/ssl/servercerts/servercert.pem

        #   Server Private Key:
        #   If the key is not combined with the certificate, use this
        #   directive to point at the key file.  
        SSLCertificateKeyFile /etc/ssl/servercerts/serverkey.pem

        #   Server Certificate Chain:
        #   Point SSLCertificateChainFile at a file containing the
        #   concatenation of PEM encoded CA certificates which form the
        #   certificate chain for the server certificate. 
        SSLCertificateChainFile /etc/ssl/certs/anraconc-CA.pem
        #
        # NOTE: here you should later on use the Common Name of your CA certificate !
        # See below ! 
.....
......

Note that I referred to the CA certificate at "/etc/ssl/certs" in the CA chain setting. The server certificate and key files are referred to already by the standard paths "/etc/ssl/servercerts/servercert.pem" and "/etc/ssl/servercerts/serverkey.pem".

With this changed configuration file we restart our Apache service:

myserv:/etc/apache2/vhosts.d # vi vhost-ssl.conf
myserv:/etc/apache2/vhosts.d # systemctl restart apache2.service
myserv:/etc/apache2/vhosts.d # systemctl status apache2.service
● apache2.service - The Apache Webserver
   Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-07-20 15:37:29 CEST; 11s ago
  Process: 20435 ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful-stop (code=exited, status=0/SUCCESS)
 Main PID: 20443 (httpd-prefork)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
    Tasks: 6
   CGroup: /system.slice/apache2.service
           ├─20443 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>
           ├─20449 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>
           ├─20450 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>
           ├─20451 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>
           ├─20452 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>
           └─20453 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>

Jul 20 15:37:29 myserv systemd[1]: Starting The Apache Webserver...
Jul 20 15:37:29 myserv start_apache2[20443]: [Sat Jul 20 15:37:29.522427 2019] [so:warn] [pid 20443] AH01574: module authn_core_module is already loaded, skipping
Jul 20 15:37:29 myserv systemd[1]: Started The Apache Webserver.

Obviously, only root needs to read "/etc/ssl/servercerts/serverkey.pem" for starting the service - although multiple "httpd-prefork" processes later on are run by the system user "wwwrun". However, the first one is run by root.

Note:
If you get any trouble here, please have a look into the Apache log files (in my case in "/var/log/apache2/error.log"). If you see something there like

[Sat Jul 20 14:56:15.132380 2019] [ssl:emerg] [pid 13320] AH02561: Failed to configure certificate myserv:443:0, check /etc/ssl/servercerts/servercert.pem
[Sat Jul 20 14:56:15.132404 2019] [ssl:emerg] [pid 13320] SSL Library Error: error:140AB18E:SSL routines:SSL_CTX_use_certificate:ca md too weak
AH00016: Configuration Failed
[Sat Jul 20 15:00:53.251242 2019] [ssl:emerg] [pid 14285] AH02561: Failed to configure certificate myserv:443:0, check /etc/ssl/servercerts/servercert.pem
[Sat Jul 20 15:00:53.251268 2019] [ssl:emerg] [pid 14285] SSL Library Error: error:140AB18E:SSL routines:SSL_CTX_use_certificate:ca md too weak
AH00016: Configuration Failed

then you are confronted with a point I mentioned in my first article. The error message criticizes that the "md"-hashing algorithm is too weak. But didn't we use SHA256?

Well, you meant to do so - because it my have been shown as the option chosen by default for the "Digest". And you did not change this setting. However, if you did not click the radio button explicitly after some other value you may not have gotten what you saw. Without your explicit interaction with the radio buttons the default is "md". So, it is important for all certificates to explicitly choose a different value first and then click on the right radio button for a digest again. The same is true for the setting of the key length and other options.

For other services than Apache2 you would basically proceed in an analogous way. So far - so good. Unfortunately, our approach is still not good enough!

How to make a CA certificate pem file system wide and permanently available on Opensuse Leap 15.x?

One of the most frustrating things that may happen to you is that you think everything you changed works now and then - after a system restart - you find a mess of non-starting or working services again. Well, this is exactly what happens if you just place a CA's certificate (pem) file into the location "/etc/ssl/certs". Just try a reboot; you will find that the file /etc/ssl/certs/anraconc-CA.pem" does not start any more afterwards - with severe consequences for the services referring to it.

"/etc/ssl/certs" is in so far a reasonable central directory as legacy applications may read it for CA certificates. So, one would use it also for our TLS/SSL-dependent server services. Especially, because you may really find it as a default in some configuration files.

The location "/etc/ssl/certs" actually is a link to "/var/lib/ca-certificates/pem" which will not help much as the contents of this directory is refreshed after each system restart, too. I was too lazy to analyze this in detail. However, you find some information in the scripts under "/usr/lib/ca-certificates/update.d/".

Another substantial piece of information is provided by the the answer to a related question at "Stack Exchange" : how-do-i-install-a-system-wide-ssl-certificate-on-opensuse.

Here, as well as in one of the "update.d"-files, the directory "/etc/pki/trust/anchors" is mentioned as a suitable place to save CA certificate files by the admin for system wide usage. So, we may think that it is enough to copy our CA pem file into the directory "/etc/pki/trust/anchors/".

Before you get frustrated again: The "trust"-program which among other things refreshes "/etc/ssl/certs" is intelligent:
It extracts the "Common Name" from the certificate and uses it to name the pem-file which it writes at system start into "/etc/ssl/certs".

So, here are the rules if all your SSL-based services shall be configured to read your trusted CA certificate at "/etc/ssl/certs":

  1. Place your CA's certificate file (as exported from TinyCA2) into "/etc/pki/trust/anchors" (and if you have sub-CAs also their certificate files!)
  2. Then refer to the file "/etc/ssl/certs/CAs_COMMON_NAME.pem" - where CAs_COMMON_NAM is the common name you used when you set up your CA. Do not use any other name you may originally have given your exported CA certificate (pem) files.

In our case we need the file name "anraconc-CA.pem". So, this is already consistent with the name which we chose at export time in the first article. However, this happened by chance, only. We could have chosen quite a different name for the pem-file. (Well, I knew about the pitfall 🙂 )

If you had given your file a name like "myCA-cacert.pem", you would experience some trouble after having placed it into "/etc/pki/trust/anchors" and referred to it in CA chain configuration settings for any service by "/etc/ssl/certs/myCA-cacert.pem". Despite the files name "myCA-cacert.pem" it would in our case appear in "/etc/ssl/certs" with the name "anraconc-CA.pem". It took me a little while to figure this out !

SSL-reconfiguration for OpenLDAP and missing rights for the system user "ldap"

After we have made our CA certificate permanent, we proceed to the reconfiguration of the OpenLDAP service. This is a bit more complicated due to two reasons: There are potentially more configuration files to take care of and there may appear a problem with read rights. In addition you have to think of the "sssd.service" for providing encrypted connections to clients for authentication services - here for LDAP-clients.

The first point depends a bit on your specific configuration. There may be a file "/etc/ldap.conf" (controlling local LDAP-clients). In my case I had to change the entry for "tls_cacertfile":

bind_policy soft
uri ldap://myserv.anraconc.de
base dc=anraconc,dc=de
nss_map_attribute uniqueMember member
ssl start_tls
pam_password exop
pam_filter objectClass=posixAccount
tls_cacertdir /etc/ssl/certs
tls_cacertfile /etc/ssl/certs/anraconc-CA.pem
....
...

However, your main configuration will probably reside in "/etc/openldap/". There is also a "/etc/openldap/ldap.conf", which we need to modify:

tls_cacert      /etc/ssl/certs/anraconc-CA.pem

Then we have in addition the central file "slapd.conf" which must get proper TLS/SSL entries :

...
TLSCACertificateFile /etc/ssl/certs/anraconc-CA.pem
TLSCertificateFile /etc/ssl/servercerts/servercert.pem
TLSCertificateKeyFile /etc/ssl/servercerts/serverkey.pem
...

Hint: On Opensuse systems the file "/etc/sysconfig/openldap" should also contain a line:

...
OPENLDAP_START_LDAP="yes"
...
## Type:           yesno
## Default:        no
## ServiceRestart: ldap
# If set to "yes" the "ldap over ssl" feature of slapd will be enabled. Don't
# forget to add the "TLSCertificateFile" and "TLSCertificateKeyFile" options 
# to the /etc/openldap/slapd.conf (man slapd.conf).
# Note: Don't confuse this with "START_TLS", the preferred method for 
#       making encrypted LDAP connections, which is enabled as soon as You
#       specify "TLSCertificateFile" and "TLSCertificateKeyFile" in your config
#       file
#
OPENLDAP_START_LDAPS="yes"
...

But setting the right value for "TLSCACertificateFile" in "/etc/openldap/ldap.conf" is not enough:

The SSL settings also play a decisive role in the file "cn=config.ldif" (in "/etc/openldap/slapd.d/") as the core of modern LDAP services is configured by ldif-files.

Warning:
When editing the file "cn= config.ldif" file you have to be extremely careful. Make a backup first! One normally should not edit this file directly!. But as our LDAP service is not running any more we are brave and use

myserv:/etc/openldap/slapd.d # vi cn\=config.ldif

(Note the escaping backslash.

Then you edit the following entries

...
olcTLSCACertificateFile: /etc/ssl/certs/anraconc-CA.pem
olcTLSCertificateFile: /etc/ssl/servercerts/servercert.pem
olcTLSCertificateKeyFile: /etc/ssl/servercerts/serverkey.pem
....

This may afterwards lead to some warnings in log files regarding the checksum for this file. You could correct this later on by using "ldapmodify" to reedit the entries again on the then (hopefully) running LDAP-server.

Unfortunately, a service restart leads to trouble:

myserv:/etc/openldap/slapd.d # rcslapd restart
Job for slapd.service failed because the control process exited with error code.
See "systemctl  status slapd.service" and "journalctl  -xe" for details.

myserv:/etc/openldap/slapd.d # rcslapd status
● slapd.service - OpenLDAP Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/slapd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2019-07-20 16:45:37 CEST; 45s ago
  Process: 13254 ExecStart=/usr/lib/openldap/start (code=exited, status=1/FAILURE)
 Main PID: 11608 (code=exited, status=0/SUCCESS)

Jul 20 16:45:37 myserv slapd[13254]: main: TLS init def ctx failed: -1
Jul 20 16:45:37 myserv slapd[13254]: slapd stopped.
Jul 20 16:45:37 myserv slapd[13254]: connections_destroy: nothing to destroy.
Jul 20 16:45:37 myserv start[13254]: Starting ldap-server
Jul 20 16:45:37 myserv systemd[1]: slapd.service: Control process exited, code=exited status=1
Jul 20 16:45:37 myserv systemd[1]: Failed to start OpenLDAP Server Daemon.
Jul 20 16:45:37 myserv systemd[1]: slapd.service: Unit entered failed state.
Jul 20 16:45:37 myserv systemd[1]: slapd.service: Failed with result 'exit-code'.

You see the message in the middle? main: TLS init def ctx failed: -1

Now, you could google. It will not help much! It took me some time to find out that I simply had a rights problem! I saw this when comparing the old contents of /etc/ssl/servercerts" from a backup with the new one:

myserv:/etc/ssl # la servercerts.orig
total 16
drwxr-xr-x  2 root root 4096 May 30  2013 .
drwxr-xr-x  7 root root 4096 Jul 20 15:03 ..
-rw-r--r--  1 root root 2155 Jul 10  2017 servercert.pem
-rw-------+ 1 root root 3272 Jul 10  2017 serverkey.pem
myserv:/etc/ssl # la servercerts
total 16
drwxr-xr-x 2 root root 4096 Jul 20 16:40 .
drwxr-xr-x 7 root root 4096 Jul 20 15:03 ..
-rw-r--r-- 1 root root 2492 Jul 20 15:19 servercert.pem
-rw------- 1 root root 3243 Jul 20 15:22 serverkey.pem

You see the tiny difference? The "+" - sign in the original rights?
Oops! ACL-settings? What for?

myserv:/etc/ssl # getfacl servercerts.orig/serverkey.pem 
# file: servercerts.orig/serverkey.pem
# owner: root
# group: root
user::rw-
user:ldap:r--
group::r--
mask::r--
other::---

OK! lets try this for the user "ldap" in the new setup:

 
myserv:/etc/ssl # setfacl -m u:ldap:r servercerts/serverkey.pem 
myserv:/etc/ssl # setfacl -m g::- servercerts/serverkey.pem
myserv:/etc/ssl # getfacl servercerts/serverkey.pem 
# file: servercerts/serverkey.pem
# file: servercerts/serverkey.pem
# owner: root
# group: root
user::rw-
user:ldap:r--
group::---
group:root:---
mask::r--
other::---

myserv:/etc/ssl # systemctl restart slapd.service
myserv:/etc/ssl # systemctl status slapd.service
● slapd.service - OpenLDAP Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/slapd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-07-20 17:00:21 CEST; 5s ago
  Process: 15869 ExecStart=/usr/lib/openldap/start (code=exited, status=0/SUCCESS)
 Main PID: 15889 (slapd)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/slapd.service
           └─15889 /usr/sbin/slapd -h ldap:/// ldaps:/// ldapi:/// -F /etc/openldap/slapd.d -u ldap -g ldap -o slp=off

Jul 20 17:00:21 myserv systemd[1]: Stopped OpenLDAP Server Daemon.
Jul 20 17:00:21 myserv systemd[1]: Starting OpenLDAP Server Daemon...
Jul 20 17:00:21 myserv slapd[15869]: @(#) $OpenLDAP: slapd 2.4.46 $
                                          opensuse-buildservice@opensuse.org
Jul 20 17:00:21 myserv slapd[15869]: ldif_read_file: checksum error on "/etc/openldap/slapd.d/cn=config.ldif"
Jul 20 17:00:21 myserv slapd[15869]: ldif_read_file: checksum error on "/etc/openldap/slapd.d/cn=config/olcDatabase={0}config.ldif"
Jul 20 17:00:21 myserv slapd[15889]: slapd starting
Jul 20 17:00:21 myserv start[15869]: Starting ldap-server
Jul 20 17:00:21 myserv systemd[1]: Started OpenLDAP Server Daemon.
myserv:/etc/ssl # 

Obviously, the user "ldap" needs read rights on the (private) key of the server certificate!
Note that the user which runs the slapd.service may have a different name on your system - e.g. "openldap".

One can dispute whether ACLs is a reasonable way to handle this problem. Of course you could try and do something else. For other services I found in backups for older SuSE installations a special group "tls" which contained root and system users which needed read rights. This group was then assigned to "/etc/ssl/servercerts/serverkey.pem".

E.g. in case you are concerned about any of these solutions you could create a separate special certificate for the LDAP service and place the key in a special directory with read access rights for the file limited to root and the user "ldap". Of course, in this case, the service's configuration files must be adjusted for all TLS/SSL relevant data. However, before you follow such an approach of different certificates for services on one and the same server, think a bit about side consequences. As the FQDN of the server has to be identical to the certificate's common name you would have to issue certificates for different common names of your various server services.
This in turn makes your network and DNS configuration more complicated - you either need to associate one IP with multiple FQDNs or provide multiple IPs for one and the same server. To avoid such trouble I would rather combine this with a container approach for each separate service.

Adjusting the configuration of the SSSD-service

Having a running LDAP service does not mean that external clients can access it. On modern systems the exchange of auth-information between clients and server services is often handled by the "sssd"-service. This is true in my case, too. Therefore, a running slapd will not be sufficient if we use the LDAP-service for authentication from some clients in the network. We need to adjust the sssd-configuration, too. The relevant file is "/etc/sssd/sssd.conf" :

...
[sssd]
config_file_version = 2
services = nss,pam
domains = default
# SSSD will not start if you do not configure any domains.
# Add new domain configurations as [domain/<NAME>] sections, and
[nss]
filter_groups = root
filter_users = root
[pam]
# Section created by YaST
[domain/default]
ldap_uri = ldap://myserv.anraconc.de
ldap_search_base = dc=anraconc,dc=de
ldap_schema = rfc2307bis
id_provider = ldap
ldap_user_uuid = entryuuid
ldap_group_uuid = entryuuid
;ldap_id_use_start_tls = True
ldap_id_use_start_tls = False
enumerate = True
chpass_provider = ldap
auth_provider = ldap
cache_credentials = False

ldap_tls_cacertdir = /etc/ssl/certs
ldap_tls_cacert = /etc/ssl/certs/anraconc-CA.pem
....
...

Again, the most important settings are done in the last 2 lines displayed.
The setting for start_tls is up to you. If you use "start_tls" then the OpenLDAP configuration must provide the necessary entries.

Note: You have to provide similar sssd-settings on external LDAP-client-systems! One such client could be an email/imap-server - we shall have a look at its configuration in the next article.

On "myserv" we restart the "sssd.service":

myserv:/etc/ssl # vi /etc/sssd/sssd.conf
myserv:/etc/ssl # rcsssd restart
myserv:/etc/ssl # systemctl status sssd.service 
● sssd.service - System Security Services Daemon
   Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-07-20 17:29:11 CEST; 18s ago
 Main PID: 20021 (sssd)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/sssd.service
           ├─20021 /usr/sbin/sssd -i --logger=files
           ├─20022 /usr/lib/sssd/sssd_be --domain default --uid 0 --gid 0 --logger=files
           ├─20023 /usr/lib/sssd/sssd_nss --uid 0 --gid 0 --logger=files
           └─20024 /usr/lib/sssd/sssd_pam --uid 0 --gid 0 --logger=files

Jul 20 17:29:10 myserv systemd[1]: Starting System Security Services Daemon...
Jul 20 17:29:10 myserv sssd[20021]: Starting up
Jul 20 17:29:11 myserv sssd[be[20022]: Starting up
Jul 20 17:29:11 myserv sssd[20024]: Starting up
Jul 20 17:29:11 myserv sssd[20023]: Starting up
Jul 20 17:29:11 myserv systemd[1]: Started System Security Services Daemon.

Conclusion

We can use TinyCA2 to replace the missing YaST-feature for an effective administration both of CAs and server certificates. The major work has to be done in adjusting the configuration of the services. But after the method for installing a CA certificate server wide and permanently became clear the service configuration became reduced to a more or less dumb editing job. This job could be delegated to a script; but be careful to check the specific names of the relevant SSL/TLS entries for each service!

A special challenge came with the LDAP service; the related system user must get read rights to the used server certificate's private key!

In the next and final article of this series

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – III

we shall have a look at the reconfiguration of a cyrus and posfix service on a different server "mymail", which would then be a client for the LDAP service on "myserv" (for the authentication of IMAP/SMTP-users). .

Links

Server wide and permanent installation of a trusted CA's certificate
stackexchange.com: how-do-i-install-a-system-wide-ssl-certificate-on-opensuse