Upgrade of a server host from Opensuse Leap 15.1 to 15.2 – I – a problem with Apache2/MPM and PHP7

The official support of Opensuse Leap 15.1 is about to end. Time to upgrade some servers to Opensuse Leap 15.2 in my home-office. While the basic upgrade of the OS was a smooth process I stumbled across some problems and challenges with a few services. I shall cover these problems step by step in this series - and hope that my remarks and hints will help some of my readers.

A central server system with network related services and with virtualized guests

One of the two central server systems in my LAN provides DHCP-, DNS-, OpenLDAP-services plus an internal HTTP-service (based on Apache2) for development work. Its list of services in addition comprises NFS, Samba and some SVN-services for multiple purposes, among others as backup space for other systems. The host also works as a CUPS server - offering some pre-defined queues for other Linux-systems in the LAN.

The system also serves as a KVM- and LXC-host for some virtualization guests: A KVM based LAMP/GIT-server for production data, a KVM based mail-server, a KVM based Kali-system plus some LXC-containers for databases. Some LVM volumes on this server are encrypted and all disk-space is based on Raid configurations - both with mdadm for different SSD-raids and with a 3ware controller for a standard hard disk based raid10. This server is also critical as it separates its internal virtual network structure from real LAN segments by a packet-filter. It must cover routing tasks.

I run this Linux host with Opensuse Leap - not SLES. So, it is a good test object for the cleanliness of Opensuse's upgrade practices. Actually, I am always a bit nervous during upgrade. Fortunately, the HW is from the time of my freelance business - and thus safe and proven to use. It contains a passively cooled Nvidia card, for which the community repositories still provides kernel modules. The 3ware drivers also have worked reasonably well over the last 4 years. So, drivers at least should not pose any problems.

To make a long story short: The basic upgrade did not cost me much headache. I outline the upgrade procedure below - and then turn to the problems I have seen in its wake.

Backup and basic upgrade procedure

It should be clear that a server as the described one requires some regular backup routine for important data. I shall not elaborate on this topic here. Regarding the backup of the "/"-partition I use an approach described in the article
Upgrade workstation from Opensuse Leap 15.1 to Leap 15.2:

I.e. I copied the whole partition to a different, external hard-disk and afterwards in addition to a free partition on the server system itself - all with the help of the "dd"-command. You may do the same with other partitions which you want to re-create quickly in case of an emergency during upgrade.

Regarding the upgrade itself I cling to the procedure described by the Linux Kamarada; see the article mentioned above. (The required work with the Opensuse repositories is much simpler for my servers than for my workstations and laptops as I use much less repositories; the only repositories aside the Update repository for the present Leap version are SUSE's "Security" and "Network" repositories plus the Nvidia community repository. I have made good experiences with this habit over the last 10 years.)

The upgrade worked almost perfectly. Some quick checks verified that my virtualized (and not yet upgraded) virtual guest systems could be started and worked as expected. The same was true for the slapd-, smb-, the wsdd- and the dns- (=named-) service.

But, there were also exceptions which required additional work:

  • Graphics Failure: The X-system did not start and the "graphical target" of systemd could not be reached directly after the system upgrade (expected).
  • Service failure: The Apache2 service did not start (unexpected).
  • Service failure: The smartd-service did not start (unexpected, but probably due to an overwritten configuration file).
  • Required Adjustment: The CUPS libraries had to be changed to Opensuse's "Printing" repository to become consistent with other Linux hosts which use printer queues on the server (expected).
  • Required Adjustment: The rkhunter-configuration had to be adjusted (expected).
  • Required Adjustment: A specific KDE-settings had to be adjusted to allow for screen-saving functions - even for SDDM screens without started KDE-sessions (unexpected).
  • Required Adjustment: "virt-viewer" access via "qemu+ssh" to some clients posed grave problems (unexpected). These problems may have existed already before, but are worth a description.

I cover the first 2 points in this article. Solving the graphics problem is a short story. The explanation and solution of the problem with the Apache service, however, requires some space. But I hope the information provided in the related text is interesting for some readers who are no experts in Opensuse's handling of the Apache service.

Nvidia - X-Windows

You may ask: Why X11 on a server at all? Good question; well, as a semi-professional in server-administration, I think that one can do some things faster than without. Normally, I access the server remotely via ssh or "ssh -X". But during my years with SLES and Opensuse I experienced some situations during which I was happy to be able to access the server directly on a graphical console; certainly not a popular view among profs. Anyway - the Nvidia/X-Window topic is a standard problem and easy to solve:

I just had to reinstall the required drivers from the Nvidia repository (via the CLI-based YaST), perform a "mkinitrd" afterwards and restart the system.

However, I nevertheless recommend not to boot the server system into the graphical systemd target, if not required. The relevant command to set the default target is

systemctl set-default multi-user.target

See the related link at the end of this post.

Off topic: If you prefer KDE as a graphical desktop environment do not even try to switch to Wayland on Leap 15.2. In my experience this is only a frustrating endeavor - as we still do not have Plasma 5.20 available but Plasma 5.18 (see a related link at the end).

Apache does not start due to a problem with "mod_php7" and MPM

Let us turn to something more interesting: An Apache problem with the current PHP-module and MPM-settings.

I admit, I am a bit old-fashioned. Most developers and admins, today, probably use Nginx and in addition Fast-CGI in combination with PHP. I, instead, cling to an old Apache installation for LAMP-services on the server named above. One reason is that I need some compatibility with things I did from 2013 and 2016 - aside of changes of PHP over the years.

So, whilst upgrading Opensuse my Apache 2.4 and PHP installations get upgraded, too.

When one activates the php-Apache-module by "a2enmod php" the presently installed version of the php-Apache-module gets loaded - on Leap 15.2 this is provided by mod_php7.so corresponding to PHP 7.4.6. (You can verify this with the command "php -v"). Using "a2enmod php7" would give you no different result. (Opensuse, unfortunately, does not support the parallel installation of multiple, different PHP-versions. See the critics in the conclusion-section.)

All upgrades from Leap 42.2 to 42.3, 15.0 and eventually to Leap 15.1 went well; but this time the Apache service did not start. Trying "rcapache2 start" followed by "rcapache2 status" (or equivalent systemctl status apache2.service) gave me an error message:

....
[php7:crit] [pid 16004:tid 139738320386304] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe. You need to recompile PHP.
Jan 26 19:15:39 MySRV start_apache2[16004]: AH00013: Pre-configuration failed
Jan 26 19:15:39 MySrv systemd[1]: apache2.service: Main process exited, code=exited, status=1/FAILURE
....

I remembered vaguely that I had experimented with MPM and prefork settings on Leap 15.1, but recompile PHP? With what options? A strange error message!

Confusion when comparing to a seemingly similar Apache2 installation on a different system

I cross-checked with a similarly configured Apache2 installation (with PHP7, but different virtual domains) on one of my my Leap 15.2 workstations; let us call it "MyWS" and the upgraded server "MySRV". Confusingly, Apache2 started on MyWS without any complaints.

You may suspect now that I may have used different Apache2 configuration directives in "/etc/sysconfig/apache2" or the files in folder "/etc/apache2/" and sub-folders - without being aware of it. But no: I checked and eliminated even small, irrelevant differences for tests. The problem on the server MySRV remained.

When I looked at the Apache2-related processes on the workstation MyWS with "ps aux | grep https" I saw that Apache2 had been started in a pre-forked way. As I was used to .... I got multiple forked processes like

....
wwwrun    7043  0.0  0.0 236180 10488 ?        S    10:41   0:00 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include .... 
.....
wwwrun    7045  0.0  0.0 236180 10488 ?        S    10:41   0:00 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include ... 
... 

The interesting thing here is the indicated program "/usr/sbin/httpd-prefork". Note that it is NOT "usr/bin/httpd"; I shall come back to this point later. The process information was of course consistent with startup information provided by "rcapache2 status" or equivalently by "systemctl status apache2".

It took me a while to find out what was going on and what I had to do. Whilst searching the Internet and experimenting with Apache I passed some interesting information about details of the Apache-installation on Opensuse which may be interesting for others, too.

MPM methods on Apache 2.4

The error message indicated that the whole problem had to do with MPMs - "Multi-Processing-Modules", i.e. the ability of Apache to handle a multitude of requests concurrently. There are 3 MPM-methods and respective pre-compiled static modules or loadable DSO modules available with Apache 2.4. See e.g.:

https://www.digitalocean.com/community/tutorials/how-to-configure-apache-http-with-mpm-event-and-php-fpm-on-ubuntu-18-04-de,
http://httpd.apache.org /docs /current /en /mpm.html
and links therein.

I quote:

  • Pre-fork: A new process is created for each incoming connection reaching the server. Each process is isolated from the others, so no memory is shared between them, even if they are performing identical calls at some point in their execution. This is a safe way to run applications linked to libraries that do not support threading—typically older applications or libraries.
  • Worker: A parent process is responsible for launching a pool of child processes, some of which are listening for new incoming connections, and others are serving the requested content. Each process is threaded (a single thread can handle one connection) so one process can handle several requests concurrently. This method of treating connections encourages better resource utilization, while still maintaining stability. This is a result of the pool of available processes, which often has free available threads ready to immediately serve new connections.
  • Event: Based on worker, this MPM goes one step further by optimizing how the parent process schedules tasks to the child processes and the threads associated to those. A connection stays open for 5 seconds by default and closes if no new event happens; this is the keep-alive directive default value, which retains the thread associated to it. The Event MPM enables the process to manage threads so that some threads are free to handle new incoming connections while others are kept bound to the live connections. Allowing re-distribution of assigned tasks to threads will make for better resource utilization and performance.

At http://httpd.apache.org/docs/trunk/misc/perf-scaling.html we furthermore read:

Httpd Configuration
The Apache 2.2 httpd is by default a pre-forking web server. When the server starts, the parent process spawns a number of child processes that do the actual work of servicing requests. But Apache httpd 2.0 introduced the concept of the Multi-Processing Module (MPM). Developers can write MPMs to suit the process- or threading- architecture of their specific operating system. Apache 2 comes with special MPMs for Windows, OS/2, Netware and BeOS. On unix-like platforms, the two most popular MPMs are Prefork and Worker. The Prefork MPM offers the same pre-forking process model that Apache 1.3 uses. The Worker MPM runs a smaller number of child processes, and spawns multiple request handling threads within each child process. In 2.4 MPMs are no longer hard-wired. They too can be exchanged via LoadModule. The default MPM in 2.4 is the event MPM.
....
The maximum number of workers, be they pre-forked child processes or threads within a process, is an indication of how many requests your server can manage concurrently. It is merely a rough estimate because the kernel can queue connection attempts for your web server. When your site becomes busy and the maximum number of workers is running, the machine doesn't hit a hard limit beyond which clients will be denied access. However, once requests start backing up, system performance is likely to degrade.

Finally, if the httpd server in question is not executing any third-party code, via mod_php, mod_perl or similar, we recommend the use of mpm_event. This MPM is ideal for situations where httpd serves as a thin layer between clients and backend servers doing the real job, such as a proxy or cache.

...
Selecting your MPM
The prime reason for selecting a threaded MPM is that threads consume fewer system resources than processes, and it takes less effort for the system to switch between threads. This is more true for some operating systems than for others. On systems like Solaris and AIX, manipulating processes is relatively expensive in terms of system resources. On these systems, running a threaded MPM makes sense.

On Linux, the threading implementation actually uses one process for each thread. Linux processes are relatively lightweight, but it means that a threaded MPM offers less of a performance advantage than in other environments.

Running a threaded MPM can cause stability problems in some situations For instance, should a child process of a preforked MPM crash, at most one client connection is affected. However, if a threaded child crashes, all the threads in that process disappear, which means all the clients currently being served by that process will see their connection aborted. Additionally, there may be so-called "thread-safety" issues, especially with third-party libraries. In threaded applications, threads may access the same variables indiscriminately, not knowing whether a variable may have been changed by another thread.

This has been a sore point within the PHP community. The PHP processor heavily relies on third-party libraries and cannot guarantee that all of these are thread-safe. The good news is that if you are running Apache on Linux, you can run PHP in the preforked MPM without fear of losing too much performance relative to the threaded option.

Ok, what do we learn from this? php_mod is not thread-safe. But regarding performance, using the "pre-fork" MPM method is not really a bad choice on a Linux system. Obviously, this method was used on the workstation. An interesting question, which I come back to later, is whether Opensuse offers dynamically loadable or static MPM modules via its repositories.

Information about the MPM used by your installation?

One (naively) expects that the Apache2 daemon binary should be callable by "/usr/sbin/httpd" on an Opensuse system. Actually, the command "which httpd" gives you "/usr/sbin/httpd" as an answer.

On the Internet you find hints that one can check the compiled and available MPMs by either "httpd -V" and/or "httpd -l". On the upgraded server MySRV I got:

MySRV:~ # httpd -V
Server version: Apache/2.4.43 (Linux/SUSE)
Server built:   2020-11-17 11:05:32.000000000 +0000
Server's Module Magic Number: 20120211:92
Server loaded:  APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
....

and

MySRV:~ # httpd -l
Compiled in modules:
  core.c
  mod_so.c
  http_core.c
  worker.c
  mod_unixd.c
  mod_systemd.c

These commands seemingly worked on my server even if the Apache2-service itself could not be started (e.g. by "systemctl start apache2"). The man-pages for "httpd" say that the "-l"-option lists modules which were compiled into the server, i.e. static modules.

What did I get on the workstation?

MyWS:~ # httpd -V
Server version: Apache/2.4.43 (Linux/SUSE)
Server built:   2021-01-27 09:11:10.000000000 +0000
Server's Module Magic Number: 20120211:92
Server loaded:  APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)

and

MyWS:~ # httpd -l
Compiled in modules:
  core.c
  mod_so.c
  http_core.c
  prefork.c
  mod_unixd.c
  mod_systemd.c

Regarding the actually running processes after Apache-startup on the workstation this result is not too surprising. However, if the listed modules are static then the "httpd" binary on the server must be another one than on the workstation. Why? And how did this come about ?

Another question is: How much can we rely on the information given by e.g. "httpd -l" in the sense that this information reflects facts about the running Apache2 service? We shall see that the answer depends ...

Removing the PHP-module on both installations

To analyze further I first aligned the list of modules on both systems via the corresponding setting in "/etc/sysconfig/apache2" to:

APACHE_MODULES="actions alias auth_basic authn_file authz_host authz_groupfile authz_core authz_user autoindex cgi dir env expires include log_config mime rewrite negotiation setenvif ssl socache_shmcb userdir reqtimeout php7 authn_core version"

and then removed the "php7" module.

(Instead of editing the file "/etc/sysconfig/apache2" directly you could also use a2enmod or a2dismod, respectively, for changing the module configuration. But I recommend to keep some commented alternatives for in the file for fast changes between repeated tests of different constellations).

In addition I used the seemingly standard Opensuse setting of

APACHE_MPM=""

in "/etc/sysconfig/apache2" for both installations.

Well, as expected no significant changes occurred on the workstation. Apache2 started successfully again (without PHP support); multiple prefork-processes were launched. On the server, however, I now got a successful start, too; "rcapache2 status" showed me a bunch of related processes afterwards:

MySRV:~ # rcapache2 status
● apache2.service - The Apache Webserver
   Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; vendor preset: >
   Active: active (running) since Sun 2021-01-31 11:45:26 CET; 23s ago
  Process: 8555 ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful->
 Main PID: 20667 (httpd-worker)
   Status: "Processing requests..."
    Tasks: 83
   CGroup: /system.slice/apache2.service
           ├─20667 /usr/sbin/httpd-worker -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid ....
           ├─20675 /usr/sbin/httpd-worker -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid ....
           ├─20676 /usr/sbin/httpd-worker -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid ....
           ├─20677 /usr/sbin/httpd-worker -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid ....
           └─20678 /usr/sbin/httpd-worker -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid ....

Jan 31 11:45:26 MySRV systemd[1]: Starting The Apache Webserver...
Jan 31 11:45:26 MyServ start_apache2[20667]: AH00548: NameVirtualHost has no effect and>
Jan 31 11:45:26 MySRV systemd[1]: Started The Apache Webserver.

But you see the difference?
=> The programs started were of the type "/usr/bin/http-worker" !

Looking at the startup scripts for the Apache2 service

To dig a bit deeper I looked at the startup scripts used on Opensuse for the Apache2 service. One reason to do so was a comment in Opensuse's "/etc/sysconfig/apache2" regarding the variable APACHE_MPM:

# MPM (multi-processing module) to use.
#
# Needed to determine with which MPM apache will run, as well as
# against which header files modules will be built. 
#
# If not set, the system will simply pick one of the installed MPMs.
#
# The implementation of the logic is in /usr/share/apache2/find_mpm,
# a script which can be used standalone as well if needed.

This indicates that some special Opensuse logic is used during startup. But the comment is also misleading: A script "find-mpm" can only be found in the folder "/usr/share/apache2/deprecated-scripts" on a Leap 15.1 and 15.2 systems - and the "script" itself is not used any more, as we shall see in a minute.

So, where do we find the startup-script for Apache? We do not have to look into the systemd configuration for the Apache2 service; the output of the command "rcapache2 status" already revealed it as
"/usr/sbin/start_apache2".

In it's script-code we find a line for the execution of a helper script:

 . /usr/share/apache2/script-helpers

and later a call to a function "find_mpm", followed by some logic for the name of a binary to be saved in the variable "apache_bin":

#
# figure out correct apache2 binary (/usr/sbin/httpd-prefork,
# /usr/sbin/httpd-worker, etc.) and serverflags
#
find_mpm
if [ -n "$HTTPD_MPM" ]; then
    apache_bin="$HTTPD_SBIN_BASE-$HTTPD_MPM"
    if ! [ -x $apache_bin ]; then
        echo >&2 "$apache_bin-$APACHE_MPM is not a valid httpd binary."
        echo >&2 "Check your APACHE_MPM setting in /etc/sysconfig/apache2."
        exit 1
    fi
else
  echo >&2 "${warn}No Apache binary found. No MPM package installed? $norm"
  echo >&2 "Hint: install the apache2-prefork package, and try again."
  exit 1
fi

Obviously, we will find the function "find_mpm" in "/usr/share/apache2/script-helpers". A look into this file shows that this really is the case. The outcome of this function influences the startup of Apache2; obviously, a different Apache variant is started for different (valid) values of the script-variable "HTTPD_MPM":

apache_bin="$HTTPD_SBIN_BASE-$HTTPD_MPM"

($HTTPD_SBIN_BASE resolves to "/usr/sbin/httpd" - this is set at the beginning of "/usr/share/apache2/script-helpers".)

Now, you can easily build your own test-script "mpm_test" including this function and print its output:

mpm_test-script:

#!/bin/bash

HTTPD_SBIN_BASE="/usr/sbin/httpd"
#
# loads sysconfig variables into environment
# return value in: APACHE_*
function load_sysconfig
{
  [ -n "$sysconfig_loaded" ] && return
  [ ! -f "$SYSCONFIG_FILE" ] && return

  . $SYSCONFIG_FILE

  export ${!APACHE_*} sysconfig_loaded=true
}
#
# finds prefered multiprocessing module
# return value in: HTTPD_MPM 
function find_mpm
{
  # load sysconfig variables if they weren't yet;
  # this has no effect when find_mpm is not called
  # from start_apache2
  load_sysconfig

  # try to read from sysconfig's APACHE_MPM
  HTTPD_MPM="$APACHE_MPM"
  # if empty, then choose one from installed
  if [ -z "$HTTPD_MPM" ]; then
      installed_mpms=""
      for i in $HTTPD_SBIN_BASE-*; do
          test -f $i || continue
          i=$(basename $i)
          i=${i#*-}
          installed_mpms="$installed_mpms $i"
      done
      # hardcoded preference here:
      for mpm in event worker prefork; do
        if [[ $installed_mpms =~ "$mpm" ]]; then
          HTTPD_MPM=$mpm
          break
        fi
      done
  fi

  # in case no 
  export HTTPD_MPM
}

# ***********************
# Action
# ***********************
find_mpm
echo $HTTPD_MPM

This script produced the following output on my server MySRV (with a running Apache2 without PHP support):

worker

On my workstation (and a running Apache2 with and without PHP support) I got:

prefork

Available and different Apache binaries

Analyzing the logic of our file, which is the same as in the beginning of the startup-script, we see that
"apache_bin" can become either "/usr/sbin/httpd-prefork" or "/usr/sbin/httpd-worker". It depends on the value of the variable "APACHE_MPM" in "/etc/sysconfig/apache2":

  • If the variable "APACHE_MPM" is not empty its value is used.
  • But if it is empty the result depends on the existence of one or multiple of the 3 files "httpd-event", "httpd-worker" or "httpd-prefork" in the folder "/usr/sbin".

You also see that the logic, which the guys from Opensuse implemented, chooses a MPM-specific variant of Apache2 according to the following fixed priority order:

event > worker > prefork    !

The "prefork" variant has the least priority - and is obviously only chosen if neither the file "httpd-worker" nor the file "httpd-event" exist - and if "APACHE_MPM" is not set otherwise.

A quick check showed that binaries "/usr/sbin/httpd-worker", "/usr/sbin/httpd-event" did not exist on my workstation MyWS; there, only "/usr/sbin/httpd-prefork" was available. But on the server MySRV I actually had two variants available "/usr/sbin/httpd-prefork" and "/usr/sbin/httpd-worker".

This fact together with the "find_mpm"-logic explained the different start-up results for Apache2 on my systems MySRV and MyWS !

But not the original cause and why Apache had worked on MySRV with Leap 15.1 before ....

Differences in the package installation?!

The existence of different binaries on MyWS and the server MySRV could only be due to a package difference, which must have been there already before the upgrade to Leap 15.2. A check in my logs and also a look into the package constellation with YaST revealed that on the Leap 15.1 version of the server-system MySRV I actually had installed the RPM "apache2-worker" at some point in time! For some experiments which I had totally forgotten. But not on the workstation. So, I had caused the trouble somehow by myself - without anticipating any consequences for a coming upgrade to Leap 15.2.

But wait a second - this does not explain why I did not run into trouble with Apache on Leap 15.1. Despite the installed package "apache_worker" and an empty "APACHE_MPM" in the sysconfig file..... Why did Apache2 start on Leap 15.1 - as my old log-files show? With "httpd-prefork" processes, by the way ...

I had to suspect a difference in the "find_mpm" logic - if I really had left "Apache_MPM" empty on the server. The latter was the case as I saw in my backups. And indeed - on Leap 15.1 the logic of the function "find_mpm" in "/usr/share/apache2/script-helpers" was different:

The old "find_mpm" from Leap 15.1

function find_mpm
{
  [ -n "$mpm_found" ] && return

  # load sysconfig variables if they weren't yet
  load_sysconfig

  # try to read from sysconfig's APACHE_MPM
  HTTPD_MPM=$APACHE_MPM
  # if empty, then choose one from installed
  if [ -z "$HTTPD_MPM" ]; then
      # guess
      for i in $HTTPD_SBIN_BASE-*; do
          test -f $i || continue
          i=$(basename $i)
          i=${i#*-}
          installed_mpms=(${installed_mpms[*]} $i)
      done
      if [ -z "${installed_mpms[*]}" ]; then
          HTTPD_MPM=""
          return
      elif [ ${#installed_mpms[*]} = 1 ]; then
          HTTPD_MPM=${installed_mpms[*]}
      else
          case ${installed_mpms[*]} in
              *prefork*)      HTTPD_MPM=prefork;;
              *worker*)       HTTPD_MPM=worker;;
              *event*)        HTTPD_MPM=event;;
          esac
      fi
  fi

  export HTTPD_MPM mpm_found=true
}

A test shows that the "case"-part of this function gives us an answer "prefork" on a system with both the "prefork" and the "worker" variants of Apache installed!

This makes me wonder why, for heavens sake, the Opensuse guys did not insert a section into their Leap 15.2 Release Notes telling us something about this change!

The solution to my Apache/MPM/PHP problem

After all the information gathering above the very trivial solution to my start-up problem was as that I just had to set the variable "APACHE_MPM" in "/etc/sysconfig/apache2" to

APACHE_MPM="prefork"

when I wanted a successful start of an Apache instance with PHP support.

What do the http-* files in "/usr/sbin/" look like for multiple MPM-installations?

What happens if we install multiple MPM-variants of Apache? From some parts of the above text you have already concluded that we really must speak of "variants" as the MPM-module is compiled in statically by the Opensuse guys.

On the server MySRV with the worker-MPM active (no PHP loaded) we can see this by using the command "apachectl -M". See the man pages. The result is:

MySRV: ~# apachectl -M
Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 mpm_worker_module (static)
 unixd_module (static)
 systemd_module (static)
 actions_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authz_host_module (shared)
...
...
 

So, we expect multiple, relatively big variants of the "http"-binary under "/usr/sbin" whenever we install the packages "apache2-worker" and "apache2-event". Which I did for fun and step-wise on the workstation MyWS. And indeed - after the installation of the RPM "apache2-worker" we see:

MyWS:~ # la /usr/sbin/ | grep httpd
lrwxrwxrwx  1 root root           22 Jan 31 16:27 httpd -> /usr/sbin/httpd-worker
-rwxr-xr-x  1 root root       616816 Jan 27 16:17 httpd-prefork
-rwxr-xr-x  1 root root       629200 Jan 27 16:17 httpd-worker
lrwxrwxrwx  1 root root           23 Jan 30 11:02 httpd2 -> /usr/sbin/httpd-prefork
lrwxrwxrwx  1 root root           13 Jan 27 16:17 httpd2-prefork -> httpd-prefork
lrwxrwxrwx  1 root root           12 Jan 27 16:17 httpd2-worker -> httpd-worker

and after an additional installation of the "event" version of Apache2 we get:

MyWS:~ # la /usr/sbin/ | grep httpd
lrwxrwxrwx  1 root root           21 Jan 31 16:29 httpd -> /usr/sbin/httpd-event
-rwxr-xr-x  1 root root       641552 Jan 27 16:17 httpd-event
-rwxr-xr-x  1 root root       616816 Jan 27 16:17 httpd-prefork
-rwxr-xr-x  1 root root       629200 Jan 27 16:17 httpd-worker
lrwxrwxrwx  1 root root           23 Jan 30 11:02 httpd2 -> /usr/sbin/httpd-prefork
lrwxrwxrwx  1 root root           11 Jan 27 16:17 httpd2-event -> httpd-event
lrwxrwxrwx  1 root root           13 Jan 27 16:17 httpd2-prefork -> httpd-prefork
lrwxrwxrwx  1 root root           12 Jan 27 16:17 httpd2-worker -> httpd-worker

At the end of both installations a post-installation script is running, which creates the binaries, the "httpd"-link and the various "httpd2"-links.

These files/links do not change when we restart Apache2 with a different MPM-setting in "/etc/sysconfig/apache2".

Can we trust in "httpd -l"? What about "apachectrl -M"?

From the above information we must conclude that we cannot trust the information provided by "httpd -l" or "httpd -V" to give us reliable information for a running Apache2 service on Opensuse. It only shows us, what the most modern MPM among the available ones on the system is. Something similar is to say about "httpd2 -l"; it shows us the other end of the spectrum. Only, if we have just one MPM variant of Apache2 installed all information gets identical and consistent.

If you have a running Apache2 service, a really reliable information regarding the compiled in MPM module is given by "Apachectrl -M" or indirectly by "ps aux | grep httpd", which reveals the type of the httpd-binary variant started.

Conclusion

Apache2 is provided on Opensuse with static compiled in MPM modules. If you want to choose between different MPMs you have to install related packages; they provide different variants of the httpd-binary which you find afterwards in the directory "/usr/sbin".

The present Leap 15.2-specific startup-logic for the Apache-service is based an Opensuse-specific script-function "find_mpm" (witin the file "/usr/share/apache2/script-helpers"). According to its logic you MUST specify the MPM-variant via setting a variable in "/etc/sysconfig/apache2" if you want to or must break the fixed preference order "even > worker > prefork".

Opensuse changed the logic of its MPM selection in the function "find-mpm" between Leap 15.1 and Leap 15.2. This change may, unfortunately, block a successful start of an upgraded Apache2-service with PHP-support. (A bit of concise information on this point in the Release Notes would have been helpful.)

In case you use "mod_php" for PHP support by Apache2 and thus add "php7" to the list of Apache-modules to be loaded at start-up, you MUST set the "APACHE_MPM"-variable to "prefork" - at least if you follow the standard Opensuse concepts for the Apache service. The enforced choice is reasonable as PHP itself is not thread-safe.

Criticism, Opensuse Apache/PHP deficits and alternatives

As I criticize the handling of Apache2 on Opensuse I also want to add that I miss the flexibility other distributions offer regarding the parallel installation of multiple PHP-versions and a switch between them on the Apache-server. At least with a fast-cgi implementation.

I furthermore do not understand why Opensuse restricts the PHP version in their Update repository to just one and in the PHP-specific repository to the latest 2 versions. And even for these two PHP versions a developer is forced to install two virtual Opensuse machines with different Apache/PHP constellations, if he does not want to change the installed packages all the time.

The somewhat unflexible provision of Apache in combination with PHP and the MPM-logic may make you think about alternatives. There are various situations:

  • If you look beyond Apache I would recommend to setup Nginx.
  • If you look at a way to combine PHP with threaded MPM variants as "worker" or "event, try an installation based on FastCGI and PHP-FPM instead of using php_mod. But consider security aspects then.
  • If you look for support of multiple PHP-versions with Nginx and/or Apache install Debian or Ubuntu in a virtual machine (on a Opensuse host 🙂 ).

For those who like to experiment I have added some links in the last section of this post.

Outlook

In the next article I shall cover the startup-problem of the "smartd"-service. This will be a much shorter story.

Links

Links related to the Apache MPM-topic
http://httpd.apache.org /docs /current /en /mpm.html
https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-apache2.html
http://httpd.apache.org/docs/trunk/misc/perf-scaling.html

Check available MPM-method on Apache
https://www.binarytides.com/check-which-mpm-multi-processing-module-apache-is-running/

Links regarding an alternative FastCGI installation with PHP-FPM
https://en.opensuse.org/SDB:Apache_FastCGI_and_PHP-FPM_configuration
https://www.p4tchwork.de/apache-php-system-resource-usage/
https://blog.wappler.systems/opensuse-leap-15-apache-2-php-fpm-http2/

Flexible Apache and PHP installations on Debian derivatives
https://www.digitalocean.com/community/tutorials/how-to-configure-apache-http-with-mpm-event-and-php-fpm-on-ubuntu-18-04-de
https://www.epidemiology.tech/apache-php7-4-wordpress-setup/
https://tecadmin.net/setup-apache-php-fpm-ubuntu-20-04/
https://www.interserver.net/tips/kb/change-php-version-apache-ubuntu/

Links regarding Nginx
https://techviewleo.com/install-php-on-opensuse-nginx-apache/
https://www.tecmint.com/install-lemp-nginx-php-mariadb-phpmyadmin-in-opensuse/
https://websiteforstudents.com/install-php-7-4-php-7-4-fpm-on-ubuntu-18-04-with-apache2-nginx/

HTTP 2 installation on Opensuse
https://en.opensuse.org/SDB:Apache_HTTP_2_configuration

Set default target on Opensuse
https://forums.opensuse.org/showthread.php/526792-How-to-change-default-runlevel

Plasma, Nvidia and Wayland
https://community.kde.org/Plasma/Wayland/Nvidia

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – II

In my last article on TinyCA2

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – I

I discussed how to open an new CA and how to create a server certificate (plus private key) with SHA256 as the preferred hashing algorithm. We saw that TinyCA2 offers an option to export the certificate's private key with or without password. I also gave an "openssl" command for stripping a key off its password later on - even if it was exported with a password in the beginning.

We have followed a policy of using a server wide TLS/SSL-certificate for all TLS/SSL-based services run on a server - in our case on a server "myserv.anraconc.de". Therefore we have copied the certificate's pem file and the key's pem file to the directory "/etc/ssl/servercerts" with file names "servercert.pem" and "serverkey.pem".

In addition we naively copied the CA certificate as file "anraconc-CA.pem" into the central directory "/etc/ssl/certs" which contains (links to) all trusted certificates on an Opensuse system.

If the private key was exported with a password we have a problem with automated system and service starts: As the private key of a server certificate must in some cases also by system users starting and running services we would be asked to provide a password interactively, whenever the service is started - e.g. during system startup. To avoid such a situation we need a private key without password - hence we have to deal carefully with read access rights. Actually, we need to find out which services - not started or run by root - need the access right for its special system user. For the time being we took away all read rights for "others" from the server key for the system "myserv".

In this second article I discuss the changes to be made to the configuration of two services - Apache2 and OpenLDAP on server "myserv". After a test we shall detect that Opensuse rebuilds the contents of "/etc/ssl/certs" during a reboot - and references of our services to the CA's certificate will fail. I shall show where to save our CA certificate to make it permanently available.

Re-configuring a local Apache service for the new CA and the changed server certificate

As I had used a server wide certificate on "myserv" already at times when YaST's CA-administration was still available, I expected that the required changes to the service configuration would be limited to providing the path to the new CA certificate. The example of the Apache service proved that this really was true.

In my case I use virtual domains with my Apache server for Intranet sites. The configuration of the SSL-setup then is to be found in either a central file as "/etc/apache2/ssl-global.conf" or in "/etc/apache2/vhosts.d/vhost-ssl.conf" for individual domains. I used the latter file. I shall only show the most simple case here - with just a main standard domain associated on an Opensuse system with the directory "/srv/www/htdocs". Other domains would get the same SSL settings (if you do not plan to issue different certificates for each intranet domain):

<VirtualHost *:443>

        #  General setup for the virtual host
        DocumentRoot "/srv/www/htdocs/"
        ServerName myserv
        #ServerAdmin webmaster@example.com
        ErrorLog /var/log/apache2/error_log
        TransferLog /var/log/apache2/access_log

        #   SSL Engine Switch:
        #   Enable/Disable SSL for this virtual host.
        SSLEngine on

        #   SSL Cipher Suite:
        #   List the ciphers that the client is permitted to negotiate.
        #   See the mod_ssl documentation for a complete list.
        SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

        #   Server Certificate:
        #   Point SSLCertificateFile at a PEM encoded certificate.  If
        #   the certificate is encrypted, then you will be prompted for a
        #   pass phrase.  
        SSLCertificateFile /etc/ssl/servercerts/servercert.pem

        #   Server Private Key:
        #   If the key is not combined with the certificate, use this
        #   directive to point at the key file.  
        SSLCertificateKeyFile /etc/ssl/servercerts/serverkey.pem

        #   Server Certificate Chain:
        #   Point SSLCertificateChainFile at a file containing the
        #   concatenation of PEM encoded CA certificates which form the
        #   certificate chain for the server certificate. 
        SSLCertificateChainFile /etc/ssl/certs/anraconc-CA.pem
        #
        # NOTE: here you should later on use the Common Name of your CA certificate !
        # See below ! 
.....
......

Note that I referred to the CA certificate at "/etc/ssl/certs" in the CA chain setting. The server certificate and key files are referred to already by the standard paths "/etc/ssl/servercerts/servercert.pem" and "/etc/ssl/servercerts/serverkey.pem".

With this changed configuration file we restart our Apache service:

myserv:/etc/apache2/vhosts.d # vi vhost-ssl.conf
myserv:/etc/apache2/vhosts.d # systemctl restart apache2.service
myserv:/etc/apache2/vhosts.d # systemctl status apache2.service
● apache2.service - The Apache Webserver
   Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-07-20 15:37:29 CEST; 11s ago
  Process: 20435 ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful-stop (code=exited, status=0/SUCCESS)
 Main PID: 20443 (httpd-prefork)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
    Tasks: 6
   CGroup: /system.slice/apache2.service
           ├─20443 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>
           ├─20449 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>
           ├─20450 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>
           ├─20451 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>
           ├─20452 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>
           └─20453 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/a>

Jul 20 15:37:29 myserv systemd[1]: Starting The Apache Webserver...
Jul 20 15:37:29 myserv start_apache2[20443]: [Sat Jul 20 15:37:29.522427 2019] [so:warn] [pid 20443] AH01574: module authn_core_module is already loaded, skipping
Jul 20 15:37:29 myserv systemd[1]: Started The Apache Webserver.

Obviously, only root needs to read "/etc/ssl/servercerts/serverkey.pem" for starting the service - although multiple "httpd-prefork" processes later on are run by the system user "wwwrun". However, the first one is run by root.

Note:
If you get any trouble here, please have a look into the Apache log files (in my case in "/var/log/apache2/error.log"). If you see something there like

[Sat Jul 20 14:56:15.132380 2019] [ssl:emerg] [pid 13320] AH02561: Failed to configure certificate myserv:443:0, check /etc/ssl/servercerts/servercert.pem
[Sat Jul 20 14:56:15.132404 2019] [ssl:emerg] [pid 13320] SSL Library Error: error:140AB18E:SSL routines:SSL_CTX_use_certificate:ca md too weak
AH00016: Configuration Failed
[Sat Jul 20 15:00:53.251242 2019] [ssl:emerg] [pid 14285] AH02561: Failed to configure certificate myserv:443:0, check /etc/ssl/servercerts/servercert.pem
[Sat Jul 20 15:00:53.251268 2019] [ssl:emerg] [pid 14285] SSL Library Error: error:140AB18E:SSL routines:SSL_CTX_use_certificate:ca md too weak
AH00016: Configuration Failed

then you are confronted with a point I mentioned in my first article. The error message criticizes that the "md"-hashing algorithm is too weak. But didn't we use SHA256?

Well, you meant to do so - because it my have been shown as the option chosen by default for the "Digest". And you did not change this setting. However, if you did not click the radio button explicitly after some other value you may not have gotten what you saw. Without your explicit interaction with the radio buttons the default is "md". So, it is important for all certificates to explicitly choose a different value first and then click on the right radio button for a digest again. The same is true for the setting of the key length and other options.

For other services than Apache2 you would basically proceed in an analogous way. So far - so good. Unfortunately, our approach is still not good enough!

How to make a CA certificate pem file system wide and permanently available on Opensuse Leap 15.x?

One of the most frustrating things that may happen to you is that you think everything you changed works now and then - after a system restart - you find a mess of non-starting or working services again. Well, this is exactly what happens if you just place a CA's certificate (pem) file into the location "/etc/ssl/certs". Just try a reboot; you will find that the file /etc/ssl/certs/anraconc-CA.pem" does not start any more afterwards - with severe consequences for the services referring to it.

"/etc/ssl/certs" is in so far a reasonable central directory as legacy applications may read it for CA certificates. So, one would use it also for our TLS/SSL-dependent server services. Especially, because you may really find it as a default in some configuration files.

The location "/etc/ssl/certs" actually is a link to "/var/lib/ca-certificates/pem" which will not help much as the contents of this directory is refreshed after each system restart, too. I was too lazy to analyze this in detail. However, you find some information in the scripts under "/usr/lib/ca-certificates/update.d/".

Another substantial piece of information is provided by the the answer to a related question at "Stack Exchange" : how-do-i-install-a-system-wide-ssl-certificate-on-opensuse.

Here, as well as in one of the "update.d"-files, the directory "/etc/pki/trust/anchors" is mentioned as a suitable place to save CA certificate files by the admin for system wide usage. So, we may think that it is enough to copy our CA pem file into the directory "/etc/pki/trust/anchors/".

Before you get frustrated again: The "trust"-program which among other things refreshes "/etc/ssl/certs" is intelligent:
It extracts the "Common Name" from the certificate and uses it to name the pem-file which it writes at system start into "/etc/ssl/certs".

So, here are the rules if all your SSL-based services shall be configured to read your trusted CA certificate at "/etc/ssl/certs":

  1. Place your CA's certificate file (as exported from TinyCA2) into "/etc/pki/trust/anchors" (and if you have sub-CAs also their certificate files!)
  2. Then refer to the file "/etc/ssl/certs/CAs_COMMON_NAME.pem" - where CAs_COMMON_NAM is the common name you used when you set up your CA. Do not use any other name you may originally have given your exported CA certificate (pem) files.

In our case we need the file name "anraconc-CA.pem". So, this is already consistent with the name which we chose at export time in the first article. However, this happened by chance, only. We could have chosen quite a different name for the pem-file. (Well, I knew about the pitfall 🙂 )

If you had given your file a name like "myCA-cacert.pem", you would experience some trouble after having placed it into "/etc/pki/trust/anchors" and referred to it in CA chain configuration settings for any service by "/etc/ssl/certs/myCA-cacert.pem". Despite the files name "myCA-cacert.pem" it would in our case appear in "/etc/ssl/certs" with the name "anraconc-CA.pem". It took me a little while to figure this out !

SSL-reconfiguration for OpenLDAP and missing rights for the system user "ldap"

After we have made our CA certificate permanent, we proceed to the reconfiguration of the OpenLDAP service. This is a bit more complicated due to two reasons: There are potentially more configuration files to take care of and there may appear a problem with read rights. In addition you have to think of the "sssd.service" for providing encrypted connections to clients for authentication services - here for LDAP-clients.

The first point depends a bit on your specific configuration. There may be a file "/etc/ldap.conf" (controlling local LDAP-clients). In my case I had to change the entry for "tls_cacertfile":

bind_policy soft
uri ldap://myserv.anraconc.de
base dc=anraconc,dc=de
nss_map_attribute uniqueMember member
ssl start_tls
pam_password exop
pam_filter objectClass=posixAccount
tls_cacertdir /etc/ssl/certs
tls_cacertfile /etc/ssl/certs/anraconc-CA.pem
....
...

However, your main configuration will probably reside in "/etc/openldap/". There is also a "/etc/openldap/ldap.conf", which we need to modify:

tls_cacert      /etc/ssl/certs/anraconc-CA.pem

Then we have in addition the central file "slapd.conf" which must get proper TLS/SSL entries :

...
TLSCACertificateFile /etc/ssl/certs/anraconc-CA.pem
TLSCertificateFile /etc/ssl/servercerts/servercert.pem
TLSCertificateKeyFile /etc/ssl/servercerts/serverkey.pem
...

Hint: On Opensuse systems the file "/etc/sysconfig/openldap" should also contain a line:

...
OPENLDAP_START_LDAP="yes"
...
## Type:           yesno
## Default:        no
## ServiceRestart: ldap
# If set to "yes" the "ldap over ssl" feature of slapd will be enabled. Don't
# forget to add the "TLSCertificateFile" and "TLSCertificateKeyFile" options 
# to the /etc/openldap/slapd.conf (man slapd.conf).
# Note: Don't confuse this with "START_TLS", the preferred method for 
#       making encrypted LDAP connections, which is enabled as soon as You
#       specify "TLSCertificateFile" and "TLSCertificateKeyFile" in your config
#       file
#
OPENLDAP_START_LDAPS="yes"
...

But setting the right value for "TLSCACertificateFile" in "/etc/openldap/ldap.conf" is not enough:

The SSL settings also play a decisive role in the file "cn=config.ldif" (in "/etc/openldap/slapd.d/") as the core of modern LDAP services is configured by ldif-files.

Warning:
When editing the file "cn= config.ldif" file you have to be extremely careful. Make a backup first! One normally should not edit this file directly!. But as our LDAP service is not running any more we are brave and use

myserv:/etc/openldap/slapd.d # vi cn\=config.ldif

(Note the escaping backslash.

Then you edit the following entries

...
olcTLSCACertificateFile: /etc/ssl/certs/anraconc-CA.pem
olcTLSCertificateFile: /etc/ssl/servercerts/servercert.pem
olcTLSCertificateKeyFile: /etc/ssl/servercerts/serverkey.pem
....

This may afterwards lead to some warnings in log files regarding the checksum for this file. You could correct this later on by using "ldapmodify" to reedit the entries again on the then (hopefully) running LDAP-server.

Unfortunately, a service restart leads to trouble:

myserv:/etc/openldap/slapd.d # rcslapd restart
Job for slapd.service failed because the control process exited with error code.
See "systemctl  status slapd.service" and "journalctl  -xe" for details.

myserv:/etc/openldap/slapd.d # rcslapd status
● slapd.service - OpenLDAP Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/slapd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2019-07-20 16:45:37 CEST; 45s ago
  Process: 13254 ExecStart=/usr/lib/openldap/start (code=exited, status=1/FAILURE)
 Main PID: 11608 (code=exited, status=0/SUCCESS)

Jul 20 16:45:37 myserv slapd[13254]: main: TLS init def ctx failed: -1
Jul 20 16:45:37 myserv slapd[13254]: slapd stopped.
Jul 20 16:45:37 myserv slapd[13254]: connections_destroy: nothing to destroy.
Jul 20 16:45:37 myserv start[13254]: Starting ldap-server
Jul 20 16:45:37 myserv systemd[1]: slapd.service: Control process exited, code=exited status=1
Jul 20 16:45:37 myserv systemd[1]: Failed to start OpenLDAP Server Daemon.
Jul 20 16:45:37 myserv systemd[1]: slapd.service: Unit entered failed state.
Jul 20 16:45:37 myserv systemd[1]: slapd.service: Failed with result 'exit-code'.

You see the message in the middle? main: TLS init def ctx failed: -1

Now, you could google. It will not help much! It took me some time to find out that I simply had a rights problem! I saw this when comparing the old contents of /etc/ssl/servercerts" from a backup with the new one:

myserv:/etc/ssl # la servercerts.orig
total 16
drwxr-xr-x  2 root root 4096 May 30  2013 .
drwxr-xr-x  7 root root 4096 Jul 20 15:03 ..
-rw-r--r--  1 root root 2155 Jul 10  2017 servercert.pem
-rw-------+ 1 root root 3272 Jul 10  2017 serverkey.pem
myserv:/etc/ssl # la servercerts
total 16
drwxr-xr-x 2 root root 4096 Jul 20 16:40 .
drwxr-xr-x 7 root root 4096 Jul 20 15:03 ..
-rw-r--r-- 1 root root 2492 Jul 20 15:19 servercert.pem
-rw------- 1 root root 3243 Jul 20 15:22 serverkey.pem

You see the tiny difference? The "+" - sign in the original rights?
Oops! ACL-settings? What for?

myserv:/etc/ssl # getfacl servercerts.orig/serverkey.pem 
# file: servercerts.orig/serverkey.pem
# owner: root
# group: root
user::rw-
user:ldap:r--
group::r--
mask::r--
other::---

OK! lets try this for the user "ldap" in the new setup:

 
myserv:/etc/ssl # setfacl -m u:ldap:r servercerts/serverkey.pem 
myserv:/etc/ssl # setfacl -m g::- servercerts/serverkey.pem
myserv:/etc/ssl # getfacl servercerts/serverkey.pem 
# file: servercerts/serverkey.pem
# file: servercerts/serverkey.pem
# owner: root
# group: root
user::rw-
user:ldap:r--
group::---
group:root:---
mask::r--
other::---

myserv:/etc/ssl # systemctl restart slapd.service
myserv:/etc/ssl # systemctl status slapd.service
● slapd.service - OpenLDAP Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/slapd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-07-20 17:00:21 CEST; 5s ago
  Process: 15869 ExecStart=/usr/lib/openldap/start (code=exited, status=0/SUCCESS)
 Main PID: 15889 (slapd)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/slapd.service
           └─15889 /usr/sbin/slapd -h ldap:/// ldaps:/// ldapi:/// -F /etc/openldap/slapd.d -u ldap -g ldap -o slp=off

Jul 20 17:00:21 myserv systemd[1]: Stopped OpenLDAP Server Daemon.
Jul 20 17:00:21 myserv systemd[1]: Starting OpenLDAP Server Daemon...
Jul 20 17:00:21 myserv slapd[15869]: @(#) $OpenLDAP: slapd 2.4.46 $
                                          opensuse-buildservice@opensuse.org
Jul 20 17:00:21 myserv slapd[15869]: ldif_read_file: checksum error on "/etc/openldap/slapd.d/cn=config.ldif"
Jul 20 17:00:21 myserv slapd[15869]: ldif_read_file: checksum error on "/etc/openldap/slapd.d/cn=config/olcDatabase={0}config.ldif"
Jul 20 17:00:21 myserv slapd[15889]: slapd starting
Jul 20 17:00:21 myserv start[15869]: Starting ldap-server
Jul 20 17:00:21 myserv systemd[1]: Started OpenLDAP Server Daemon.
myserv:/etc/ssl # 

Obviously, the user "ldap" needs read rights on the (private) key of the server certificate!
Note that the user which runs the slapd.service may have a different name on your system - e.g. "openldap".

One can dispute whether ACLs is a reasonable way to handle this problem. Of course you could try and do something else. For other services I found in backups for older SuSE installations a special group "tls" which contained root and system users which needed read rights. This group was then assigned to "/etc/ssl/servercerts/serverkey.pem".

E.g. in case you are concerned about any of these solutions you could create a separate special certificate for the LDAP service and place the key in a special directory with read access rights for the file limited to root and the user "ldap". Of course, in this case, the service's configuration files must be adjusted for all TLS/SSL relevant data. However, before you follow such an approach of different certificates for services on one and the same server, think a bit about side consequences. As the FQDN of the server has to be identical to the certificate's common name you would have to issue certificates for different common names of your various server services.
This in turn makes your network and DNS configuration more complicated - you either need to associate one IP with multiple FQDNs or provide multiple IPs for one and the same server. To avoid such trouble I would rather combine this with a container approach for each separate service.

Adjusting the configuration of the SSSD-service

Having a running LDAP service does not mean that external clients can access it. On modern systems the exchange of auth-information between clients and server services is often handled by the "sssd"-service. This is true in my case, too. Therefore, a running slapd will not be sufficient if we use the LDAP-service for authentication from some clients in the network. We need to adjust the sssd-configuration, too. The relevant file is "/etc/sssd/sssd.conf" :

...
[sssd]
config_file_version = 2
services = nss,pam
domains = default
# SSSD will not start if you do not configure any domains.
# Add new domain configurations as [domain/<NAME>] sections, and
[nss]
filter_groups = root
filter_users = root
[pam]
# Section created by YaST
[domain/default]
ldap_uri = ldap://myserv.anraconc.de
ldap_search_base = dc=anraconc,dc=de
ldap_schema = rfc2307bis
id_provider = ldap
ldap_user_uuid = entryuuid
ldap_group_uuid = entryuuid
;ldap_id_use_start_tls = True
ldap_id_use_start_tls = False
enumerate = True
chpass_provider = ldap
auth_provider = ldap
cache_credentials = False

ldap_tls_cacertdir = /etc/ssl/certs
ldap_tls_cacert = /etc/ssl/certs/anraconc-CA.pem
....
...

Again, the most important settings are done in the last 2 lines displayed.
The setting for start_tls is up to you. If you use "start_tls" then the OpenLDAP configuration must provide the necessary entries.

Note: You have to provide similar sssd-settings on external LDAP-client-systems! One such client could be an email/imap-server - we shall have a look at its configuration in the next article.

On "myserv" we restart the "sssd.service":

myserv:/etc/ssl # vi /etc/sssd/sssd.conf
myserv:/etc/ssl # rcsssd restart
myserv:/etc/ssl # systemctl status sssd.service 
● sssd.service - System Security Services Daemon
   Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-07-20 17:29:11 CEST; 18s ago
 Main PID: 20021 (sssd)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/sssd.service
           ├─20021 /usr/sbin/sssd -i --logger=files
           ├─20022 /usr/lib/sssd/sssd_be --domain default --uid 0 --gid 0 --logger=files
           ├─20023 /usr/lib/sssd/sssd_nss --uid 0 --gid 0 --logger=files
           └─20024 /usr/lib/sssd/sssd_pam --uid 0 --gid 0 --logger=files

Jul 20 17:29:10 myserv systemd[1]: Starting System Security Services Daemon...
Jul 20 17:29:10 myserv sssd[20021]: Starting up
Jul 20 17:29:11 myserv sssd[be[20022]: Starting up
Jul 20 17:29:11 myserv sssd[20024]: Starting up
Jul 20 17:29:11 myserv sssd[20023]: Starting up
Jul 20 17:29:11 myserv systemd[1]: Started System Security Services Daemon.

Conclusion

We can use TinyCA2 to replace the missing YaST-feature for an effective administration both of CAs and server certificates. The major work has to be done in adjusting the configuration of the services. But after the method for installing a CA certificate server wide and permanently became clear the service configuration became reduced to a more or less dumb editing job. This job could be delegated to a script; but be careful to check the specific names of the relevant SSL/TLS entries for each service!

A special challenge came with the LDAP service; the related system user must get read rights to the used server certificate's private key!

In the next and final article of this series

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – III

we shall have a look at the reconfiguration of a cyrus and posfix service on a different server "mymail", which would then be a client for the LDAP service on "myserv" (for the authentication of IMAP/SMTP-users). .

Links

Server wide and permanent installation of a trusted CA's certificate
stackexchange.com: how-do-i-install-a-system-wide-ssl-certificate-on-opensuse

Opensuse Leap 42.1, Apache 2.4, Nagios-Modul und veraltete Access Right Syntax

Vor kurzem hatte ich das zweifelhafte Vergnügen, einen virtualisierten LAMP-Server von Opensuse 13.1 auf Opensuse Leap 42.1 zu hieven. Das ging zu meiner Überraschung weitgehend problemfrei über die Bühne. Eine kleine Besonderheit gab es aber schon.

Problem mit Apache 2.4 und der Konfiguration des Nagios-Moduls

Das Problem, dass bei der Umstellung auftauchte, war der Übergang des Apache-Servers zu einer neuen Version (2.2 => 2.4). Auf dem alten Server lief auch das Nagios-Modul für Apache mit. Nach dem Upgrade auf Opensuse Leap 42.1 verweigerte Apache den Start.

lamp:~ # systemctl start apache2.service 
Job for apache2.service failed. See "systemctl status apache2.service" and "journalctl -xn" for details.
xlamp:~ # systemctl start apache2.service 
Job for apache2.service failed. See "systemctl status apache2.service" and "journalctl -xn" for details.

 
und

lmp:~ # systemctl status apache2.service 
apache2.service - The Apache Webserver
   Loaded: loaded (/usr/lib/systemd/system/apache2.service; disabled)
   Active: failed (Result: exit-code) since Wed 2016-10-19 17:47:09 CEST; 37s ago
  Process: 2279 ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful-stop (code=exited, status=1/FAILURE)
  Process: 2272 ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start (code=exited, status=1/FAILURE)
 Main PID: 2272 (code=exited, status=1/FAILURE)

Oct 19 17:47:09 lmp start_apache2[2272]: AH00526: Syntax error on line 15 of /etc/apache2/conf.d/nagios.conf:
Oct 19 17:47:09 lmp start_apache2[2272]: Invalid command 'Order', perhaps misspelled or defined by a module ...ation
Oct 19 17:47:09 lmp systemd[1]: apache2.service: main process exited, code=exited, status=1/FAILURE
Oct 19 17:47:09 lmp start_apache2[2279]: AH00526: Syntax error on line 15 of /etc/apache2/conf.d/nagios.conf:
Oct 19 17:47:09 lmp start_apache2[2279]: Invalid command 'Order', perhaps misspelled or defined by a module ...ation
Oct 19 17:47:09 lmp systemd[1]: apache2.service: control process exited, code=exited status=1
Oct 19 17:47:09 lmp systemd[1]: Failed to start The Apache Webserver.
Oct 19 17:47:09 lmp systemd[1]: Unit apache2.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.

 

Problemlösung 1 - Korrektur des Files nagios.conf

Die Ursache dieses Problems liegt in der Anweisungssyntax der Konfigurationsdatei zum Nagios-Modul und nicht im Nagios-Modul selbst. Die Anweisung zu den Zugangsrechten für Webserver-Verzeichnisse sind zwar noch für Apache-Versionen < 2.4 gültig; nicht aber für die Version 2.4.

Nehmen wir als Beispiel die Direktiven zu einem Verzeichnis aus der Datei "/etc/apache2/conf.d/nagios.conf":

   
<Directory "/usr/lib/nagios/cgi">
#  SSLRequireSSL
   Options ExecCGI
   AllowOverride None
   Order allow,deny
   Allow from all
#  Order deny,allow
#  Deny from all
#  Allow from 127.0.0.1
   AuthName "Nagios Access"
   AuthType Basic
   AuthUserFile /etc/nagios/htpasswd.users
   Require valid-user
</Directory>

Hier werden die Zugangsrechte in klassischer Manier durch die Statements

   Order allow,deny
   Allow from all

geregelt. Diese Statements sind unter Apache 2.4 aber nicht mehr zu verwenden!

Nähere Informationen zur neuen Syntax liefert die Dokumentation unter:
https://httpd.apache.org/docs/2.4/upgrading.html

Die obige Sequenz ist - falls man die Zugriffsberechtigung tatsächlich so handhaben will - durch das Statement

Require all granted

zu ersetzen. Also

   
<Directory "/usr/lib/nagios/cgi">
#  SSLRequireSSL
   Options ExecCGI
   AllowOverride None
#	
   Require all granted 
   #   Order allow,deny
   #   Allow from all
#
#  Order deny,allow
#  Deny from all
#  Allow from 127.0.0.1
   AuthName "Nagios Access"
   AuthType Basic
   AuthUserFile /etc/nagios/htpasswd.users
   Require valid-user
</Directory>

Die auskommentierte Sequenz

#  Order deny,allow
#  Deny from all
#  Allow from 127.0.0.1

wäre übrigens zu ersetzen durch

Require host 127.0.0.1

Führt man die vorgeschlagenen Änderungen für alle Directories in der "nagios.conf" durch, so läuft der Neustart u.U. erfolgreich. In meinem Fall war das nicht so, da ich natürlich auch in anderen Konfigurationsfiles (u.a. zu Webdomainen) weitere Access Right Statements mit alter Syntax eingefügt hatte. Hat man auch die dortigen Fehler korrigiert, läuft Apache2 wieder.

Problemlösung 2 - Nutzung eines Kompatibilitätsmoduls

Die Seite https://httpd.apache.org/docs/2.4/upgrading.html empfiehlt, wenn möglich, alle Änderungen auf einen Schlag durchzuführen. Das kann je nach Apache-Host und selbst eingeführten Zugangsrechten für die implementierten Web-Domainen aber aufwändig werden.

Ist einem diese Arbeit erstmal zu viel und will man nach dem Upgrade zunächst nur den Start des Apache-Servers trotz der neuen Syntax sicherstellen, so kann man das Modul "mod_access_compat" zu den Modulen hinzufügen, die der Apache-Server laden soll.

Hierzu benutzt man z.B. das Kommando "a2enmod" (s. etwa http://www.sysadminslife.com/linux/aktivierte-apache-module-anzeigen-deaktivieren-aktivieren/.
Unter Opensuse kann man die Module aber auch in einer entsprechenden Zeile der Datei "/etc/sysconfig/apache2" hinterlegen; also z.B.:

APACHE_MODULES="actions alias auth_basic authn_file authz_host authz_groupfile authz_user autoindex cgi dir env expires include log_config mime mod_access_compat mod_rewrite negotiation setenvif ssl userdir php5 reqtimeout authn_core authz_core version"

Viel Spaß weiter beim Betrieb von Apache!