Upgrade of a server host from Opensuse Leap 15.1 to 15.2 – I – a problem with Apache2/MPM and PHP7

The official support of Opensuse Leap 15.1 is about to end. Time to upgrade some servers to Opensuse Leap 15.2 in my home-office. While the basic upgrade of the OS was a smooth process I stumbled across some problems and challenges with a few services. I shall cover these problems step by step in this series - and hope that my remarks and hints will help some of my readers.

A central server system with network related services and with virtualized guests

One of the two central server systems in my LAN provides DHCP-, DNS-, OpenLDAP-services plus an internal HTTP-service (based on Apache2) for development work. Its list of services in addition comprises NFS, Samba and some SVN-services for multiple purposes, among others as backup space for other systems. The host also works as a CUPS server - offering some pre-defined queues for other Linux-systems in the LAN.

The system also serves as a KVM- and LXC-host for some virtualization guests: A KVM based LAMP/GIT-server for production data, a KVM based mail-server, a KVM based Kali-system plus some LXC-containers for databases. Some LVM volumes on this server are encrypted and all disk-space is based on Raid configurations - both with mdadm for different SSD-raids and with a 3ware controller for a standard hard disk based raid10. This server is also critical as it separates its internal virtual network structure from real LAN segments by a packet-filter. It must cover routing tasks.

I run this Linux host with Opensuse Leap - not SLES. So, it is a good test object for the cleanliness of Opensuse's upgrade practices. Actually, I am always a bit nervous during upgrade. Fortunately, the HW is from the time of my freelance business - and thus safe and proven to use. It contains a passively cooled Nvidia card, for which the community repositories still provides kernel modules. The 3ware drivers also have worked reasonably well over the last 4 years. So, drivers at least should not pose any problems.

To make a long story short: The basic upgrade did not cost me much headache. I outline the upgrade procedure below - and then turn to the problems I have seen in its wake.

Backup and basic upgrade procedure

It should be clear that a server as the described one requires some regular backup routine for important data. I shall not elaborate on this topic here. Regarding the backup of the "/"-partition I use an approach described in the article
Upgrade workstation from Opensuse Leap 15.1 to Leap 15.2:

I.e. I copied the whole partition to a different, external hard-disk and afterwards in addition to a free partition on the server system itself - all with the help of the "dd"-command. You may do the same with other partitions which you want to re-create quickly in case of an emergency during upgrade.

Regarding the upgrade itself I cling to the procedure described by the Linux Kamarada; see the article mentioned above. (The required work with the Opensuse repositories is much simpler for my servers than for my workstations and laptops as I use much less repositories; the only repositories aside the Update repository for the present Leap version are SUSE's "Security" and "Network" repositories plus the Nvidia community repository. I have made good experiences with this habit over the last 10 years.)

The upgrade worked almost perfectly. Some quick checks verified that my virtualized (and not yet upgraded) virtual guest systems could be started and worked as expected. The same was true for the slapd-, smb-, the wsdd- and the dns- (=named-) service.

But, there were also exceptions which required additional work:

  • Graphics Failure: The X-system did not start and the "graphical target" of systemd could not be reached directly after the system upgrade (expected).
  • Service failure: The Apache2 service did not start (unexpected).
  • Service failure: The smartd-service did not start (unexpected, but probably due to an overwritten configuration file).
  • Required Adjustment: The CUPS libraries had to be changed to Opensuse's "Printing" repository to become consistent with other Linux hosts which use printer queues on the server (expected).
  • Required Adjustment: The rkhunter-configuration had to be adjusted (expected).
  • Required Adjustment: A specific KDE-settings had to be adjusted to allow for screen-saving functions - even for SDDM screens without started KDE-sessions (unexpected).
  • Required Adjustment: "virt-viewer" access via "qemu+ssh" to some clients posed grave problems (unexpected). These problems may have existed already before, but are worth a description.

I cover the first 2 points in this article. Solving the graphics problem is a short story. The explanation and solution of the problem with the Apache service, however, requires some space. But I hope the information provided in the related text is interesting for some readers who are no experts in Opensuse's handling of the Apache service.

Nvidia - X-Windows

You may ask: Why X11 on a server at all? Good question; well, as a semi-professional in server-administration, I think that one can do some things faster than without. Normally, I access the server remotely via ssh or "ssh -X". But during my years with SLES and Opensuse I experienced some situations during which I was happy to be able to access the server directly on a graphical console; certainly not a popular view among profs. Anyway - the Nvidia/X-Window topic is a standard problem and easy to solve:

I just had to reinstall the required drivers from the Nvidia repository (via the CLI-based YaST), perform a "mkinitrd" afterwards and restart the system.

However, I nevertheless recommend not to boot the server system into the graphical systemd target, if not required. The relevant command to set the default target is

systemctl set-default multi-user.target

See the related link at the end of this post.

Off topic: If you prefer KDE as a graphical desktop environment do not even try to switch to Wayland on Leap 15.2. In my experience this is only a frustrating endeavor - as we still do not have Plasma 5.20 available but Plasma 5.18 (see a related link at the end).

Apache does not start due to a problem with "mod_php7" and MPM

Let us turn to something more interesting: An Apache problem with the current PHP-module and MPM-settings.

I admit, I am a bit old-fashioned. Most developers and admins, today, probably use Nginx and in addition Fast-CGI in combination with PHP. I, instead, cling to an old Apache installation for LAMP-services on the server named above. One reason is that I need some compatibility with things I did from 2013 and 2016 - aside of changes of PHP over the years.

So, whilst upgrading Opensuse my Apache 2.4 and PHP installations get upgraded, too.

When one activates the php-Apache-module by "a2enmod php" the presently installed version of the php-Apache-module gets loaded - on Leap 15.2 this is provided by mod_php7.so corresponding to PHP 7.4.6. (You can verify this with the command "php -v"). Using "a2enmod php7" would give you no different result. (Opensuse, unfortunately, does not support the parallel installation of multiple, different PHP-versions. See the critics in the conclusion-section.)

All upgrades from Leap 42.2 to 42.3, 15.0 and eventually to Leap 15.1 went well; but this time the Apache service did not start. Trying "rcapache2 start" followed by "rcapache2 status" (or equivalent systemctl status apache2.service) gave me an error message:

....
[php7:crit] [pid 16004:tid 139738320386304] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe. You need to recompile PHP.
Jan 26 19:15:39 MySRV start_apache2[16004]: AH00013: Pre-configuration failed
Jan 26 19:15:39 MySrv systemd[1]: apache2.service: Main process exited, code=exited, status=1/FAILURE
....

I remembered vaguely that I had experimented with MPM and prefork settings on Leap 15.1, but recompile PHP? With what options? A strange error message!

Confusion when comparing to a seemingly similar Apache2 installation on a different system

I cross-checked with a similarly configured Apache2 installation (with PHP7, but different virtual domains) on one of my my Leap 15.2 workstations; let us call it "MyWS" and the upgraded server "MySRV". Confusingly, Apache2 started on MyWS without any complaints.

You may suspect now that I may have used different Apache2 configuration directives in "/etc/sysconfig/apache2" or the files in folder "/etc/apache2/" and sub-folders - without being aware of it. But no: I checked and eliminated even small, irrelevant differences for tests. The problem on the server MySRV remained.

When I looked at the Apache2-related processes on the workstation MyWS with "ps aux | grep https" I saw that Apache2 had been started in a pre-forked way. As I was used to .... I got multiple forked processes like

....
wwwrun    7043  0.0  0.0 236180 10488 ?        S    10:41   0:00 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include .... 
.....
wwwrun    7045  0.0  0.0 236180 10488 ?        S    10:41   0:00 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include ... 
... 

The interesting thing here is the indicated program "/usr/sbin/httpd-prefork". Note that it is NOT "usr/bin/httpd"; I shall come back to this point later. The process information was of course consistent with startup information provided by "rcapache2 status" or equivalently by "systemctl status apache2".

It took me a while to find out what was going on and what I had to do. Whilst searching the Internet and experimenting with Apache I passed some interesting information about details of the Apache-installation on Opensuse which may be interesting for others, too.

MPM methods on Apache 2.4

The error message indicated that the whole problem had to do with MPMs - "Multi-Processing-Modules", i.e. the ability of Apache to handle a multitude of requests concurrently. There are 3 MPM-methods and respective pre-compiled static modules or loadable DSO modules available with Apache 2.4. See e.g.:

https://www.digitalocean.com/community/tutorials/how-to-configure-apache-http-with-mpm-event-and-php-fpm-on-ubuntu-18-04-de,
http://httpd.apache.org /docs /current /en /mpm.html
and links therein.

I quote:

  • Pre-fork: A new process is created for each incoming connection reaching the server. Each process is isolated from the others, so no memory is shared between them, even if they are performing identical calls at some point in their execution. This is a safe way to run applications linked to libraries that do not support threading—typically older applications or libraries.
  • Worker: A parent process is responsible for launching a pool of child processes, some of which are listening for new incoming connections, and others are serving the requested content. Each process is threaded (a single thread can handle one connection) so one process can handle several requests concurrently. This method of treating connections encourages better resource utilization, while still maintaining stability. This is a result of the pool of available processes, which often has free available threads ready to immediately serve new connections.
  • Event: Based on worker, this MPM goes one step further by optimizing how the parent process schedules tasks to the child processes and the threads associated to those. A connection stays open for 5 seconds by default and closes if no new event happens; this is the keep-alive directive default value, which retains the thread associated to it. The Event MPM enables the process to manage threads so that some threads are free to handle new incoming connections while others are kept bound to the live connections. Allowing re-distribution of assigned tasks to threads will make for better resource utilization and performance.

At http://httpd.apache.org/docs/trunk/misc/perf-scaling.html we furthermore read:

Httpd Configuration
The Apache 2.2 httpd is by default a pre-forking web server. When the server starts, the parent process spawns a number of child processes that do the actual work of servicing requests. But Apache httpd 2.0 introduced the concept of the Multi-Processing Module (MPM). Developers can write MPMs to suit the process- or threading- architecture of their specific operating system. Apache 2 comes with special MPMs for Windows, OS/2, Netware and BeOS. On unix-like platforms, the two most popular MPMs are Prefork and Worker. The Prefork MPM offers the same pre-forking process model that Apache 1.3 uses. The Worker MPM runs a smaller number of child processes, and spawns multiple request handling threads within each child process. In 2.4 MPMs are no longer hard-wired. They too can be exchanged via LoadModule. The default MPM in 2.4 is the event MPM.
....
The maximum number of workers, be they pre-forked child processes or threads within a process, is an indication of how many requests your server can manage concurrently. It is merely a rough estimate because the kernel can queue connection attempts for your web server. When your site becomes busy and the maximum number of workers is running, the machine doesn't hit a hard limit beyond which clients will be denied access. However, once requests start backing up, system performance is likely to degrade.

Finally, if the httpd server in question is not executing any third-party code, via mod_php, mod_perl or similar, we recommend the use of mpm_event. This MPM is ideal for situations where httpd serves as a thin layer between clients and backend servers doing the real job, such as a proxy or cache.

...
Selecting your MPM
The prime reason for selecting a threaded MPM is that threads consume fewer system resources than processes, and it takes less effort for the system to switch between threads. This is more true for some operating systems than for others. On systems like Solaris and AIX, manipulating processes is relatively expensive in terms of system resources. On these systems, running a threaded MPM makes sense.

On Linux, the threading implementation actually uses one process for each thread. Linux processes are relatively lightweight, but it means that a threaded MPM offers less of a performance advantage than in other environments.

Running a threaded MPM can cause stability problems in some situations For instance, should a child process of a preforked MPM crash, at most one client connection is affected. However, if a threaded child crashes, all the threads in that process disappear, which means all the clients currently being served by that process will see their connection aborted. Additionally, there may be so-called "thread-safety" issues, especially with third-party libraries. In threaded applications, threads may access the same variables indiscriminately, not knowing whether a variable may have been changed by another thread.

This has been a sore point within the PHP community. The PHP processor heavily relies on third-party libraries and cannot guarantee that all of these are thread-safe. The good news is that if you are running Apache on Linux, you can run PHP in the preforked MPM without fear of losing too much performance relative to the threaded option.

Ok, what do we learn from this? php_mod is not thread-safe. But regarding performance, using the "pre-fork" MPM method is not really a bad choice on a Linux system. Obviously, this method was used on the workstation. An interesting question, which I come back to later, is whether Opensuse offers dynamically loadable or static MPM modules via its repositories.

Information about the MPM used by your installation?

One (naively) expects that the Apache2 daemon binary should be callable by "/usr/sbin/httpd" on an Opensuse system. Actually, the command "which httpd" gives you "/usr/sbin/httpd" as an answer.

On the Internet you find hints that one can check the compiled and available MPMs by either "httpd -V" and/or "httpd -l". On the upgraded server MySRV I got:

MySRV:~ # httpd -V
Server version: Apache/2.4.43 (Linux/SUSE)
Server built:   2020-11-17 11:05:32.000000000 +0000
Server's Module Magic Number: 20120211:92
Server loaded:  APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
....

and

MySRV:~ # httpd -l
Compiled in modules:
  core.c
  mod_so.c
  http_core.c
  worker.c
  mod_unixd.c
  mod_systemd.c

These commands seemingly worked on my server even if the Apache2-service itself could not be started (e.g. by "systemctl start apache2"). The man-pages for "httpd" say that the "-l"-option lists modules which were compiled into the server, i.e. static modules.

What did I get on the workstation?

MyWS:~ # httpd -V
Server version: Apache/2.4.43 (Linux/SUSE)
Server built:   2021-01-27 09:11:10.000000000 +0000
Server's Module Magic Number: 20120211:92
Server loaded:  APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)

and

MyWS:~ # httpd -l
Compiled in modules:
  core.c
  mod_so.c
  http_core.c
  prefork.c
  mod_unixd.c
  mod_systemd.c

Regarding the actually running processes after Apache-startup on the workstation this result is not too surprising. However, if the listed modules are static then the "httpd" binary on the server must be another one than on the workstation. Why? And how did this come about ?

Another question is: How much can we rely on the information given by e.g. "httpd -l" in the sense that this information reflects facts about the running Apache2 service? We shall see that the answer depends ...

Removing the PHP-module on both installations

To analyze further I first aligned the list of modules on both systems via the corresponding setting in "/etc/sysconfig/apache2" to:

APACHE_MODULES="actions alias auth_basic authn_file authz_host authz_groupfile authz_core authz_user autoindex cgi dir env expires include log_config mime rewrite negotiation setenvif ssl socache_shmcb userdir reqtimeout php7 authn_core version"

and then removed the "php7" module.

(Instead of editing the file "/etc/sysconfig/apache2" directly you could also use a2enmod or a2dismod, respectively, for changing the module configuration. But I recommend to keep some commented alternatives for in the file for fast changes between repeated tests of different constellations).

In addition I used the seemingly standard Opensuse setting of

APACHE_MPM=""

in "/etc/sysconfig/apache2" for both installations.

Well, as expected no significant changes occurred on the workstation. Apache2 started successfully again (without PHP support); multiple prefork-processes were launched. On the server, however, I now got a successful start, too; "rcapache2 status" showed me a bunch of related processes afterwards:

MySRV:~ # rcapache2 status
● apache2.service - The Apache Webserver
   Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; vendor preset: >
   Active: active (running) since Sun 2021-01-31 11:45:26 CET; 23s ago
  Process: 8555 ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful->
 Main PID: 20667 (httpd-worker)
   Status: "Processing requests..."
    Tasks: 83
   CGroup: /system.slice/apache2.service
           ├─20667 /usr/sbin/httpd-worker -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid ....
           ├─20675 /usr/sbin/httpd-worker -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid ....
           ├─20676 /usr/sbin/httpd-worker -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid ....
           ├─20677 /usr/sbin/httpd-worker -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid ....
           └─20678 /usr/sbin/httpd-worker -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid ....

Jan 31 11:45:26 MySRV systemd[1]: Starting The Apache Webserver...
Jan 31 11:45:26 MyServ start_apache2[20667]: AH00548: NameVirtualHost has no effect and>
Jan 31 11:45:26 MySRV systemd[1]: Started The Apache Webserver.

But you see the difference?
=> The programs started were of the type "/usr/bin/http-worker" !

Looking at the startup scripts for the Apache2 service

To dig a bit deeper I looked at the startup scripts used on Opensuse for the Apache2 service. One reason to do so was a comment in Opensuse's "/etc/sysconfig/apache2" regarding the variable APACHE_MPM:

# MPM (multi-processing module) to use.
#
# Needed to determine with which MPM apache will run, as well as
# against which header files modules will be built. 
#
# If not set, the system will simply pick one of the installed MPMs.
#
# The implementation of the logic is in /usr/share/apache2/find_mpm,
# a script which can be used standalone as well if needed.

This indicates that some special Opensuse logic is used during startup. But the comment is also misleading: A script "find-mpm" can only be found in the folder "/usr/share/apache2/deprecated-scripts" on a Leap 15.1 and 15.2 systems - and the "script" itself is not used any more, as we shall see in a minute.

So, where do we find the startup-script for Apache? We do not have to look into the systemd configuration for the Apache2 service; the output of the command "rcapache2 status" already revealed it as
"/usr/sbin/start_apache2".

In it's script-code we find a line for the execution of a helper script:

 . /usr/share/apache2/script-helpers

and later a call to a function "find_mpm", followed by some logic for the name of a binary to be saved in the variable "apache_bin":

#
# figure out correct apache2 binary (/usr/sbin/httpd-prefork,
# /usr/sbin/httpd-worker, etc.) and serverflags
#
find_mpm
if [ -n "$HTTPD_MPM" ]; then
    apache_bin="$HTTPD_SBIN_BASE-$HTTPD_MPM"
    if ! [ -x $apache_bin ]; then
        echo >&2 "$apache_bin-$APACHE_MPM is not a valid httpd binary."
        echo >&2 "Check your APACHE_MPM setting in /etc/sysconfig/apache2."
        exit 1
    fi
else
  echo >&2 "${warn}No Apache binary found. No MPM package installed? $norm"
  echo >&2 "Hint: install the apache2-prefork package, and try again."
  exit 1
fi

Obviously, we will find the function "find_mpm" in "/usr/share/apache2/script-helpers". A look into this file shows that this really is the case. The outcome of this function influences the startup of Apache2; obviously, a different Apache variant is started for different (valid) values of the script-variable "HTTPD_MPM":

apache_bin="$HTTPD_SBIN_BASE-$HTTPD_MPM"

($HTTPD_SBIN_BASE resolves to "/usr/sbin/httpd" - this is set at the beginning of "/usr/share/apache2/script-helpers".)

Now, you can easily build your own test-script "mpm_test" including this function and print its output:

mpm_test-script:

#!/bin/bash

HTTPD_SBIN_BASE="/usr/sbin/httpd"
#
# loads sysconfig variables into environment
# return value in: APACHE_*
function load_sysconfig
{
  [ -n "$sysconfig_loaded" ] && return
  [ ! -f "$SYSCONFIG_FILE" ] && return

  . $SYSCONFIG_FILE

  export ${!APACHE_*} sysconfig_loaded=true
}
#
# finds prefered multiprocessing module
# return value in: HTTPD_MPM 
function find_mpm
{
  # load sysconfig variables if they weren't yet;
  # this has no effect when find_mpm is not called
  # from start_apache2
  load_sysconfig

  # try to read from sysconfig's APACHE_MPM
  HTTPD_MPM="$APACHE_MPM"
  # if empty, then choose one from installed
  if [ -z "$HTTPD_MPM" ]; then
      installed_mpms=""
      for i in $HTTPD_SBIN_BASE-*; do
          test -f $i || continue
          i=$(basename $i)
          i=${i#*-}
          installed_mpms="$installed_mpms $i"
      done
      # hardcoded preference here:
      for mpm in event worker prefork; do
        if [[ $installed_mpms =~ "$mpm" ]]; then
          HTTPD_MPM=$mpm
          break
        fi
      done
  fi

  # in case no 
  export HTTPD_MPM
}

# ***********************
# Action
# ***********************
find_mpm
echo $HTTPD_MPM

This script produced the following output on my server MySRV (with a running Apache2 without PHP support):

worker

On my workstation (and a running Apache2 with and without PHP support) I got:

prefork

Available and different Apache binaries

Analyzing the logic of our file, which is the same as in the beginning of the startup-script, we see that
"apache_bin" can become either "/usr/sbin/httpd-prefork" or "/usr/sbin/httpd-worker". It depends on the value of the variable "APACHE_MPM" in "/etc/sysconfig/apache2":

  • If the variable "APACHE_MPM" is not empty its value is used.
  • But if it is empty the result depends on the existence of one or multiple of the 3 files "httpd-event", "httpd-worker" or "httpd-prefork" in the folder "/usr/sbin".

You also see that the logic, which the guys from Opensuse implemented, chooses a MPM-specific variant of Apache2 according to the following fixed priority order:

event > worker > prefork    !

The "prefork" variant has the least priority - and is obviously only chosen if neither the file "httpd-worker" nor the file "httpd-event" exist - and if "APACHE_MPM" is not set otherwise.

A quick check showed that binaries "/usr/sbin/httpd-worker", "/usr/sbin/httpd-event" did not exist on my workstation MyWS; there, only "/usr/sbin/httpd-prefork" was available. But on the server MySRV I actually had two variants available "/usr/sbin/httpd-prefork" and "/usr/sbin/httpd-worker".

This fact together with the "find_mpm"-logic explained the different start-up results for Apache2 on my systems MySRV and MyWS !

But not the original cause and why Apache had worked on MySRV with Leap 15.1 before ....

Differences in the package installation?!

The existence of different binaries on MyWS and the server MySRV could only be due to a package difference, which must have been there already before the upgrade to Leap 15.2. A check in my logs and also a look into the package constellation with YaST revealed that on the Leap 15.1 version of the server-system MySRV I actually had installed the RPM "apache2-worker" at some point in time! For some experiments which I had totally forgotten. But not on the workstation. So, I had caused the trouble somehow by myself - without anticipating any consequences for a coming upgrade to Leap 15.2.

But wait a second - this does not explain why I did not run into trouble with Apache on Leap 15.1. Despite the installed package "apache_worker" and an empty "APACHE_MPM" in the sysconfig file..... Why did Apache2 start on Leap 15.1 - as my old log-files show? With "httpd-prefork" processes, by the way ...

I had to suspect a difference in the "find_mpm" logic - if I really had left "Apache_MPM" empty on the server. The latter was the case as I saw in my backups. And indeed - on Leap 15.1 the logic of the function "find_mpm" in "/usr/share/apache2/script-helpers" was different:

The old "find_mpm" from Leap 15.1

function find_mpm
{
  [ -n "$mpm_found" ] && return

  # load sysconfig variables if they weren't yet
  load_sysconfig

  # try to read from sysconfig's APACHE_MPM
  HTTPD_MPM=$APACHE_MPM
  # if empty, then choose one from installed
  if [ -z "$HTTPD_MPM" ]; then
      # guess
      for i in $HTTPD_SBIN_BASE-*; do
          test -f $i || continue
          i=$(basename $i)
          i=${i#*-}
          installed_mpms=(${installed_mpms[*]} $i)
      done
      if [ -z "${installed_mpms[*]}" ]; then
          HTTPD_MPM=""
          return
      elif [ ${#installed_mpms[*]} = 1 ]; then
          HTTPD_MPM=${installed_mpms[*]}
      else
          case ${installed_mpms[*]} in
              *prefork*)      HTTPD_MPM=prefork;;
              *worker*)       HTTPD_MPM=worker;;
              *event*)        HTTPD_MPM=event;;
          esac
      fi
  fi

  export HTTPD_MPM mpm_found=true
}

A test shows that the "case"-part of this function gives us an answer "prefork" on a system with both the "prefork" and the "worker" variants of Apache installed!

This makes me wonder why, for heavens sake, the Opensuse guys did not insert a section into their Leap 15.2 Release Notes telling us something about this change!

The solution to my Apache/MPM/PHP problem

After all the information gathering above the very trivial solution to my start-up problem was as that I just had to set the variable "APACHE_MPM" in "/etc/sysconfig/apache2" to

APACHE_MPM="prefork"

when I wanted a successful start of an Apache instance with PHP support.

What do the http-* files in "/usr/sbin/" look like for multiple MPM-installations?

What happens if we install multiple MPM-variants of Apache? From some parts of the above text you have already concluded that we really must speak of "variants" as the MPM-module is compiled in statically by the Opensuse guys.

On the server MySRV with the worker-MPM active (no PHP loaded) we can see this by using the command "apachectl -M". See the man pages. The result is:

MySRV: ~# apachectl -M
Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 mpm_worker_module (static)
 unixd_module (static)
 systemd_module (static)
 actions_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authz_host_module (shared)
...
...
 

So, we expect multiple, relatively big variants of the "http"-binary under "/usr/sbin" whenever we install the packages "apache2-worker" and "apache2-event". Which I did for fun and step-wise on the workstation MyWS. And indeed - after the installation of the RPM "apache2-worker" we see:

MyWS:~ # la /usr/sbin/ | grep httpd
lrwxrwxrwx  1 root root           22 Jan 31 16:27 httpd -> /usr/sbin/httpd-worker
-rwxr-xr-x  1 root root       616816 Jan 27 16:17 httpd-prefork
-rwxr-xr-x  1 root root       629200 Jan 27 16:17 httpd-worker
lrwxrwxrwx  1 root root           23 Jan 30 11:02 httpd2 -> /usr/sbin/httpd-prefork
lrwxrwxrwx  1 root root           13 Jan 27 16:17 httpd2-prefork -> httpd-prefork
lrwxrwxrwx  1 root root           12 Jan 27 16:17 httpd2-worker -> httpd-worker

and after an additional installation of the "event" version of Apache2 we get:

MyWS:~ # la /usr/sbin/ | grep httpd
lrwxrwxrwx  1 root root           21 Jan 31 16:29 httpd -> /usr/sbin/httpd-event
-rwxr-xr-x  1 root root       641552 Jan 27 16:17 httpd-event
-rwxr-xr-x  1 root root       616816 Jan 27 16:17 httpd-prefork
-rwxr-xr-x  1 root root       629200 Jan 27 16:17 httpd-worker
lrwxrwxrwx  1 root root           23 Jan 30 11:02 httpd2 -> /usr/sbin/httpd-prefork
lrwxrwxrwx  1 root root           11 Jan 27 16:17 httpd2-event -> httpd-event
lrwxrwxrwx  1 root root           13 Jan 27 16:17 httpd2-prefork -> httpd-prefork
lrwxrwxrwx  1 root root           12 Jan 27 16:17 httpd2-worker -> httpd-worker

At the end of both installations a post-installation script is running, which creates the binaries, the "httpd"-link and the various "httpd2"-links.

These files/links do not change when we restart Apache2 with a different MPM-setting in "/etc/sysconfig/apache2".

Can we trust in "httpd -l"? What about "apachectrl -M"?

From the above information we must conclude that we cannot trust the information provided by "httpd -l" or "httpd -V" to give us reliable information for a running Apache2 service on Opensuse. It only shows us, what the most modern MPM among the available ones on the system is. Something similar is to say about "httpd2 -l"; it shows us the other end of the spectrum. Only, if we have just one MPM variant of Apache2 installed all information gets identical and consistent.

If you have a running Apache2 service, a really reliable information regarding the compiled in MPM module is given by "Apachectrl -M" or indirectly by "ps aux | grep httpd", which reveals the type of the httpd-binary variant started.

Conclusion

Apache2 is provided on Opensuse with static compiled in MPM modules. If you want to choose between different MPMs you have to install related packages; they provide different variants of the httpd-binary which you find afterwards in the directory "/usr/sbin".

The present Leap 15.2-specific startup-logic for the Apache-service is based an Opensuse-specific script-function "find_mpm" (witin the file "/usr/share/apache2/script-helpers"). According to its logic you MUST specify the MPM-variant via setting a variable in "/etc/sysconfig/apache2" if you want to or must break the fixed preference order "even > worker > prefork".

Opensuse changed the logic of its MPM selection in the function "find-mpm" between Leap 15.1 and Leap 15.2. This change may, unfortunately, block a successful start of an upgraded Apache2-service with PHP-support. (A bit of concise information on this point in the Release Notes would have been helpful.)

In case you use "mod_php" for PHP support by Apache2 and thus add "php7" to the list of Apache-modules to be loaded at start-up, you MUST set the "APACHE_MPM"-variable to "prefork" - at least if you follow the standard Opensuse concepts for the Apache service. The enforced choice is reasonable as PHP itself is not thread-safe.

Criticism, Opensuse Apache/PHP deficits and alternatives

As I criticize the handling of Apache2 on Opensuse I also want to add that I miss the flexibility other distributions offer regarding the parallel installation of multiple PHP-versions and a switch between them on the Apache-server. At least with a fast-cgi implementation.

I furthermore do not understand why Opensuse restricts the PHP version in their Update repository to just one and in the PHP-specific repository to the latest 2 versions. And even for these two PHP versions a developer is forced to install two virtual Opensuse machines with different Apache/PHP constellations, if he does not want to change the installed packages all the time.

The somewhat unflexible provision of Apache in combination with PHP and the MPM-logic may make you think about alternatives. There are various situations:

  • If you look beyond Apache I would recommend to setup Nginx.
  • If you look at a way to combine PHP with threaded MPM variants as "worker" or "event, try an installation based on FastCGI and PHP-FPM instead of using php_mod. But consider security aspects then.
  • If you look for support of multiple PHP-versions with Nginx and/or Apache install Debian or Ubuntu in a virtual machine (on a Opensuse host 🙂 ).

For those who like to experiment I have added some links in the last section of this post.

Outlook

In the next article I shall cover the startup-problem of the "smartd"-service. This will be a much shorter story.

Links

Links related to the Apache MPM-topic
http://httpd.apache.org /docs /current /en /mpm.html
https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-apache2.html
http://httpd.apache.org/docs/trunk/misc/perf-scaling.html

Check available MPM-method on Apache
https://www.binarytides.com/check-which-mpm-multi-processing-module-apache-is-running/

Links regarding an alternative FastCGI installation with PHP-FPM
https://en.opensuse.org/SDB:Apache_FastCGI_and_PHP-FPM_configuration
https://www.p4tchwork.de/apache-php-system-resource-usage/
https://blog.wappler.systems/opensuse-leap-15-apache-2-php-fpm-http2/

Flexible Apache and PHP installations on Debian derivatives
https://www.digitalocean.com/community/tutorials/how-to-configure-apache-http-with-mpm-event-and-php-fpm-on-ubuntu-18-04-de
https://www.epidemiology.tech/apache-php7-4-wordpress-setup/
https://tecadmin.net/setup-apache-php-fpm-ubuntu-20-04/
https://www.interserver.net/tips/kb/change-php-version-apache-ubuntu/

Links regarding Nginx
https://techviewleo.com/install-php-on-opensuse-nginx-apache/
https://www.tecmint.com/install-lemp-nginx-php-mariadb-phpmyadmin-in-opensuse/
https://websiteforstudents.com/install-php-7-4-php-7-4-fpm-on-ubuntu-18-04-with-apache2-nginx/

HTTP 2 installation on Opensuse
https://en.opensuse.org/SDB:Apache_HTTP_2_configuration

Set default target on Opensuse
https://forums.opensuse.org/showthread.php/526792-How-to-change-default-runlevel

Plasma, Nvidia and Wayland
https://community.kde.org/Plasma/Wayland/Nvidia

Deep Dreams of a CNN trained on MNIST data – III – catching dream patterns at smaller length scales

In the first two articles of this series on Convolutional Neural Networks [CNNs] and "Deep Dream" pictures

Deep Dreams of a CNN trained on MNIST data – II – some code for pattern carving
Deep Dreams of a CNN trained on MNIST data – I – a first approach based on one selected map of a convolutional layer

I introduced the concept of CNN-based "pattern carving" and a related algorithm; it consists of a small number of iterations over of a sequence of image manipulation steps (downscaling to a resolution the CNN can handle, detection and amplification of a map triggering pattern by a CNN, upscaling to the original size and normalization). The included pattern detection and amplification algorithm is nothing else than the OIP-algorithm, which I discussed in another article series on CNNs in this blog. The OIP-algorithm is able to create a pattern, which triggers a chosen CNN map optimally, out of a chaotic fluctuation background. The difference is that we apply such an OIP-algorithms now to a structured input image - the basis for a "dream". The "carving algorithm" is just a simplifying variation of more advanced Deep Dream algorithms; it can be combined with simple CNNs trained on gray and low-resolution images.

In the last article I provided some code which creates dream like pictures with the help of carving by a CNN trained on MNIST data. Nice, but ... In the original theory of "Deep Dreaming" people from Google and others applied their algorithms to a cascade of so called "octaves": Octaves represent the structures within an image at different levels of resolution. Unfortunately, at first sight, such an approach seems to be beyond the capabilities of our MNIST CNN, because the latter works on a fixed and very coarse resolution scale of 28x28 pixels, only.

As we cannot work on different scales of resolution directly: Can we handle pattern detection and amplification on different length scales in a different way? Can we somehow extend the carving process to smaller length scales and a possible detection of smaller map-triggering patterns within the input image?
The answer in short is: Yes, we can.
We "just" have to replace working with "octaves" by looking at sub-segments of the images - and apply our "carving" algorithm with up and down-scaling to these sub-segments as well as to the full image during an iteration process. We shall test the effects of such an approach in this article for a single selected sub-area of the input image, before we apply it more thoroughly to different input images in further articles.

A first step towards Deep Dreams "dreaming" detail structures of a chosen image ...

We again use the image of a bunch of roses, which we worked with in the last articles, as a starting point for the creation of a Deep Dream picture. The easiest way to define sub-areas of our (quadratic) input image is to divide it into 4 (quadratic) adjacent sub-segments or sub-images, each with half of the side length of the original image. We thus get two rows, each with two neighboring sub-areas of the size 280x280 px. Then we could apply the carving algorithm to one selected or to all of the sub-segments. But how do we combine such an approach with an overall treatment of the full image? The answer is that we have to work in parallel on both length-scales involved. We can do this within each cycle of down- and up-scaling according to the following scheme:

  • Step 1: Pick the input image IMG at the full working size (here 560 x 560 px) - and reshape it into a tensor suitable for Tensorflow 2 [TF2] functions.
  • Step 2: Down- and upscale the input image "IMG" (560x560 => 28x28 => 560x560) - with information loss
  • Step 3: Calculate the difference between the re-up-scaled image to the original image as a tensor for later detail corrections.
  • Step 4: Subdivide IMG into 4 quadrants (each with a size of of 280x280 px). Save the quadrants as sub-images S_IMG_1, .... S_IMG_4 - and create respective tensors.
  • Step 5: For all sub-images: Down- and up-scale the sub-images S_IMG_m (280x280 => 28x28 => 280x280) - with information loss.
  • Step 6: For all sub-images: Determine the differences between the re-upscaled sub-images and the original sub-images as tensors for later detail corrections.
  • Step 7: Loop A [around 4 to 6 iterations]
    • Step LA-1: Pick the down-scaled image IMG_d (of size 28x28 px) as the present input image for the CNN and the OIP-analysis
    • Step LA-2: Apply the OIP-algorithm for N epochs (20 < N < 40) to the downscaled image IMG_d (28x28)
    • Step LA-3: Upscale the resulting changed version of image IMG_d to the original IMG- size by bicubic interpolation => IMG_u
    • Step LA-4: Add the (constant) correction tensor for details to IMG_u.
    • (Step LA-5: Loop B over Sub-Images)
      • Step LB-1: Cut out the area corresponding to sub-image S_IMG_m from the changed full image IMG_u. Use it as Sub_IMG_m during the following steps.
      • Step LB-2: Downscale the present sub-image Sub_IMG_m to the size suitable for the CNN (here: 28x28) by bicubic interpolation => Sub_IMG_M_d.
      • Step LB-3: Apply the OIP-algorithm for N epochs (20 < N < 40) to the downscaled sub-image (28x28) Sub_IMG_m_d
      • Step LB-4: Upscale the changed Sub_IMG_m_d to the original size of Sub_Img_m by bicubic interpolation => Sub_IMG_m_u
      • Step LB-5: Add the (constant) correction tensor to the tensor for the upscaled sub-image Sub_IMG_m_u
      • Step LB-6: Replace the sub-image region of the upscaled full image IMG_u with the (corrected) sub-image Sub_IMG_m_u
      • Step LB-7: Downscale the new full image IMG_u again (to 28x28 => IMG_d) and use the resulting IMG_d for the next iteration of Loop A

The Loop B has been placed in brackets because we are going to apply the suggested technique only to a single one of the 4 sub-image quadrants in this article.

Code fragments for the sub-image preparation

The following code for a Jupyter cell prepares the quadrants of the original image IMG - here of a bunch of roses. The code is straight forward and easy to understand.

# ****************************
# Work with sub-Images  
# ************************

%matplotlib inline

import PIL
from PIL import Image, ImageOps

fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 16
fig_size[1] = 20
fig_3 = plt.figure(3)
ax3_1_1 = fig_3.add_subplot(541)
ax3_1_2 = fig_3.add_subplot(542)
ax3_1_3 = fig_3.add_subplot(543)
ax3_1_4 = fig_3.add_subplot(544)
ax3_2_1 = fig_3.add_subplot(545)
ax3_2_2 = fig_3.add_subplot(546)
ax3_2_3 = fig_3.add_subplot(547)
ax3_2_4 = fig_3.add_subplot(548)
ax3_3_1 = fig_3.add_subplot(5,4,9)
ax3_3_2 = fig_3.add_subplot(5,4,10)
ax3_3_3 = fig_3.add_subplot(5,4,11)
ax3_3_4 = fig_3.add_subplot(5,4,12)
ax3_4_1 = fig_3.add_subplot(5,4,13)
ax3_4_2 = fig_3.add_subplot(5,4,14)
ax3_4_3 = fig_3.add_subplot(5,4,15)
ax3_4_4 = fig_3.add_subplot(5,4,16)
ax3_5_1 = fig_3.add_subplot(5,4,17)
ax3_5_2 = fig_3.add_subplot(5,4,18)
ax3_5_3 = fig_3.add_subplot(5,4,19)
ax3_5_4 = fig_3.add_subplot(5,4,20)

# size to work with 
# ******************
img_wk_size = 560 

# bring the orig img down to (560, 560) 
# ***************************************
imgvc = Image.open("rosen_orig_farbe.jpg")
imgvc_wk_size = imgvc.resize((img_wk_size,img_wk_size), resample=PIL.Image.BICUBIC)
# Change to np arrays 
ay_picc = np.array(imgvc_wk_size)
print("ay_picc.shape = ", ay_picc.shape)
print("r = ", ay_picc[0,0,0],    " g = ", ay_picc[0,0,1],    " b = " , ay_picc[0,0,2] )
print("r = ", ay_picc[200,0,0],  " g = ", ay_picc[200,0,1],  " b = " , ay_picc[200,0,2] )

# Turn color to gray 
#Red * 0.3 + Green * 0.59 + Blue * 0.11
#Red * 0.2126 + Green * 0.7152 + Blue * 0.0722
#Red * 0.299 + Green * 0.587 + Blue * 0.114

ay_picc_g = ( 0.299  * ay_picc[:,:,0] + 0.587  * ay_picc[:,:,1] + 0.114  * ay_picc[:,:,2] )  
ay_picc_g = ay_picc_g.astype('float32') 
t_picc_g  = ay_picc_g.reshape((1, img_wk_size, img_wk_size, 1))
t_picc_g  = tf.image.per_image_standardization(t_picc_g)

# downsize to (28,28)  
t_picc_g_28 = tf.image.resize(t_picc_g, [28,28], method="bicubic", antialias=True)
t_picc_g_28 = tf.image.per_image_standardization(t_picc_g_28)

# get the correction for the full image 
t_picc_g_wk_size = tf.image.resize(t_picc_g_28, [img_wk_size,img_wk_size], method="bicubic", antialias=True)
t_picc_g_wk_size = tf.image.per_image_standardization(t_picc_g_wk_size)

t_picc_g_wk_size_corr = t_picc_g - t_picc_g_wk_size
t_picc_g_wk_size_re   = t_picc_g_wk_size + t_picc_g_wk_size_corr


# Display wk_size orig images  
ax3_1_1.imshow(imgvc_wk_size)
ax3_1_2.imshow(t_picc_g[0,:,:,0], cmap=plt.cm.gray)
ax3_1_3.imshow(t_picc_g_28[0,:,:,0], cmap=plt.cm.gray)
ax3_1_4.imshow(t_picc_g_wk_size_re[0,:,:,0], cmap=plt.cm.gray)


# Split in 4 sub-images 
# ***********************

half_wk_size  = int(img_wk_size / 2)  

t_picc_g_1  = t_picc_g[:, 0:half_wk_size, 0:half_wk_size, :]
t_picc_g_2  = t_picc_g[:, 0:half_wk_size, half_wk_size:img_wk_size, :]
t_picc_g_3  = t_picc_g[:, half_wk_size:img_wk_size, 0:half_wk_size, :]
t_picc_g_4  = t_picc_g[:, half_wk_size:img_wk_size, half_wk_size:img_wk_size, :]

# Display wk_size orig images  
ax3_2_1.imshow(t_picc_g_1[0,:,:,0], cmap=plt.cm.gray)
ax3_2_2.imshow(t_picc_g_2[0,:,:,0], cmap=plt.cm.gray)
ax3_2_3.imshow(t_picc_g_3[0,:,:,0], cmap=plt.cm.gray)
ax3_2_4.imshow(t_picc_g_4[0,:,:,0], cmap=plt.cm.gray)


# Downscale sub-images 
t_picc_g_1_28 = tf.image.resize(t_picc_g_1, [28,28], method="bicubic", antialias=True)
t_picc_g_2_28 = tf.image.resize(t_picc_g_2, [28,28], method="bicubic", antialias=True)
t_picc_g_3_28 = tf.image.resize(t_picc_g_3, [28,28], method="bicubic", antialias=True)
t_picc_g_4_28 = tf.image.resize(t_picc_g_4, [28,28], method="bicubic", antialias=True)


# Display downscales sub-images  
ax3_3_1.imshow(t_picc_g_1_28[0,:,:,0], cmap=plt.cm.gray)
ax3_3_2.imshow(t_picc_g_2_28[0,:,:,0], cmap=plt.cm.gray)
ax3_3_3.imshow(t_picc_g_3_28[0,:,:,0], cmap=plt.cm.gray)
ax3_3_4.imshow(t_picc_g_4_28[0,:,:,0], cmap=plt.cm.gray)


# get correction values for upsizing 
t_picc_g_1_wk_half = tf.image.resize(t_picc_g_1_28, [half_wk_size,half_wk_size], method="bicubic", antialias=True)
t_picc_g_1_wk_half = tf.image.per_image_standardization(t_picc_g_1_wk_half)
t_picc_g_2_wk_half = tf.image.resize(t_picc_g_2_28, [half_wk_size,half_wk_size], method="bicubic", antialias=True)
t_picc_g_2_wk_half = tf.image.per_image_standardization(t_picc_g_2_wk_half)
t_picc_g_3_wk_half = tf.image.resize(t_picc_g_3_28, [half_wk_size,half_wk_size], method="bicubic", antialias=True)
t_picc_g_3_wk_half = tf.image.per_image_standardization(t_picc_g_3_wk_half)
t_picc_g_4_wk_half = tf.image.resize(t_picc_g_4_28, [half_wk_size,half_wk_size], method="bicubic", antialias=True)
t_picc_g_4_wk_half = tf.image.per_image_standardization(t_picc_g_4_wk_half)
 
    
t_picc_g_1_corr =  t_picc_g_1 - t_picc_g_1_wk_half   
t_picc_g_2_corr =  t_picc_g_2 - t_picc_g_2_wk_half   
t_picc_g_3_corr =  t_picc_g_3 - t_picc_g_3_wk_half   
t_picc_g_4_corr =  t_picc_g_4 - t_picc_g_4_wk_half   

t_picc_g_1_re   = t_picc_g_1_wk_half + t_picc_g_1_corr 
t_picc_g_2_re   = t_picc_g_2_wk_half + t_picc_g_2_corr 
t_picc_g_3_re   = t_picc_g_3_wk_half + t_picc_g_3_corr 
t_picc_g_4_re   = t_picc_g_4_wk_half + t_picc_g_4_corr 

print(t_picc_g_1_re.shape)

# Display downscales sub-images  
ax3_4_1.imshow(t_picc_g_1_re[0,:,:,0], cmap=plt.cm.gray)
ax3_4_2.imshow(t_picc_g_2_re[0,:,:,0], cmap=plt.cm.gray)
ax3_4_3.imshow(t_picc_g_3_re[0,:,:,0], cmap=plt.cm.gray)
ax3_4_4.imshow(t_picc_g_4_re[0,:,:,0], cmap=plt.cm.gray)

ay_img_comp = np.zeros((img_wk_size,img_wk_size))
ay_t_comp   = ay_img_comp.reshape((1,img_wk_size, img_wk_size,1))
t_pic_comp  = ay_t_comp
t_pic_comp[0, 0:half_wk_size, 0:half_wk_size, 0]                       = t_picc_g_1_re[0, :, :, 0]
t_pic_comp[0, 0:half_wk_size, half_wk_size:img_wk_size, 0]             = t_picc_g_2_re[0, :, :, 0]
t_pic_comp[0, half_wk_size:img_wk_size, 0:half_wk_size, 0]             = t_picc_g_3_re[0, :, :, 0]
t_pic_comp[0, half_wk_size:img_wk_size, half_wk_size:img_wk_size, 0]   = t_picc_g_4_re[0, :, :, 0]

ax3_5_1.imshow(t_picc_g[0,:,:,0], cmap=plt.cm.gray)
ax3_5_2.imshow(t_pic_comp[0,:,:,0], cmap=plt.cm.gray)

 
Note the computation of the correction tensors; we prove their effectiveness below by displaying respective results for the full image, cut out and re-upscaled, corrected quadrants and replaced areas of the original image.

The results in the prepared image frames look like:

Good!

Some code for the carving algorithm - applied to the full image AND a single selected sub-image

Now, for a first test, we concentrate on the sub-image in the lower-right corner - and apply the above algorithm. A corresponding Jupyter cell code is given below:

# *************************************************************************
# OIP analysis on sub-image tiles (to be used after the previous cell)
# **************************************************************************
# Note: To be applied after previous cell !!!
# ******

#interactive plotting - will be used in a future version when we deal with "octaves" 
#%matplotlib notebook 
#plt.ion()

%matplotlib inline

# preparation of figure frames 
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 12
fig_size[1] = 12
fig_4 = plt.figure(4)
ax4_1_1 = fig_4.add_subplot(441)
ax4_1_2 = fig_4.add_subplot(442)
ax4_1_3 = fig_4.add_subplot(443)
ax4_1_4 = fig_4.add_subplot(444)

ax4_2_1 = fig_4.add_subplot(445)
ax4_2_2 = fig_4.add_subplot(446)
ax4_2_3 = fig_4.add_subplot(447)
ax4_2_4 = fig_4.add_subplot(448)

ax4_3_1 = fig_4.add_subplot(4,4,9)
ax4_3_2 = fig_4.add_subplot(4,4,10)
ax4_3_3 = fig_4.add_subplot(4,4,11)
ax4_3_4 = fig_4.add_subplot(4,4,12)

ax4_4_1 = fig_4.add_subplot(4,4,13)
ax4_4_2 = fig_4.add_subplot(4,4,14)
ax4_4_3 = fig_4.add_subplot(4,4,15)
ax4_4_4 = fig_4.add_subplot(4,4,16)

fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 12
fig_size[1] = 6
fig_5 = plt.figure(5)
axa5_1 = fig_5.add_subplot(241)
axa5_2 = fig_5.add_subplot(242)
axa5_3 = fig_5.add_subplot(243)
axa5_4 = fig_5.add_subplot(244)
axa5_5 = fig_5.add_subplot(245)
axa5_6 = fig_5.add_subplot(246)
axa5_7 = fig_5.add_subplot(247)
axa5_8 = fig_5.add_subplot(248)
li_axa5 = [axa5_1, axa5_2, axa5_3, axa5_4, axa5_5, axa5_6, axa5_7, axa5_8]

# Some general OIP run parameters 
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
n_epochs  = 30          # should be divisible by 5  
n_steps   = 6           # number of intermediate reports 
epsilon   = 0.01        # step size for gradient correction  
conv_criterion = 2.e-4  # criterion for a potential stop of optimization 

# image width parameters
# ~~~~~~~~~~~~~~~~~~~~~
img_wk_size   = 560    # must be identical to the last cell   
half_wk_size  = int(img_wk_size / 2) 

# min / max values of the input image 
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Note : to be used for contrast enhancement 
min1 = tf.reduce_min(t_picc_g)
max1 = tf.reduce_max(t_picc_g)
# parameters to deal with the spectral distribution of gray values
spect_dist = min(abs(min1), abs(max1))
spect_fact   = 0.85
height_fact  = 1.1

# Set the gray downscaled input image as a startng point for the iteration loop 
# ~~~~~~~~~~~~~----------~~~~--------------~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MyOIP._initial_inp_img_data = t_picc_g_28

# ************
# Main Loop 
# ***********

num_main_iterate = 4
# map_index_main = -1       # take a cost function for a full layer 
map_index_main = 56         # map-index to be used for the OIP analysis of the full image  
map_index_sub  = 56         # map-index to be used for the OIP analysis of the sub-images   


for j in range(num_main_iterate):
   
    # ******************************************************
    # deal with the full (downscaled) input image 
    # ******************************************************
    
    # Apply OIP-algorithm to the whole downscaled image  
    # ~~~~~~~~~~~~~~~~~~~-----------~~~~~~~~~~~~~~~~~~~~~~~
    map_index_main_oip = map_index_main   # map-index we are interested in 

    # Perform the OIP analysis 
    MyOIP._derive_OIP(map_index = map_index_oip, 
                      n_epochs = n_epochs, n_steps = n_steps, 
                      epsilon = epsilon , conv_criterion = conv_criterion, 
                      b_stop_with_convergence=False,
                      b_print=False,
                      li_axa = li_axa5,
                      ax1_1 = ax4_1_1, ax1_2 = ax4_1_2)

    # display the modified downscaled image  (without and with contrast)
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    t_oip_g_28  = MyOIP._inp_img_data
    ay_oip_g_28 = t_oip_g_28[0,:,:,0].numpy()
    ay_oip_g_28_cont = MyOIP._transform_tensor_to_img(T_img=t_oip_g_28[0,:,:,0], centre_move=0.33, fact=1.0)
    ax4_1_3.imshow(ay_oip_g_28_cont, cmap=plt.cm.gray)
    ax4_1_4.imshow(t_picc_g_28[0, :, :, 0], cmap=plt.cm.gray)
    
    # rescale to 560 and re-add details via the correction tensor  
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    t_oip_g_wk_size        = tf.image.resize(t_oip_g_28, [img_wk_size,img_wk_size], 
                                             method="bicubic", antialias=True)
    t_oip_g_wk_size_re     = t_oip_g_wk_size + t_picc_g_wk_size_corr 
    
    # standardize to get an intermediate image 
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    # t_oip_g_loop           = t_oip_g_wk_size_re.numpy()
    # t_oip_g_wk_size_re_std = tf.image.per_image_standardization(t_oip_g_wk_size_re)
    t_oip_g_wk_size_re_std = tf.image.per_image_standardization(t_oip_g_wk_size_re)
    t_oip_g_loop           = t_oip_g_wk_size_re_std.numpy()
    
    # contrast required as the reduction of irrelvant pixels has smoothed out the gray scale beneath 
    # ~~~~~~~~~~~~~~~~~
    # the added details at pattern areas = high level of whitened socket + relative small detail variation    
    t_oip_g_wk_size_re_std_plt = tf.clip_by_value(height_fact*t_oip_g_wk_size_re_std, -spect_dist*spect_fact, spect_dist*spect_fact)
    ax4_2_1.imshow(t_oip_g_wk_size[0,:,:,0], cmap=plt.cm.gray)
    ax4_2_2.imshow(t_oip_g_wk_size_re_std[0,:,:,0], cmap=plt.cm.gray)
    ax4_2_3.imshow(t_oip_g_wk_size_re_std_plt[0,:,:,0], cmap=plt.cm.gray)
    ax4_2_4.imshow(t_picc_g[0,:,:,0], cmap=plt.cm.gray)
   

    # *************************************************************
    # deal with a chosen sub-image - in this version: "t_oip_4_g"
    # **************************************************************
   
    # Cut out and downscale a sub-image region 
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    t_oip_4_g     = t_oip_g_loop[:, half_wk_size:img_wk_size, half_wk_size:img_wk_size, :]
    t_oip_4_g_28  = tf.image.resize(t_oip_4_g, [28,28], method="bicubic", antialias=True)
    
    # use the downscaled sub-image as an input image to the OIP-algorithm 
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    MyOIP._initial_inp_img_data = t_oip_4_g_28
    
    
    # Perform the OIP analysis on the sub-image
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    map_index_sub_oip  = map_index_sub   # we may vary this in later versions 
    
    MyOIP._derive_OIP(map_index = map_index_sub_oip, 
                      n_epochs = n_epochs, n_steps = n_steps, 
                      epsilon = epsilon , conv_criterion = conv_criterion, 
                      b_stop_with_convergence=False,
                      b_print=False,
                      li_axa = li_axa5,
                      ax1_1 = ax4_3_1, ax1_2 = ax4_3_2)
    
    
    # display the modified sub-image without and with contrast  
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    t_oip_4_g_28  = MyOIP._inp_img_data
    ay_oip_4_g_28 = t_oip_4_g_28[0,:,:,0].numpy()
    ay_oip_4_g_28_cont = MyOIP._transform_tensor_to_img(T_img=t_oip_4_g_28[0,:,:,0], centre_move=0.33, fact=1.0)
    ax4_3_3.imshow(ay_oip_4_g_28_cont, cmap=plt.cm.gray)
    ax4_3_4.imshow(t_oip_4_g[0, :, :, 0], cmap=plt.cm.gray)
    
    # upscaling of the present sub-image 
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    t_pic_4_g_wk_half = tf.image.resize(t_oip_4_g_28, [half_wk_size,half_wk_size], method="bicubic", antialias=True)
    
    # add the detail correction to the manipulated upscaled sub-image
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    t_pic_4_g_wk_half = t_pic_4_g_wk_half + t_picc_g_4_corr
    #t_pic_4_g_wk_half = tf.image.per_image_standardization(t_pic_4_g_wk_half)
    ax4_4_1.imshow(t_pic_4_g_wk_half[0, :, :, 0], cmap=plt.cm.gray)
    ax4_4_2.imshow(t_oip_4_g[0, :, :, 0], cmap=plt.cm.gray)
    
    # Overwrite the related region in the full image with the new sub-image  
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    t_oip_g_loop[0, half_wk_size:img_wk_size, half_wk_size:img_wk_size, 0]  = t_pic_4_g_wk_half[0, :, :, 0]
    t_oip_g_loop_std = tf.image.per_image_standardization(t_oip_g_loop)
    
    #ax4_4_3.imshow(t_oip_g_loop[0, :, :, 0], cmap=plt.cm.gray)   
    ax4_4_3.imshow(t_oip_g_loop_std[0, :, :, 0], cmap=plt.cm.gray)   
    ay_oip_g_loop_std_plt = tf.clip_by_value(height_fact*t_oip_g_loop_std, -spect_dist*spect_fact, spect_dist*spect_fact)
    ax4_4_4.imshow(ay_oip_g_loop_std_plt[0, :, :, 0], cmap=plt.cm.gray)
    
    # Downscale the resulting new full image and feed it into into the next iteration of the main loop
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    MyOIP._initial_inp_img_data = tf.image.resize(t_oip_g_loop_std, [28,28], method="bicubic", antialias=True)
   

 

The result of these operations after 4 iterations is displayed in the following image:

We recognize again the worm-like shape resulting for map 56, which we have seen in the last article, already. (By the way: Our CNN's map 56 originally is strongly triggered for the shapes of handwritten 9-digits and plays a significant role in classifying respective images correctly). But there is a new feature which appeared in the lower right part of the image - a kind of wheel or two closely neighbored 9-like shapes.

The following code compares the final result with the original image:

fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 12
fig_size[1] = 12
fig_7X = plt.figure(7)
ax1_7X_1 = fig_7X.add_subplot(221)
ax1_7X_2 = fig_7X.add_subplot(222)
ax1_7X_3 = fig_7X.add_subplot(223)
ax1_7X_4 = fig_7X.add_subplot(224)

ay_pic_full_cont = MyOIP._transform_tensor_to_img(T_img=t_oip_g_loop_std[0, :, :, 0], centre_move=0.46, fact=0.8)

ax1_7X_1.imshow(imgvc_wk_size)
ax1_7X_2.imshow(t_picc_g[0,:,:,0], cmap=plt.cm.gray)
ax1_7X_3.imshow(ay_oip_g_loop_std_plt[0, :, :, 0], cmap=plt.cm.gray)
ax1_7X_4.imshow(ay_pic_full_cont, cmap=plt.cm.gray)

Giving:

Those who do not like the relative strong contrast may do their own experiments with other parameters for contrast enhancement.

You see: Simplifying prejudices about the world, which get or got manifested in inflexible neural networks (or brains ?), may lead to strange, if not bad dreams. May also some politicians learn from this ... hmmm, just joking, as this requires both an openness and capacity for learning ... can't await Jan, 21st ...

Conclusion

Our "carving algorithm", which corresponds to an amplification of traces of map-related patterns that a CNN detects in an input image, can also be used for sub-image-areas and their pixel information. We can therefore use even very limited CNNs trained on low resolution MNIST images to create a "Deep MNIST Dream" out of a chosen arbitrary input image. At least in principle.

Addendum 06.01.2021: There is, however, a pitfall - our CNN (for MNIST or other image samples) may have been trained and work on standardized image/pixel data, only. But even if the overall input image may have been standardized before operating on it in the sense of the algorithm discussed in this article, we should take into account that a sub-image area may contain pixel data which are far from a standardized distribution. So, our corrections for details may require more than just the calculation of a difference term. We may have to reverse intermediate standardization steps, too. We shall take care of this in forthcoming code changes.

In the next article we are going to extend our algorithm to multiple cascaded sub-areas of our image - and we vary the maps used for different sub-areas at the same time.