Category Archives: CNNs

Articles on experiments with Convolutional Neural Networks

A simple CNN for the MNIST dataset – V – about the difference of activation patterns and features

Posted on by

In my last article of my introductory series on “Convolutional Neural Networks” [CNNs] I described how we can visualize the output of different maps at convolutional (or pooling) layers of a CNN.

A simple CNN for the MNIST dataset – IV – Visualizing the activation output of convolutional layers and maps
A simple CNN for the MNIST dataset – III – inclusion of a learning-rate scheduler, momentum and a L2-regularizer
A simple CNN for the MNIST datasets – II – building the CNN with Keras and a first test
A simple CNN for the MNIST datasets – I – CNN basics

We are now well equipped to look a bit closer at the maps of a trained CNN. The output of the last convolutional layer is of course of special interest: It is fed (in the form of a flattened input vector) into the MLP-part of the CNN for a classification analysis. As an MLP detects “patterns” the question arises whether we actually can see common “patterns” in the visualized maps of different images belonging to the same class. In our case we shall have a look at the maps of different MNIST images of a handwritten “4”.

Note for my readers, 20.08.2020:
This article has recently been revised and completely rewritten. It required a much more careful description of what we mean by “patterns” and “features” – and what we can say about them when looking at images of activation outputs on higher convolutional layers. I also postponed a thorough “philosophical” argumentation against a humanized usage of the term “features” to a later article in this series.

Objectives

We saw already in the last article that the images of maps get more and more abstract when we move to higher convolutional layers – i.e. layers deeper inside a CNN. At the same time we loose resolution due to intermediate pooling operations. It is quite obvious that we cannot see much of any original “features” of a handwritten “4” any longer in a (3×3)-map, whose values are produced by a sequence of complex transformation operations.

Nevertheless people talk about “feature detection” performed by CNNs – and they refer to “features” in a very concrete and descriptive way (e.g. “eyes”, “spectacles”, “bows”). How can this be? What is the connection of abstract activation patterns in low resolution maps to original “features” of an image? What is meant when CNN experts claim that neurons of higher CNN layers are allegedly able to “detect features”?

We cannot give a full answer, yet. We still need some more Python programs tools. But, wat we are going to do in this article are three things:

  1. Objective 1: I will try to describe the assumed relation between maps and “features”. To start with I shall make a clear distinction between “feature” patterns in input images and patterns in and across the maps of convolutional layers. The rest of the discussion will remain a bit theoretical; but it will use the fact that convolutions at higher layers combine filtered results in specific ways to create new maps. For the time being we cannot do more. We shall actually look at visualizations of “features” in forthcoming articles of this series. Promised.
  2. Objective 2: We follow three different input images, each representing a “4”, as they get processed from one convolutional layer to the next convolutional layer of our CNN. We shall compare the resulting outputs of all feature maps at each convolutional layer.
  3. Objective 3: We try to identify common “patterns” for our different “4” images across the maps of the highest convolutional layer.

We shall visualize each “map” by an image – reflecting the values calculated by the CNN-filters for all points in each map. Note that an individual value at a map point results from adding up many weighted values provided by the maps of lower layers and feeding the result into an activation function. We speak of “activation” values or “map activations”. So our 2-nd objective is to follow the map activations of an input image up to the highest convolutional layer. An interesting question will be if the chain of complex transformation operations leads to visually detectable similarities across the map outputs for the different images of a “4”.

The eventual classification of a CNN is done by its embedded MLP which analyzes information collected at the last convolutional layer. Regarding this input to the MLP we can make the following statements:

The convolutions and pooling operations project information of relatively large parts of the original image into a representation space of very low dimensionality. Each map on the third layer provides a 3×3 value tensor, only. However, we combine the points of all (128) maps together in a flattened input vector to the MLP. This input vector consists of more nodes than the original image itself.

Thus the sequence of convolutional and pooling layers in the end transforms the original images into another representation space of somewhat higher dimensionality (9×128 vs. 28×28). This transformation is associated with the hope that in the new representation space a MLP may find patterns which allow for a better classification of the original images than a direct analysis of the image data. This explains objective 3: We try to play the MLPs role by literally looking at the eventual map activations. We try to find out which patterns are representative for a “4” by comparing the activations of different “4” images of the MNIST dataset.

Enumbering the layers

To distinguish a higher Convolutional [Conv] or Pooling [Pool] layer from a lower one we give them a number “Conv_N” or “Pool_N”.

Our CNN has a sequence of

  • Conv_1 (32 26×26 maps filtering the input image),
  • Pool_1 (32 13×13 maps with half the resolution due to max-pooling),
  • Conv_2 (64 11×11 maps filtering combined maps of Pool_1),
  • Pool_2 (64 5×5 maps with half the resolution due to max-pooling),
  • Conv_3 (128 3×3 maps filtering combined maps of Pool_2).

Patterns in maps?

We have seen already in the last article that the “patterns” which are displayed in a map of a higher layer Conv_N, with N ≥ 2, are rather abstract ones. The images of the maps at Conv_3 do not reflect figurative elements or geometrical patterns of the input images any more – at least not in a directly visible way. It does not help that the activations are probably triggered by some characteristic pixel patterns in the original images.

The convolutions and the pooling operation transform the original image information into more and more abstract representation spaces of shrinking dimensionality and resolution. This is due to the fact that the activation of a point in a map on a layer Conv_(N+1) results

  • from a specific combination of multiple maps of a layer Conv_N or Pool_N
  • and from a loss of resolution due to intermediate pooling.

It is not possible to directly guess in what way active points or activated areas within
a certain map at the third convolutional layer relate to or how they depend on “original and specific patterns in the input image”. If you do not believe me: Well, just look at the maps of the 3rd convolutional layer presented in the last article and tell me: What patterns in the initial image did these maps react to? Without some sophisticated numerical experiments you won’t be able to figure that out.

Patterns in the input image vs. patterns within and across maps

The above remarks indicate already that “patterns” may occur at different levels of consideration and abstraction. We talk about patterns in the input image and patterns within as well as across the maps of convolutional (or pooling) layers. To avoid confusion I already now want to make the following distinction:

  • (Original) input patterns [OIP]: When I speak of (original) “input patterns” I mean patterns or figurative elements in the input image. In more mathematical terms I mean patterns within the input image which correspond to a kind of fixed and strong correlation between the values of pixels distributed over a sufficiently well defined geometrical area with a certain shape. Examples could be line-like elements, bow segments, two connected circles or combined rectangles. But OIPs may be of a much more complex and abstract kind and consist of strange sub-features – and they may not reflect a real world entity or a combination of such entities. An OIP may reside at one or multiple locations in different input images.
  • Filter correlation patterns [FCP]: A CNN produces maps by filtering input data (Conv level 1) or by filtering maps of a lower layer and combining the results. By doing so a higher layer may detect patterns in the filter results of a lower layer. I call a pattern across the maps of a convolutional or pooling layer Conv_N or Pool_N as seen by Conv_(N+1) a FCP.
    Note: Because a 3×3 filter for a map of Conv_(N+1) has fixed parameters per map of the previous layer Conv_N or Pool_N, it combines multiple maps (filters) of Conv_N in a specific, unique way.

Anybody who ever worked with image processing and filters knows that combining basic filters may lead to the display of weirdly looking, combined information residing in complex regions on the original image. E.g., a certain combination of filters may emphasize diagonal lines or bows with some distance in between and suppress all other features. Therefore, it is at least plausible that a map of a higher convolutional layer can be translated back to an OIP. Meaning:

A high activation of certain or multiple points inside a map on Conv_3 may reflect some typical OIP pattern in the input image.

But: At the moment we have no direct proof for such an idea. And it is not at all obvious what kind of OIP pattern this may be for a distinct map – and whether it can directly be described in terms of basic geometrical elements of a figurative number representation in the MNIST case. By just looking at the maps of a layer and their activated points we do not get any clue about this.

If, however, activated maps somehow really correspond to OIPs then a FCP over multiple maps may be associated with a combination of distinct OIPs in an input image.

What are “features” then?

In many textbooks maps are also called “feature maps“. As far I understand it the authors call a “feature” what I called an OIP above. By talking about a “feature” the authors most often refer to a pattern which a CNN somehow detects or identifies in the input images.

Typical examples of “features” text-book authors often discuss and even use in illustrations are very concrete: ears, eyes, feathers, wings, a mustache, leaves, wheels, sun-glasses … I.e., a lot of authors typically name features which human beings identify as physical entities or as entities, for which we have clear conceptual ideas in our mind. I think such examples trigger ideas about CNNs which are too far-fetched and which “humanize” stupid algorithmic processes.

The arguments in favor of the detection of features in the sense of conceptual entities are typically a bit nebulous – to say the least. E.g. in a relatively new book on “Generative Deep Learning” you see a series of CNN neuron layers associated with rather dubious and unclear images of triangles etc. and at the last convolutional layer we suddenly see pretty clear sketches of a mustache, a certain hairdress, eyes, lips, a shirt, an ear .. “. The related text goes like follows (I retranslated the text from the German version of the book): “Layer 1 consists of neurons which activate themselves stronger, when they recognize certain elementary and basic features in the input image, e.g. borders. The output of these neurons is then forwarded to the neurons of layer 2 which can use this information to detect more complex features – and so on across the following layers.” Yeah, “neurons activate themselves” as they “recognize” features – and suddenly the neurons at a high enough layer see a “spectacle”. 🙁

I think it would probably be more correct to say the following:

The activation of a map of a high convolutional layer may indicate the appearance of some kind of (complex) pattern or a sequence of patterns within an input image, for which a specific filter combination produces relatively high values in a low dimensional output space.

Note: At our level of analyzing CNNs even this carefully formulated idea is speculation. Which we will have to prove somehow … Where we stand right now, we are unfortunately not yet ready to identify OIPs or repeated OIP sequences associated with maps. This will be the topic of forthcoming articles.

It is indeed an interesting question whether a trained CNN “detects” patterns in the sense of entities with an underlying “concept”. I would say: Certainly not. At least not pure CNNs. I think, we should be very careful with the use of the term “feature”. Based on the filtering convolutions perform we might say:

A “feature” (hopefully) is a pattern in the sense of defined geometrical pixel correlation in an image.

Not more, not less. Such a “feature” may or may not correspond to entities, which a human being could identify and for which he or she has a concept for. A feature is just a pixel correlation whose appearance triggers output neurons in high level maps.

By the way there are 2 more points regarding the idea of feature detection:

  • A feature or OIP may be located at different places in different images of something like a “5”. Due to different sizes of the depicted “5” and translational effects. So keep in mind that if maps do indeed relate to features it has to be explained how convolutional filtering can account for any translational invariance of the “detection” of a pattern in an image.
  • The concrete examples given for “features” by many authors imply that the features are more or less the same for two differently trained CNNs. Well, regarding the point that training corresponds to finding a minimum on a rather complex multidimensional hyperplane this raises the question how well defined such a (global) minimum really is and whether it or other valid side minima are approached.

Keep these points in mind until we come back to related experiments in further articles.

From “features” to FCPs on the last Conv-layer?

However and independent of how a CNN really reacts to OIPs or “features”, we should not forget the following:
In the end a CNN – more precisely its embedded MLP – reacts to FCPs on the last convolutional level. In our CNN an FCP on the third convolutional layer with specific active points across 128 (3×3)-maps obviously can obviously tell the MLP something about the class an input image belongs to: We have proven already that the MLP part of our simple CNN guesses the class the original image belongs to with a surprisingly high accuracy. And by construction it obviously does so by just analyzing the 128 (3×3)-activation values of the third layer – arranged into a flattened vector.

From a classification point of view it, therefore, seems to be legitimate to look out for any FCP across the maps on Conv_3. As we can visualize the maps it is reasonable to literally look for common activation patterns which different images of handwritten “4”s may trigger on the maps of the last convolutional level. The basic idea behind this experimental step is:

OIPs which are typical for images of a “4” trigger and activate certain maps or points within certain maps. Across all maps we then may see a characteristic FCP for a “4”, which not only a MLP but also we intelligent humans could identify.

Or: Multiple characteristic features in images of a “4” may trigger characteristic FCPs which in turn can be used indicators of a class an image belongs to by an MLP. Well, let us see how far we get with this kind of theory.

Levels of “abstractions”

Let us take a MNIST image which represents something which a European would consider to be a clear representation of a “4”.

In the second image I used the “jet”-color map; i.e. dark blue indicates a low intensity value while colors from light blue to green to yellow and red indicate growing intensity values.

The first conv2D-layer (“Conv2d_1”) produces the following 32 maps of my chosen “4”-image after training:

We see that the filters, which were established during training emphasize general contours but also focus on certain image regions. However, the original “4” is still clearly visible on very many maps as the convolution does not yet reduce resolution too much.

By the way: When looking at the maps the first time I found it surprising that the application of a simple linear 3×3 filter with stride 1 could emphasize an overall oval region and suppress the pixels which formed the “4” inside of this region. A closer look revealed however that the oval region existed already in the original image data. It was emphasized by an inversion of the pixel values …

Pooling
The second Conv2D-layer already combines information of larger areas of the image – as a max (!) pooling layer was applied before. We loose resolution here. But there is a gain, too: the next convolution can filter (already filtered) information over larger areas of the original image.

But note: In other types of more advanced and modern CNNs pooling only is involved after two or more successive convolutions have happened. The direct succession of convolutions corresponds to a direct and unique combination of filters at the same level of resolution.

The 2nd convolution
As we use 64 convolutional maps on the 2nd layer level we allow for a multitude of different new convolutions. It is to be understood that each new map at the 2nd cConv layer is the result of a special unique combination of filtered information of all 32 previous maps (of Pool_1). Each of the previous 32 maps contributes through a specific unique filter and respective convolution operation to a single specific map at layer 2. Remember that we get 3×3 x 32 x 64 parameters for connecting the maps of Pool_1 to maps of Conv_2. It is this unique combination of already filtered results which enriches the analysis of the original image for more complex patterns than just the ones emphasized by the first convolutional filters.

As the max-condition of the pooling layer was applied first and because larger areas are now analyzed we are not too astonished to see that the filters dissolve the original “4”-shape and indicate more general geometrical patterns – which actually reflect specific correlations of map patterns on layer Conv_1.

I find it interesting that our “4” triggers more horizontally activations within some maps on this already abstract level than vertical ones. One should not confuse these patterns with horizontal patterns in the original image. The relation of original patterns with these activations is already much more complex.

The third convolutional layer applies filters which now cover almost the full original image and combine and mix at the same time information from the already rather abstract results of layer 2 – and of all the 64 maps there in parallel.

We again see a dominance of horizontal patterns. We see clearly that on this level any reference to something like an arrangement of parallel vertical lines crossed by a horizontal line is completely lost. Instead the CNN has transformed the original distribution of black (dark grey) pixels into multiple abstract configuration spaces with 2 axes, which only coarsely reflecting the original image area – namely by 3×3 maps; i.e. spaces with a very poor resolution.

What we see here are “correlations” of filtered and transformed original pixel clusters over relatively large areas. But no constructive concept of certain line arrangements.

Now, if this were the level of “FCP-patterns” which the MLP-part of the CNN uses to determine that we have a “4” then we would bet that such abstract patterns (active points on 9×9 grids) appear in a similar way on the maps of the 3rd Conv layer for other MNIST images of a “4”, too.

Well, how similar do different map representations of “4”s look like on the 3rd Conv2D-layer?

What makes a four a four in the eyes of the CNN?

The last question corresponds to the question of what activation outputs of “4”s really have in common. Let us take 3 different images of a “4”:

The same with the “jet”-color-map:

 

Already with our eyes we see that there are similarities but also quite a lot of differences.

Different “4”-representations on the 2nd Conv-layer

Below we see comparison of the 64 maps on the 2nd Conv-layer for our three “4”-images.

If you move your head backwards and ignore details you see that certain maps are not filled in all three map-pictures. Unfortunately, this is no common feature of “4”-representations. Below you see images of the activation of a “1” and a “2”. There the same maps are not activated at all.

We also see that on this level it is still important which points within a map are activated – and not which map on average. The original shape of the underlying number is reflected in the maps’ activations.

Now, regarding the “4”-representations you may say: Well, I still recognize some common line patterns – e.g. parallel lines in a certain 75 degree angle on the 11×11 grids. Yes, but these lines are almost dissolved by the next pooling step:

Consider in addition that the next (3rd) convolution combines 3×3-data of all of the displayed 5×5-maps. Then, probably, we can hardly speak of a concept of abstract line configurations any more …

“4”-representations on the third Conv-layer

Below you find the activation outputs on the 3rd Conv2D-layer for our three different “4”-images:

When we look at details we see that prominent “features” in one map of a specific 4-image do NOT appear in a fully comparable way in the eventual convolutional maps for another image of a “4”. Some of the maps (i.e. filters after 4 transformations) produce really different results for our three images.

But there are common elements, too: I have marked only some of the points which show a significant intensity in all of the maps. But does this mean these individual common points are decisive for a classification of a “4”? We cannot be sure about it – probably it is their combination which is relevant.

So, what we ended up with is that we find some common points or some common point-relations in a few of the 128 “3×3”-maps of our three images of handwritten “4”s.

But how does this compare with maps of images of other digits? Well, look at he maps on the 3rd layer for images of a “1” and a “2” respectively:

On the 3rd layer it becomes more important which maps are not activated at all. But still the activation patterns within certain maps seem to be of importance for an eventual classification.

Conclusion

The maps of a CNN are created by an effective and guided optimization process. The results indicate the eventual detection of rather abstract patterns within and across filter maps on higher convolutional layers.

But these patterns (FCP-patterns) should not be confused with figurative elements or “features” in the original input images. Activation patterns at best vaguely remind of the original image features. At our level of analysis of a CNN we can only speculate about some correspondence of map activations with original features or patterns in an input image.

But it seems pretty clear that patterns in or across maps do not indicate any kind of constructive concept which describes how to build a “4” from underlying more elementary features in the sense of combine-able independent entities. There is no sign of conceptual constructive idea of how to denote a “4”. At least not in pure CNNs … Things may be a bit different in convolutional “autoencoders” (combinations of convolutional encoders and decoders), but this is another story we will come back to in this blog. Right now we would say that abstract (FCP-) patterns in maps of higher convolutional layers result from intricate filter combinations. These filters may react to certain patterns in an input image – but whether these patterns correspond to entities a human being would use to write down and thereby construct a “4” or an “8” is questionable.

We saw that the abstract information maps at the third layer of our CNN do show some common elements between the images belonging to the same class – and delicate differences with respect to activations resulting from images of other classes. However, the differences reside in details and the situation remains complicated. In the end the MLP-part of a CNN still has a lot of work to do. It must perform its classification task based on the correlation or anti-correlation of “point”-like elements in a multitude of maps – and probably even based on the activation level (i.e. output numbers) at these points.

This is seemingly very different from a conscious consideration process and weighing of alternatives which a human brain performs when it looks at sketches of numbers. When in doubt our brain tries to find traces consistent with a construction process defined for writing down a “4”, i.e. signs of a certain arrangement of straight and curved lines. A human brain, thus, would refer to arrangements of line elements, bows or circles – but not to relations of individual points in an extremely coarse and abstract representation space after some mathematical transformations. You may now argue that we do not need such a process when looking at clear representations of a “4” – we look and just know that its a “4”. I do not doubt that a brain may use maps, too – but I want to point out that a conscious intelligent thought process and conceptual ideas about entities involve constructive operations and not just a passive application of filters. Even from this extremely simplifying point of view CNNs are stupid though efficient algorithms. And authors writing about “features” should avoid any kind of a humanized interpretation.

In the next article

A simple CNN for the MNIST dataset – VI – classification by activation patterns and the role of the CNN’s MLP part

we shall look at the whole procedure again, but then we compare common elements of a “4” with those of a “9” on the 3rd convolutional layer. Then the key question will be: ” What do “4”s have in common on the last convolutional maps which corresponding activations of “9”s do not show – and vice versa.

This will become especially interesting in cases for which a distinction was difficult for pure MLPs. You remember the confusion matrix for the MNIST dataset? See:
A simple Python program for an ANN to cover the MNIST dataset – XI – confusion matrix
We saw at that point in time that pure MLPs had some difficulties to distinct badly written “4”s from “9s”. We will see that the better distinction abilities of CNNs in the end depend on very few point like elements of the eventual activation on the last layer before the MLP.

Further articles in this series

A simple CNN for the MNIST dataset – VII – outline of steps to visualize image patterns which trigger filter maps
A simple CNN for the MNIST dataset – VI – classification by activation patterns and the role of the CNN’s MLP part

 

A simple CNN for the MNIST dataset – IV – Visualizing the activation output of convolutional layers and maps

Posted on by

In the first three articles of this series on a (very) simple CNN for the MNIST dataset

A simple CNN for the MNIST dataset – III – inclusion of a learning-rate scheduler, momentum and a L2-regularizer
A simple CNN for the MNIST datasets – II – building the CNN with Keras and a first test
A simple CNN for the MNIST datasets – I – CNN basics

we invested some work into building layers and into the parameterization of a training run. Our rewards comprised a high accuracy value of around 99.35% and watching interactive plots during training.

But a CNN offers much more information which is worth and instructive to look at. In the first article I have talked a bit about feature detection happening via the “convolution” of filters with the original image data or the data produced at feature maps of previous layers. What if we could see what different filters do to the underlying data? Can we have a look at the output selected “feature maps” produce for a specific input image?

Yes, we can. And it is intriguing! The objective of this article is to plot images of the feature map output at a chosen convolutional or pooling layer of our CNN. This is accompanied by the hope to better understand the concept of abstract features extracted from an input image.

I follow an original idea published by F. Chollet (in his book “Deep Learning mit Python und Keras”, mitp Verlag) and adapt it to the code discussed in the previous articles.

Referring to inputs and outputs of models and layers

So far we have dealt with a complete CNN with a multitude of layers that produce intermediate tensors and a “one-hot”-encoded output to indicate the prediction for a hand-written digit represented by a MNIST image. The CNN itself was handled by Keras in form of a sequential model of defined convolutional and pooling layers plus layers of a multi-layer perceptron [MLP]. By the definition of such a “model” Keras does all the work required for forward and backward propagation steps in the background. After training we can “predict” the outcome for any new digit image which we feed into the CNN: We just have to fetch the data form th eoutput layer (at the end of the MLP) after a forward propagation with the weights optimized during training.

But now, we need something else:

We need a model which gives us the output, i.e. a 2-dimensional tensor – of a specific map of an intermediate Conv-layer as a prediction for an input image!

I.e. we want the output of a sub-model of our CNN containing only a part of the layers. How can we define such an (additional) model based on the layers of our complete original CNN-model?

Well, with Keras we can build a general model based on any (partial) graph of connected layers which somebody has set up. The input of such a model must follow rules appropriate to the receiving layer and the output can be that of a defined subsequent layer or map. Setting up layers and models can on a very basic level be done with the so called “Functional API of Keras“. This API enables us to directly refer to methods of the classes “Layer”, “Model”, “Input” and “Output”.

A model – as an instance of the Model-class – can be called like a function for its input (in tensor form) and it returns its output (in tensor form). As we deal with classes you will not be surprised over the fact that we can refer to the input-layer of a general model via the model’s instance name – let us say “cnnx” – and an instance attribute. A model has a unique input layer which later is fed by tensor input data. We can refer to this input layer via the attribute “input” of the model object. So, e.g. “cnnx.input” gives us a clear unique reference to the input layer. With the attribute “output” of a model we get a reference to the output layer.

But, how can we refer to the output of a specific layer or map of a CNN-model? If you look it up in the Keras documentation you will find that we can give each layer of a model a specific “name“. And a Keras model, of course, has a method to retrieve a reference to a layer via its name:

cnnx.get_layer(layer_name) .

Each convolutional layer of our CNN is an instance of the class “Conv2D-Layer” with an attribute “output” – this comprises the multidimensional tensor delivered by the activation function of the layer’s nodes (or units in Keras slang). Such a tensor has in general 4 axes for images:

sample-number of the batch, px width, px height, filter number

The “filter number” identifies a map of the Conv2D-layer. To get the “image”-data provided of a specific map (identified by “map-number”) we have to address the array as

cnnx.get_layer(layer_name)[sample-number, :, :, map-number]

We know already that these data are values in a certain range (here above 0, due to our choice of the activation function as “relu”).

Hint regarding wording: F. Chollet calls the output of the activation functions of the nodes of a layer or map the “activation” of the layer or map, repsectively. We shall use this wording in the code we are going to build.

Displaying a specific image

It may be necessary later on to depict a chosen input image for our analysis – e.g. a MNIST image of the test data set. How can we do this? We just fill a new Jupyter cell with the following code: 

ay_img = test_imgs[7:8]
plt.imshow(ay_img[0,:,:,0], cmap=plt.cm.binary)

This code lines would plot the eighths sample image of the already shuffled test data set.

Using layer names and saving as well as restoring a model

We first must extend our previously defined functions to be able to deal with layer names. We change the code in our Jupyter Cell 8 (see the last article) in the following way:

Jupyter Cell 8: Setting up a training run

  
# Perform a training run 
# ********************

# Prepare the plotting 
# The really important command for interactive (=interediate) plot updating
%matplotlib notebook
plt.ion()

#sizing
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 8
fig_size[1] = 3

# One figure 
# -----------
fig1 = plt.figure(1)
#fig2 = plt.figure(2)

# first figure with two plot-areas with axes 
# --------------------------------------------
ax1_1 = fig1.add_subplot(121)
ax1_2 = fig1.add_subplot(122)
fig1.canvas.draw()

# second figure with just one plot area with axes
# -------------------------------------------------
#ax2 = fig2.add_subplot(121)
#ax2_1 = fig2.add_subplot(121)
#ax2_2 = fig2.add_subplot(122)
#fig2.canvas.draw()

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Parameterization of the training run 

#build = False
build = True
if cnn == None:
    build = True
    x_optimizer = None 
batch_size=64
epochs=80
reset = False 
#reset = True # we want training to start again with the initial weights

nmy_loss    ='categorical_crossentropy'
my_metrics =['accuracy']

my_regularizer = None
my_regularizer = 'l2'
my_reg_param_l2 = 0.001
#my_reg_param_l2 = 0.01
my_reg_param_l1 = 0.01


my_optimizer      = 'rmsprop'       # Present alternatives:  rmsprop, nadam, adamax 
my_momentum       = 0.5           # momentum value 
my_lr_sched       = 'powerSched'    # Present alternatrives: None, powerSched, exponential 
#my_lr_sched       = None           # Present alternatrives: None, powerSched, exponential 
my_lr_init        = 0.001           # initial leaning rate  
my_lr_decay_steps = 1               # decay steps = 1 
my_lr_decay_rate  = 0.001           # decay rate 


li_conv_1    = [32, (3,3), 0] 
li_conv_2    = [64, (3,3), 0] 
li_conv_3    = [128, (3,3), 0] 
li_Conv      = [li_conv_1, li_conv_2, li_conv_3]
li_Conv_Name = ["Conv2D_1", "Conv2D_2", "Conv2D_3"]
li_pool_1    = [(2,2)]
li_pool_2    = [(2,2)]
li_Pool      = [li_pool_1, li_pool_2]
li_Pool_Name = ["Max_Pool_1", "Max_Pool_2", "Max_Pool_3"]
li_dense_1   = [100, 0]
#li_dense_2  = [30, 0]
li_dense_3   = [10, 0]
li_MLP       = [li_dense_1, li_dense_2, li_dense_3]
li_MLP       = [li_dense_1, li_dense_3]
input_shape  = (28,28,1)

try: 
    if gpu:
        with tf.device("/GPU:0"):
            cnn, fit_time, history, x_optimizer  = train( cnn, build, train_imgs, train_labels, 
                                            li_Conv, li_Conv_Name, li_Pool, li_Pool_Name, li_MLP, input_shape, 
                                            reset, epochs, batch_size, 
                                            my_loss=my_loss, my_metrics=my_metrics, 
                                            my_regularizer=my_regularizer, 
                                            my_reg_param_l2=my_reg_param_l2, my_reg_param_l1=my_reg_param_l1,  
                                            my_optimizer=my_optimizer, my_momentum = 0.8,  
                                            my_lr_sched=my_lr_sched, 
                                            my_lr_init=my_lr_init, my_lr_decay_steps=my_lr_decay_steps, 
                                            my_lr_decay_rate=my_lr_decay_rate,  
                                            fig1=fig1, ax1_1=ax1_1, ax1_2=ax1_2
                                            )
        print('Time_GPU: ', fit_time)  
    else:
        with tf.device("/CPU:0"):
            cnn, fit_time, history = train( cnn, build, train_imgs, train_labels, 
                                            li_Conv, li_Conv_Name, li_Pool, li_Pool_Name, li_MLP, input_shape, 
                                            reset, epochs, batch_size, 
                                            my_loss=my_loss, my_metrics=my_metrics, 
                                            my_regularizer=my_regularizer, 
                                            my_reg_param_l2=my_reg_param_l2, my_reg_param_l1=my_reg_param_l1,  
                                            my_optimizer=my_optimizer, my_momentum = 0.8, 
                                            my_lr_sched=my_lr_sched, 
                                            my_lr_init=my_lr_init, my_lr_decay_steps=my_lr_decay_steps, 
                                            my_lr_decay_rate=my_lr_decay_rate,  
                                            fig1=fig1, ax1_1=ax1_1, ax1_2=ax1_2
                                            )
        print('Time_CPU: ', fit_time)  
except SystemExit:
    print("stopped due to exception")

 
You see that I added a list

li_Conv_Name = [“Conv2D_1”, “Conv2D_2”, “Conv2D_3”]
…
li_Pool_Name = [“Max_Pool_1”, “Max_Pool_2”, “Max_Pool_3”]

which provides names of the (presently three) defined convolutional and (presently two) pooling layers. The interface to the training function has, of course, to be extended to accept these arrays. The function “train()” in Jupyter cell 7 (see the last article) is modified accordingly:

Jupyter cell 7: Trigger (re-) building and training of the CNN 

# Training 2 - with test data integrated 
# *****************************************
def train( cnn, build, train_imgs, train_labels, 
           li_Conv, li_Conv_Name, li_Pool, li_Pool_Name, li_MLP, input_shape, 
           reset=True, epochs=5, batch_size=64, 
           my_loss='categorical_crossentropy', my_metrics=['accuracy'], 
           my_regularizer=None, 
           my_reg_param_l2=0.01, my_reg_param_l1=0.01, 
           my_optimizer='rmsprop', my_momentum=0.0, 
           my_lr_sched=None, 
           my_lr_init=0.001, my_lr_decay_steps=1, my_lr_decay_rate=0.00001,
           fig1=None, ax1_1=None, ax1_2=None
):
    
    if build:
        # build cnn layers - now with regularizer - 200603 rm
        cnn = build_cnn_simple( li_Conv, li_Conv_Name, li_Pool, li_Pool_Name, li_MLP, input_shape, 
                                my_regularizer = my_regularizer, 
                                my_reg_param_l2 = my_reg_param_l2, my_reg_param_l1 = my_reg_param_l1)
        
        # compile - now with lr_scheduler - 200603
        cnn = my_compile(cnn=cnn, 
                         my_loss=my_loss, my_metrics=my_metrics, 
                         my_optimizer=my_optimizer, my_momentum=my_momentum, 
                         my_lr_sched=my_lr_sched,
                         my_lr_init=my_lr_init, my_lr_decay_steps=my_lr_decay_steps, 
                         my_lr_decay_rate=my_lr_decay_rate)        
        
        # save the inital (!) weights to be able to restore them  
        cnn.save_weights('cnn_weights.h5') # save the initial weights 
         
        
    # reset weights(standard)
    if reset:
        cnn.load_weights('cnn_weights.h5')
 
    # Callback list 
    # ~~~~~~~~~~~~~
    use_scheduler = True
    if my_lr_sched == None:
        use_scheduler = False
    lr_history = LrHistory(use_scheduler)
    callbacks_list = [lr_history]
    if fig1 != None:
        epoch_plot = EpochPlot(epochs, fig1, ax1_1, ax1_2)
        callbacks_list.append(epoch_plot)
    
    start_t = time.perf_counter()
    if reset:
        history = cnn.fit(train_imgs, train_labels, initial_epoch=0, epochs=epochs, batch_size=batch_size, verbose=1, shuffle=True, 
                  validation_data=(test_imgs, test_labels), callbacks=callbacks_list) 
    else:
        history = cnn.fit(train_imgs, train_labels, epochs=epochs, batch_size=batch_size, verbose=1, shuffle=True, 
                validation_data=(test_imgs, test_labels), callbacks=callbacks_list ) 
    end_t = time.perf_counter()
    fit_t = end_t - start_t
    
    # save the model 
    cnn.save('cnn.h5')
    
    return cnn, fit_t, history, x_optimizer  # we return cnn to be able to use it by other Jupyter functions

 
We transfer the name-lists further on to the function “build_cnn_simple()“:

Jupyter Cell 4: Build a simple CNN

# Sequential layer model of our CNN
# ***********************************

# important !!
# ~~~~~~~~~~~
cnn = None
x_optimizers = None 

# function to build the CNN 
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
def build_cnn_simple(li_Conv, li_Conv_Name, li_Pool, li_Pool_Name, li_MLP, input_shape, 
                     my_regularizer=None, 
                     my_reg_param_l2=0.01, my_reg_param_l1=0.01 ):

    use_regularizer = True
    if my_regularizer == None:
        use_regularizer = 
False  
    
    # activation functions to be used in Conv-layers 
    li_conv_act_funcs = ['relu', 'sigmoid', 'elu', 'tanh']
    # activation functions to be used in MLP hidden layers  
    li_mlp_h_act_funcs = ['relu', 'sigmoid', 'tanh']
    # activation functions to be used in MLP output layers  
    li_mlp_o_act_funcs = ['softmax', 'sigmoid']

    # dictionary for regularizer functions
    d_reg = {
        'l2': regularizers.l2,  
        'l1': regularizers.l1
    }
    if use_regularizer: 
        if my_regularizer not in d_reg:
            print("regularizer " + my_regularizer + " not known!")
            sys.exit()
        else: 
            regul = d_reg[my_regularizer] 
        if my_regularizer == 'l2':
            reg_param = my_reg_param_l2
        elif my_regularizer == 'l1':
            reg_param = my_reg_param_l1
    
    
    # Build the Conv part of the CNN
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    num_conv_layers = len(li_Conv)
    num_pool_layers = len(li_Pool)
    if num_pool_layers != num_conv_layers - 1: 
        print("\nNumber of pool layers does not fit to number of Conv-layers")
        sys.exit()
    rg_il = range(num_conv_layers)

    # Define a sequential CNN model
    # ~~~~~~~~~~~~~~~~~~~~~~~~~-----
    cnn = models.Sequential()

    # in our simple model each con2D layer is followed by a Pooling layer (with the exeception of the last one) 
    for il in rg_il:
        # add the convolutional layer 
        num_filters  = li_Conv[il][0]
        t_fkern_size = li_Conv[il][1]
        cact         = li_conv_act_funcs[li_Conv[il][2]]
        cname        = li_Conv_Name[il]
        if il==0:
            cnn.add(layers.Conv2D(num_filters, t_fkern_size, activation=cact, name=cname,  
                                  input_shape=input_shape))
        else:
            cnn.add(layers.Conv2D(num_filters, t_fkern_size, activation=cact, name=cname))
        
        # add the pooling layer 
        if il < num_pool_layers:
            t_pkern_size = li_Pool[il][0]
            pname        = li_Pool_Name[il] 
            cnn.add(layers.MaxPooling2D(t_pkern_size, name=pname))
            

    # Build the MLP part of the CNN
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    num_mlp_layers = len(li_MLP)
    rg_im = range(num_mlp_layers)

    cnn.add(layers.Flatten())

    for im in rg_im:
        # add the dense layer 
        n_nodes = li_MLP[im][0]
        if im < num_mlp_layers - 1:  
            m_act   =  li_mlp_h_act_funcs[li_MLP[im][1]]
            if use_regularizer:
                cnn.add(layers.Dense(n_nodes, activation=m_act, kernel_regularizer=regul(reg_param)))
            else:
                cnn.add(layers.Dense(n_nodes, activation=m_act))
        else: 
            m_act   =  li_mlp_o_act_funcs[li_MLP[im][1]]
            if use_regularizer:
                cnn.add(layers.Dense(n_nodes, activation=m_act, kernel_regularizer=regul(reg_param)))
            else:
                cnn.add(layers.Dense(n_nodes, activation=m_act))
                
    return cnn

 
The layer names are transferred to Keras via the parameter “name” of the Model’s method “model.add()” to add a layer, e.g.:

cnn.add(layers.Conv2D(num_filters, t_fkern_size, activation=cact, name=cname))

Note that all other Jupyter cells remain unchanged.

Saving and restoring a model

Predictions of a neural network require a forward propagation of an input and thus a precise definition of layers and weights. In the last article we have already seen how we save and reload weight data of a model. However, weights make only a part of the information defining a model in a certain state. For seeing the activation of certain maps of a trained model we would like to be able to reload the full model in its trained status. Keras offers a very simple method to save and reload the complete set of data for a given model-state:

cnn.save(filename.h5′)
cnnx = models.load_model(‘filename.h5’)

This statement creates a file with the name name “filename.h5” in the h5-format (for large hierarchically organized data) in our Jupyter environment. You would of course replace “filename” by a more appropriate name to characterize your saved model-state. In my combined Eclipse-Jupyter-environment the standard path for such files points to the directory where I keep my notebooks. We included a corresponding statement at the end of the function “train()”. The attentive reader has certainly noticed this fact already.

A function to build a model for the retrieval and display of the activations of maps

We now build a new function to do the plotting of the outputs of all maps of a layer.

Jupyter Cell 9 – filling a grid with output-images of all maps of a layer 

# Function to plot the activations of a layer 
# -------------------------------------------
# Adaption of a method originally designed by F.Chollet 

def img_grid_of_layer_activation(d_img_sets, model_fname='cnn.h5', layer_name='', img_set="test_imgs", num_img=8, 
                                 scale_img_vals=False):
    '''
    Input parameter: 
    -----------------
    d_img_sets: dictionary with available img_sets, which contain img tensors (presently: train_imgs, test_imgs)  
    model_fname: Name of the file containing the models data 
    layer_name: name of the layer for which we plot the activation; the name must be known to the Keras model (string) 
    image_set: The set of images we pick a specific image from (string)
    num_img: The sample number of the image in the chosen set (integer) 
    scale_img_vals: False: Do NOT scale (standardize) and clip (!) the pixel values. True: Standardize the values. (Boolean)
        
    Hints: 
    -----------------
    We assume quadratic images 
    '''
    
    # Load a model 
    cnnx = models.load_model(model_fname)
    
    # get the output of a certain named layer - this includes all maps
    # https://keras.io/getting_started/faq/#how-can-i-obtain-the-output-of-an-intermediate-layer-feature-extraction
    cnnx_layer_output = cnnx.get_layer(layer_name).output

    # build a new model for input "cnnx.input" and output "output_of_layer"
    # ~~~~~~~~~~~~~~~~~
    # Keras knows the required connections and intermediat layers from its tensorflow graphs - otherwise we get an error 
    # The new model can make predictions for a suitable input in the required tensor form   
    mod_lay = models.Model(inputs=cnnx.input, outputs=cnnx_layer_output)
    
    # Pick the input image from a set of respective tensors 
    if img_set not in d_img_sets:
        print("img set " + img_set + " is not known!")
        sys.exit()
    # slicing to get te right tensor 
    ay_img = d_img_sets[img_set][num_img:(num_img+1)]
    
    # Use the tensor data as input for a prediction of model "mod_lay" 
    lay_activation = mod_lay.predict(ay_img) 
    print("shape of layer " + layer_name + " : ", lay_activation.shape )
    
    # number of maps of the selected layer 
    n_maps   = lay_activation.shape[-1]

    # size of an image - we assume quadratic images 
    img_size = lay_activation.shape[1]

    # Only for testing: plot an image for a selected  
    # map_nr = 1 
    #plt.matshow(lay_activation[0,:,:,map_nr], cmap='viridis')

    # We work with a grid of images for all maps  
    # ~~~~~~~~~~~~~~~----------------------------
    # the grid is build top-down (!) 
with num_cols and num_rows
    # dimensions for the grid 
    num_imgs_per_row = 8 
    num_cols = num_imgs_per_row
    num_rows = n_maps // num_imgs_per_row
    #print("img_size = ", img_size, " num_cols = ", num_cols, " num_rows = ", num_rows)

    # grid 
    dim_hor = num_imgs_per_row * img_size
    dim_ver = num_rows * img_size
    img_grid = np.zeros( (dim_ver, dim_hor) )   # horizontal, vertical matrix  
    print(img_grid.shape)

    # double loop to fill the grid 
    n = 0
    for row in range(num_rows):
        for col in range(num_cols):
            n += 1
            #print("n = ", n, "row = ", row, " col = ", col)
            present_img = lay_activation[0, :, :, row*num_imgs_per_row + col]

            # standardization and clipping of the img data  
            if scale_img_vals:
                present_img -= present_img.mean()
                if present_img.std() != 0.0: # standard deviation
                    present_img /= present_img.std()
                    #present_img /= (present_img.std() +1.e-8)
                    present_img *= 64
                    present_img += 128
                present_img = np.clip(present_img, 0, 255).astype('uint8') # limit values to 255

            # place the img-data at the right space and position in the grid 
            # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            # the following is only used if we had reversed vertical direction by accident  
            #img_grid[row*img_size:(row+1)*(img_size), col*img_size:(col+1)*(img_size)] = np.flip(present_img, 0)
            img_grid[row*img_size:(row+1)*(img_size), col*img_size:(col+1)*(img_size)] = present_img
 
    return img_grid, img_size, dim_hor, dim_ver

 
I explain the core parts of this code in the next two sections.

Explanation 1: A model for the prediction of the activation output of a (convolutional layer) layer

In a first step of the function “img_grid_of_layer_activation()” we load a CNN model saved at the end of a previous training run:

cnnx = models.load_model(model_fname)

The file-name “Model_fname” is a parameter. With the lines

cnnx_layer_output = cnnx.get_layer(layer_name).output
mod_lay = models.Model(inputs=cnnx.input, outputs=cnnx_layer_output)

we define a new model “cnnx” comprising all layers (of the loaded model) in between cnnx.input and cnnx_layer_output. “cnnx_layer_output” serves as an output layer of this new model “cnnx”. This model – as every working CNN model – can make predictions for a given input tensor. The output of this prediction is a tensor produced by cnnx_layer_output; a typical shape of the tensor is: 

shape of layer Conv2D_1 :  (1, 26, 26, 32)

From this tensor we can retrieve the size of the comprised quadratic image data.

Explanation 2: A grid to collect “image data” of the activations of all maps of a (convolutional) layer

Matplotlib can plot a grid of equally sized images. We use such a grid to collect the activation data produced by all maps of a chosen layer, which was given by its name as an input parameter.

The first statements define the number of images in a row of the grid – i.e. the number of columns of the grid. With the number of layer maps this in turn defines the required number of rows in the grid. From the number of pixel data in the tensor we can now define the grid dimensions in terms of pixels. The double loop eventually fills in the image data extracted from the tensors produced by the layer maps.

If requested by a function parameter “scale_img_vals=True” we standardize the image data and limit the pixel values to a maximum of 255 (clipping). This can in some cases be useful to get a better graphical representation of the
activation data with some color maps.

Our function “mg_grid_of_layer_activation()” returns the grid and dimensional data.

Note that the grid is oriented from its top downwards and from the left to the right side.

Plotting the output of a layer

In a further Jupyter cell we prepare and perform a call of our new function. Afterwards we plot resulting information in two figures.

Jupyter Cell 10 – plotting the activations of a layer 

# Plot the img grid of a layers activation 
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# global dict for the image sets 
d_img_sets= {'train_imgs':train_imgs, 'test_imgs':test_imgs}

# layer - pick one of the names which you defined for your model 
layer_name = "Conv2D_1"

# choose a image_set and an img number 
img_set = "test_imgs"
num_img = 19


# Two figures 
# -----------
fig1 = plt.figure(1)  # figure for th einput img
fig2 = plt.figure(2)  # figure for the activation outputs of th emaps 

ay_img = test_imgs[num_img:num_img+1]
plt.imshow(ay_img[0,:,:,0], cmap=plt.cm.binary)

# getting the img grid 
img_grid, img_size, dim_hor, dim_ver = img_grid_of_layer_activation(
                                        d_img_sets, model_fname='cnn.h5', layer_name=layer_name, 
                                        img_set=img_set, num_img=num_img, 
                                        scale_img_vals=False)
# Define reasonable figure dimensions by scaling the grid-size  
scale = 1.6 / (img_size)
fig2 = plt.figure( figsize=(scale * dim_hor, scale * dim_ver) )
#axes 
ax = fig2.gca()
ax.set_xlim(-0,dim_hor-1.0)
ax.set_ylim(dim_ver-1.0, 0)  # the grid is oriented top-down 
#ax.set_ylim(-0,dim_ver-1.0) # normally wrong

# setting labels - tick positions and grid lines  
ax.set_xticks(np.arange(img_size-0.5, dim_hor, img_size))
ax.set_yticks(np.arange(img_size-0.5, dim_ver, img_size))
ax.set_xticklabels([]) # no labels should be printed 
ax.set_yticklabels([])

# preparing the grid 
plt.grid(b=True, linestyle='-', linewidth='.5', color='#ddd', alpha=0.7)

# color-map 
#cmap = 'viridis'
#cmap = 'inferno'
#cmap = 'jet'
cmap = 'magma'

plt.imshow(img_grid, aspect='auto', cmap=cmap)

 
The first figure contains the original MNIST image. The second figure will contain the grid with its images of the maps’ output. The code is straightforward; the corrections of the dimensions have to do with the display of intermittent lines to separate the different images. Statements like “ax.set_xticklabels([])” set the tick-mark-texts to empty strings. At the end of the code we choose a color map.

Note that I avoided to standardize the image data. Clipping suppresses extreme values; however, the map-related filters react to these values. So, let us keep the full value spectrum for a while …

Training run to get a reference model

I performed a training run with the following setting and saved the last model: 

build = True
if cnn == None:
    build = True
    x_optimizer = None 
batch_size=64
epochs=80
reset = False # we want training to start again with the initial weights
#reset = True # we want training to start again with the initial weights

my_loss    ='categorical_crossentropy'
my_metrics =['accuracy']

my_regularizer = None
my_regularizer = 'l2'
my_reg_param_l2 = 0.001
#my_reg_param_l2 = 0.01
my_reg_param_l1 = 0.01


my_optimizer      = 'rmsprop'       # Present alternatives:  rmsprop, nadam, adamax 
my_momentum       = 0.5           # momentum value 
my_lr_sched       = 'powerSched'    # Present alternatrives: 
None, powerSched, exponential 
#my_lr_sched       = None           # Present alternatrives: None, powerSched, exponential 
my_lr_init        = 0.001           # initial leaning rate  
my_lr_decay_steps = 1               # decay steps = 1 
my_lr_decay_rate  = 0.001           # decay rate 


li_conv_1    = [32, (3,3), 0] 
li_conv_2    = [64, (3,3), 0] 
li_conv_3    = [128, (3,3), 0] 
li_Conv      = [li_conv_1, li_conv_2, li_conv_3]
li_Conv_Name = ["Conv2D_1", "Conv2D_2", "Conv2D_3"]
li_pool_1    = [(2,2)]
li_pool_2    = [(2,2)]
li_Pool      = [li_pool_1, li_pool_2]
li_Pool_Name = ["Max_Pool_1", "Max_Pool_2", "Max_Pool_3"]
li_dense_1   = [100, 0]
#li_dense_2  = [30, 0]
li_dense_3   = [10, 0]
li_MLP       = [li_dense_1, li_dense_2, li_dense_3]
li_MLP       = [li_dense_1, li_dense_3]
input_shape  = (28,28,1)

 

This run gives us the following results:

and 

Epoch 80/80
933/938 [============================>.] - ETA: 0s - loss: 0.0030 - accuracy: 0.9998
present lr:  1.31509732e-05
present iteration: 75040
938/938 [==============================] - 4s 5ms/step - loss: 0.0030 - accuracy: 0.9998 - val_loss: 0.0267 - val_accuracy: 0.9944

Tests and first impressions of the convolutional layer output

Ok, let us test the code to plot the maps’ output. For the input data

# layer - pick one of the names which you defined for your model 
layer_name = "Conv2D_1"

# choose a image_set and an img number 
img_set = "test_imgs"
num_img = 19

we get the following results:

Layer “Conv2D_1”

Layer “Conv2D_2”

Layer “Conv2D_3”

Conclusion

Keras’ flexibility regarding model definitions allows for the definition of new models based on parts of the original CNN. The output layer of these new models can be set to any of the convolutional or pooling layers. With predictions for an input image we can extract the activation results of all maps of a layer. These data can be visualized in form of a grid that shows the reaction of a layer to the input image. A first test shows that the representations of the input get more and more abstract with higher convolutional layers.

In the next article

A simple CNN for the MNIST dataset – V – about the difference of activation patterns and features

we shall have a closer look of what these abstractions may mean for the classification of certain digit images.

Links

https://keras.io/getting_started/faq/#how-can-i-obtain-the-output-of-an-intermediate-layer-feature-extraction

https://machinelearningmastery.com/how-to-visualize-filters-and-feature-maps-in-convolutional-neural-networks/

https://towardsdatascience.com/visualizing-intermediate-activation-in-convolutional-neural-networks-with-keras-260b36d60d0

https://hackernoon.com/visualizing-parts-of-convolutional-neural-networks-using-keras-and-cats-5cc01b214e59

https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/5.4-visualizing-what-convnets-learn.ipynb

A simple CNN for the MNIST dataset – III – inclusion of a learning-rate scheduler, momentum and a L2-regularizer

Posted on by

In the last article of this series

A simple CNN for the MNIST datasets – II – building the CNN with Keras and a first test
A simple CNN for the MNIST datasets – I – CNN basics

we saw that we could easily create a Convolutional Neural Network [CNN] with Keras and apply it to the MNIST dataset. But this was only a quick win. Readers who followed me through the MLP-oriented article series may have noticed that we are far, yet, from having the full control over a variety of things we would use to optimize the classification algorithm. After our experience with MLPs you may rightfully argue that

  • both a systematic reduction of the learning rate in the vicinity of a (hopefully) global minimum of the multidimensional loss function hyperplane in the {weights, costs}-space
  • and the use of a L2- or L1-regularizer

could help to squeeze out the last percentage of accuracy of a CNN. Actually, in our first test run we did not even care about how the RMSProp algorithm invoked a learning rate and what the value(s) may have been. A learning rate “lr” was no parameter of our CNN setup so far. The same is true for the use and parameterization of a regularizer.

I find it interesting that some of the introductory books on “Deep Learning” with Keras do not discuss the handling of learning rates beyond providing a fixed (!) rate parameter on rare occasions to a SGD- or RMSProp-optimizer. Folks out there seem to rely more on the type and basic settings of an optimizer than on a variable learning rate. Within the list of books I appended to the last article, only the book of A. Geron gives you some really useful hints. So, we better find information within the online documentation of the Keras framework to get control over learning rates and regularizers.

Against my original plans, the implementation of schedulers for the learning rate and the usage of a L2-regularizer will be the main topic of this article. But I shall – as a side-step – also cover the use of “callbacks” for some interactive plotting during a training run. This is a bit more interesting than just plotting key data (loss, accuracy) after training – as we did for MLPs.

Learning rate scheduler and momentum

A systematic reduction of the learning rate or the momentum can be handled with Keras via a parameterization of a scheduler. Such a scheduler can e.g. be invoked by a chosen optimizer. As the optimizer came into our CNN with the model.compile()-function we would not be astonished, if we had to include the scheduler there, too. The Keras version of Tensorflow 2 offers a variety of (partially) configurable schedulers. And there are, in addition, multiple different ways how to integrate a systematic decline of the learning rate into a Keras CNN model. I choose a very simple method for this article, which is, however, specific for the TF2 variant of Keras (tensorflow.keras). The book of Geron discusses other alternatives (e.g. the invocation of a scheduler via a so called callback for batches; see below).

The optimizer for the gradient calculation directly accepts parameters for an (initial and constant) learning rate (parameter “learning_rate”) and a momentum (parameter “momentum”). An example with the SGD-optimizer would be: 

optimizer =  keras.optimizers.SGD(learning_rate=0.001, momentum=0.5)
cnn.compile(optimizer=optimizer, ....

If you just provide a parameter “learning_rate=0.001” then the optimizer will use a constant learning rate. However, if we provide a scheduler object – an instance of a scheduler class – then we can control a certain decline of the learning rate. What scheduler classes are available? You find them here:
keras.io/api/optimizers/ learning rate schedules/

A simple scheduler which allows for the “power scheduling” which we implemented for our MLPs is the scheduler “InverseTimeDecay“. It can be found under the library branch optimizers.schedules. We take this scheduler as an example in this article.

But how do we deliver the scheduler object to the optimizer? With tensorflow.keras and TF > 2.0 this is simply done by providing it as the parameter (object) for the learning_rate to the optimizer, as e.g. i the following code example: 

from tensorflow.keras import optimizers
from tensorflow.keras.optimizers import schedules
...
...
scheduled_lr_rate = schedules.InverseTimeDecay(lr_init, lr_decay_steps, lr_decay_rate) 
optimizer=optimizers.RMSprop(learning_rate=scheduled_lr_rate, momentum=my_momentum)
...

Here “lr_init” defines an initial learning rate (as 0.001), “lr_decay_steps” a number of steps, after which the rate is changed, and lr_decay_rate a decay rate to be applied in the formula of the scheduler. Note that different schedulers take different parameters – so look them up before applying them. Also check that your optimizer accepts a scheduler object as a parameter …

The scheduler instance works as if we had a function returning an output – a learning rate – for an input, which corresponds to a number of “steps“:

learning_rate=scheduler(steps) .

Now what do we mean by “steps”? Epochs? No, actually a step here corresponds practically to the result of an iterator over batches. Each optimization “step” in the sense of a weight adjustment after a gradient analysis for a mini-batch defines a step for the scheduler. If you want to change the learning rate on the level of epochs only, then you must either adjust the “decay_step”-parameter or write a so called “callback-function” invoked after each epoch and control the learning rate yourself.

Momentum
Note, by the way, that in addition to the scheduler we also provided a value for the “momentum” parameter an optimizer may use during the gradient evolution via some specific adaptive formula. How the real weight changes are calculated based on gradient development and momentum parameters is of course specific for an optimizer.

Required import statements

We use the Jupyter cells of the code we built in the last article as far as possible. We need to perform some more imports to work with schedulers, regularizers and plotting. You can exchange the contents of the first Jupyter cell with the following statements:

Jupyter Cell 1:

  
import numpy as np
import scipy
import time 
import sys 

from sklearn.preprocessing import StandardScaler
import tensorflow as tf
from tensorflow import keras as K
from tensorflow.python.keras import backend as B 
from tensorflow.keras import models
from tensorflow.keras import layers
from tensorflow.keras import regularizers
from tensorflow.keras import optimizers
from tensorflow.keras.optimizers import schedules
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.datasets import mnist

from tensorflow.python.client import device_lib

import matplotlib as mpl
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
import matplotlib.patches as mpat 

import os

 
Then we have two unchanged cells:

Jupyter Cell 2:

#gpu = False 
gpu = True
if gpu: 
#    GPU = True;  CPU = False; num_GPU = 1; num_CPU = 1
    GPU = True;  CPU = False; num_GPU = 1; num_CPU = 1
else: 
    GPU = False; CPU = True;  num_CPU = 1; num_GPU = 0

config = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=6,
                        inter_op_parallelism_threads=1, 
                        allow_soft_placement=True,
                        device_count = {'CPU' : num_CPU,
                                        'GPU' : num_GPU}, 
                        log_device_placement=True

                       )
config.gpu_options.per_process_gpu_memory_fraction=0.35
config.gpu_options.force_gpu_compatible = True
B.set_session(tf.compat.v1.Session(config=config))

 
and

Jupyter Cell 3:

# load MNIST 
# **********
#@tf.function
def load_Mnist():
    mnist = K.datasets.mnist
    (X_train, y_train), (X_test, y_test) = mnist.load_data()

    #print(X_train.shape)
    #print(X_test.shape)

    # preprocess - flatten 
    len_train =  X_train.shape[0]
    len_test  =  X_test.shape[0]
    X_train = X_train.reshape(len_train, 28*28) 
    X_test  = X_test.reshape(len_test, 28*28) 

    #concatenate
    _X = np.concatenate((X_train, X_test), axis=0)
    _y = np.concatenate((y_train, y_test), axis=0)

    _dim_X = _X.shape[0]

    # 32-bit
    _X = _X.astype(np.float32)
    _y = _y.astype(np.int32)

    # normalize  
    scaler = StandardScaler()
    _X = scaler.fit_transform(_X)

    # mixing the training indices - MUST happen BEFORE encoding
    shuffled_index = np.random.permutation(_dim_X)
    _X, _y = _X[shuffled_index], _y[shuffled_index]

    # split again 
    num_test  = 10000
    num_train = _dim_X - num_test
    X_train, X_test, y_train, y_test = _X[:num_train], _X[num_train:], _y[:num_train], _y[num_train:]

    # reshape to Keras tensor requirements 
    train_imgs = X_train.reshape((num_train, 28, 28, 1))
    test_imgs  = X_test.reshape((num_test, 28, 28, 1))
    #print(train_imgs.shape)
    #print(test_imgs.shape)

    # one-hot-encoding
    train_labels = to_categorical(y_train)
    test_labels  = to_categorical(y_test)
    #print(test_labels[4])
    
    return train_imgs, test_imgs, train_labels, test_labels

if gpu:
    with tf.device("/GPU:0"):
        train_imgs, test_imgs, train_labels, test_labels= load_Mnist()
else:
    with tf.device("/CPU:0"):
        train_imgs, test_imgs, train_labels, test_labels= load_Mnist()

 

Include the regularizer via the layer definitions of the CNN

A “regularizer” modifies the loss function via a sum over contributions delivered by a (common) function of the weight at every node. In Keras this sum is split up into contributions at the different layers, which we define via the “model.add()”-function. You can build layers with or without regularization contributions. Already for our present simple CNN case this is very useful:

In a first trial we only want to add weights to the hidden and the output layers of the concluding MLP-part of our CNN. To do this we have to provide a parameter “kernel_regularizer” to “model.add()”, which specifies the type of regularizer to use. We restrict the regularizers in our example to a “l2”- and a “l1”-regularizer, for which Keras provides predefined functions/objects. This leads us to the following change of the function “build_cnn_simple()”, which we used in the last article:

Jupyter Cell 4:

# Sequential layer 
model of our CNN
# ***********************************

# just for illustration - the real parameters are fed later 
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
li_conv_1   = [32, (3,3), 0] 
li_conv_2   = [64, (3,3), 0] 
li_conv_3   = [64, (3,3), 0] 
li_Conv     = [li_conv_1, li_conv_2, li_conv_3]
li_pool_1   = [(2,2)]
li_pool_2   = [(2,2)]
li_Pool     = [li_pool_1, li_pool_2]
li_dense_1  = [70, 0]
li_dense_2  = [10, 0]
li_MLP      = [li_dense_1, li_dense_2]
input_shape = (28,28,1)

# important !!
# ~~~~~~~~~~~
cnn = None
x_optimizers = None 

def build_cnn_simple(li_Conv, li_Pool, li_MLP, input_shape, 
                     my_regularizer=None, 
                     my_reg_param_l2=0.01, my_reg_param_l1=0.01 ):

    use_regularizer = True
    if my_regularizer == None:
        use_regularizer = False  
    
    # activation functions to be used in Conv-layers 
    li_conv_act_funcs = ['relu', 'sigmoid', 'elu', 'tanh']
    # activation functions to be used in MLP hidden layers  
    li_mlp_h_act_funcs = ['relu', 'sigmoid', 'tanh']
    # activation functions to be used in MLP output layers  
    li_mlp_o_act_funcs = ['softmax', 'sigmoid']

    # dictionary for regularizer functions
    d_reg = {
        'l2': regularizers.l2,  
        'l1': regularizers.l1
    }
    if use_regularizer: 
        if my_regularizer not in d_reg:
            print("regularizer " + my_regularizer + " not known!")
            sys.exit()
        else: 
            regul = d_reg[my_regularizer] 
        if my_regularizer == 'l2':
            reg_param = my_reg_param_l2
        elif my_regularizer == 'l1':
            reg_param = my_reg_param_l1
    
    
    # Build the Conv part of the CNN
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    num_conv_layers = len(li_Conv)
    num_pool_layers = len(li_Pool)
    if num_pool_layers != num_conv_layers - 1: 
        print("\nNumber of pool layers does not fit to number of Conv-layers")
        sys.exit()
    rg_il = range(num_conv_layers)

    # Define a sequential model
    # ~~~~~~~~~~~~~~~~~~~~~~~~~
    cnn = models.Sequential()

    for il in rg_il:
        # add the convolutional layer 
        num_filters = li_Conv[il][0]
        t_fkern_size = li_Conv[il][1]
        cact        = li_conv_act_funcs[li_Conv[il][2]]
        if il==0:
            cnn.add(layers.Conv2D(num_filters, t_fkern_size, activation=cact, input_shape=input_shape))
        else:
            cnn.add(layers.Conv2D(num_filters, t_fkern_size, activation=cact))
        
        # add the pooling layer 
        if il < num_pool_layers:
            t_pkern_size = li_Pool[il][0]
            cnn.add(layers.MaxPooling2D(t_pkern_size))
            

    # Build the MLP part of the CNN
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    num_mlp_layers = len(li_MLP)
    rg_im = range(num_mlp_layers)

    cnn.add(layers.Flatten())

    for im in rg_im:
        # add the dense layer 
        n_nodes = li_MLP[im][0]
        if im < num_mlp_layers - 1:  
            m_act   =  li_mlp_h_act_funcs[li_MLP[im][1]]
            if use_regularizer:
                cnn.add(layers.Dense(n_nodes, activation=m_act, kernel_regularizer=regul(reg_param)))
            else:
                cnn.add(layers.Dense(n_nodes, activation=m_act))
        else: 
            m_act   =  li_mlp_o_act_funcs[li_MLP[im][1]]
            if use_regularizer:
                cnn.add(layers.Dense(n_nodes, activation=m_act, kernel_regularizer=regul(reg_param)))
            else:
                cnn.add(layers.Dense(n_nodes, activation=m_act))
                
    return cnn

 

I have used a dictionary to enable an indirect call of the regularizer object. The invocation of a regularizer happens in
the statements:

cnn.add(layers.Dense(n_nodes, activation=m_act, kernel_regularizer=regul(reg_param)))

We only use a regularizer for MLP-layers in our example.

Note that I used predefined regularizer objects above. If you want to use a regularizer object defined and customized by yourself you can either define a simple regularizer function (which accepts the weights as a parameter) or define a child class of the “keras.regularizers.Regularizer”-class. You find hints how to do this at https://keras.io/api/layers/regularizers/ and in the book of Aurelien Geron (see the literature list at the end of the last article).

Although I have limited the use of a regularizer in the code above to the MLP layers you are of course free to apply a regularizer also to convolutional layers. (The book of A. Geron gives some hints how to avoid code redundancies in chapter 11).

Include a scheduler via a parameter to the optimizer of the CNN

As we saw above a scheduler for the learning rate can be provided as a parameter to the optimizer of a CNN. As the optimizer objects are invoked via the function “model.compile()”, we prepare our own function “my_compile()” with an appropriate interface, which then calls “model.compile()” with the necessary parameters. For indirect calls of predefined scheduler objects we again use a dictionary:

Jupyter Cell 5:

  
# compilation function for further parameters 
def my_compile(cnn, 
               my_loss='categorical_crossentropy', my_metrics=['accuracy'], 
               my_optimizer='rmsprop', my_momentum=0.0, 
               my_lr_sched='powerSched', 
               my_lr_init=0.01, my_lr_decay_steps=1, my_lr_decay_rate=0.00001  ):
    
    # dictionary for the indirct call of optimizers 
    d_optimizer = {
        'rmsprop': optimizers.RMSprop,  
        'nadam':   optimizers.Nadam,
        'adamax':  optimizers.Adamax
    }
    if my_optimizer not in d_optimizer:
        print("my_optimzer" + my_optimizer + " not known!")
        sys.exit()
    else: 
        optim = d_optimizer[my_optimizer]
        
    use_scheduler = True
    if my_lr_sched == None:
        use_scheduler = False
        print("\n No scheduler will be used")
    
    # dictionary for the indirct call of scheduler funcs 
    d_sched = {
        'powerSched' : schedules.InverseTimeDecay,  
        'exponential': schedules.ExponentialDecay
    }
    if use_scheduler:
        if my_lr_sched not in d_sched:
            print("lr scheduler " + my_lr_sched + " not known!")
            sys.exit()
        else: 
            sched = d_sched[my_lr_sched] 
    
    if use_scheduler:
        print("\n lr_init = ", my_lr_init, " decay_steps = ", my_lr_decay_steps, " decay_rate = ", my_lr_decay_rate)
        scheduled_lr_rate   = sched(my_lr_init, my_lr_decay_steps, my_lr_decay_rate)
        optimizer = optim(learning_rate=scheduled_lr_rate, momentum=my_momentum)  
    else:
        print("\n lr_init = ", my_lr_init)
        optimizer=optimizers.RMSprop(learning_rate=my_lr_init, momentum=my_momentum)
    
    cnn.compile(optimizer=optimizer, loss=my_loss, metrics=my_metrics)  
    
    return cnn

 
You see that, for the time being, I only offered the choice between the predefined schedulers “InverseTimeDecay” and “ExponentialDecay“. If no scheduler name is provided then only a constant learning rate is delivered to the chosen optimizer.

You find some basic information on optimizers and schedulers here:
https://keras.io/api/optimizers/
nhttps://keras.io/api/optimizers/ learning rate schedules/.

Note that you can build your own scheduler object via defining a child class of “keras.callbacks.LearningRateScheduler” and provide with a list of other callbacks to the “model.fit()“-function; see
https://keras.io/api/callbacks/ learning rate scheduler/.
or the book of Aurelien Geron for more information.

Two callbacks – one for printing the learning rate and an other for plotting during training

As I am interested in the changes of the learning rate with the steps during an epoch I want to print them after each epoch during training. In addition I want to plot key data produced during the training of a CNN model.

For both purposes we can use a convenient mechanism Keras offers – namely “callbacks”, which can either be invoked at the beginning/end of the treatment of a mini-batch or at the beginning/end of each epoch. Information on callbacks is provided here:
https://keras.io/api/callbacks/
https://keras.io/guides/ writing your own callbacks/

It is best to just look at examples to get the basic points; below we invoke two callback objects which provide data for us at the end of each epoch.

Jupyter Cell 6:

  
# Preparing some callbacks 
# **************************

# Callback class to print information on the iteration and the present learning rate 
class LrHistory(K.callbacks.Callback):
    def __init__(self, use_scheduler):
        self.use_scheduler = use_scheduler
        
    def on_train_begin(self, logs={}):
        self.lr = []

    def on_epoch_end(self, epoch, logs={}):
        if self.use_scheduler: 
            optimizer = self.model.optimizer
            iterations = optimizer.iterations
            present_lr = optimizer.lr(iterations)   
        else:
            present_lr = optimizer.lr
        self.lr.append(present_lr)
        #print("\npresent lr:", present_lr) 
        tf.print("\npresent lr: ", present_lr)
        tf.print("present iteration:",  iterations)


# Callback class to support interactive printing  
class EpochPlot(K.callbacks.Callback):
    def __init__(self, epochs, fig1, ax1_1, ax2_2):
    #def __init__(self, epochs):
        self.fig1  = fig1
        self.ax1_1 = ax1_1
        self.ax1_2 = ax1_2
        self.epochs = epochs
        rg_i = range(epochs)
        self.lin92 = []
        for i in rg_i:
            self.lin92.append(0.992)

    def on_train_begin(self, logs={}):
        self.train_loss = []
        self.train_acc  = []
        self.val_loss   = []
        self.val_acc    = []
        
    def on_epoch_end(self, epoch, logs={}):
        if epoch == 0:
            for key in logs.keys():
                print(key)
        self.train_loss.append(logs.get('loss'))
        self.train_acc.append(logs.get('accuracy'))
        self.val_loss.append(logs.get('val_loss'))
        self.val_acc.append(logs.get('val_accuracy'))
        if len(self.train_acc) > 0: 
            self.ax1_1.clear()
            self.ax1_1.set_xlim (0, self.epochs+1)
            self.ax1_1.set_ylim (0.97, 1.001)
            self.ax1_1.plot(self.train_acc, 'g' )
            self.ax1_1.plot(self.val_acc,   'r' )
            self.ax1_1.plot(self.lin92,     'b', linestyle='--' )
            
            self.ax1_2.clear()
            self.ax1_2.set_xlim (0, self.epochs+1)
            self.ax1_2.set_ylim (0.0, 0.2)
            self.ax1_2.plot(self.train_loss, 'g' )
     
       self.ax1_2.plot(self.val_loss,   'r' )
 
            self.fig1.canvas.draw()

 
From looking at the code we see that a callback can be defined as a child class of a base class “keras.callbacks.Callback” or of some specialized predefined callback classes listed under “https://keras.io/api/callbacks/“. They are all useful, but perhaps the classes “ModelCheckpoint”, “LearningRateScheduler” (see above) and “EarlyStopping” are of direct practical interest for people who want to move beyond the scope of this article. In the above example I, however, only used the base class “callbacks.Callback”.

Printing steps, iterations and the learning rate
Let us look at the first callback, which provides a printout of the learning rate. This is a bit intricate: Regarding the learning rate I have already mentioned that a scheduler changes it after each operation on a batch; such a step corresponds to a variable “iterations” which is counted up by the optimizer during training after the treatment of each mini-batch. [Readers from my series on MLPs remember that gradient descent can be based on a gradient evaluation over the samples of mini-batches – instead of using a gradient evaluation based on all samples (batch gradient descent or full gradient descent) or of each individual sample (stochastic gradient descent).]

As we defined a batch size = 64 for the fit()-method of our CNN in the last article the number of optimizer iterations (=steps) per epoch is quite big: 60000/64 => 938 (with a smaller last batch).

Normally, the constant initial learning rate of an optimizer could be retrieved in a callback by a property “lr” as in “self.model.optimizer.lr”. However, in our example this attribute now points to a method “schedule()” of the scheduler object. (See schedule() method of the LearningRateScheduler class). We must provide the number of “iterations” (= number of steps) as a parameter to this schedule()-function and take the returned result as the present lr-value.

Our own callback class “LrHistory(K.callbacks.Callback)” gets three methods which are called at different types of “events” (Javascript developers should feel at home 🙂 ):

  • At initialization (__init__()) we provide a parameter which defines whether we used a scheduler at all, or a constant lr.
  • At the beginning of the training (on_train_begin()) we just instantiate a list, which we later fill with retrieved lr-values; we could use this list for some evaluation e.g. at the end of the training.
  • At the end of each epoch (on_epoch-end()) we determine the present learning rate via calling the scheduler (behind the “lr” attribute) – if we used one – for the number of iterations passed.

This explains the following statements: 

optimizer = self.model.optimizer
iterations = optimizer.iterations
present_lr = optimizer.lr(iterations)

Otherwise we just print the constant learning rate used by the optimizer (e.g. a default value).

Note that we use the “tf.print()” method to do the printing and not the Python “print()” function. This is basically done for convenience reasons: We thus avoid a special evaluation of the tensor and a manual transformation to a Numpy value. Do not forget: All data in Keras/TF are basically tensors and, therefore, the Python print() function would print extra information besides the expected numerical value!

Plotting at the end of an epoch

With our MLP we never used interactive plots in our Jupyter notebooks. We always plotted after training based on arrays or lists filled during
the training iterations with intermediately achieved values of accuracy and loss. But seeing the evolution of key metric data during training is a bit more rewarding; we get a feeling for what happens and can stop a training run with problematic hyperparameters before we waste too much valuable GPU or CPU time. So, how do we do change plots during training in our Jupyter environment?

The first point is that we need to prepare the environment of our Jupyter notebook; we do this before we start the training run. Below you find the initial statements of the respective Jupyter cell; the really important statement is the call of the ion-function: “plt.ion()”. For some more information see
https://linux-blog.anracom.com/2019/12/26/matplotlib-jupyter-and-updating-multiple-interactive-plots/
and links at the end of the article. 

 
# Perform a training run 
# ********************
# Prepare the plotting 
# The really important commands for interactive (=intermediate) plot updating
%matplotlib notebook
plt.ion()

#sizing
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 8
fig_size[1] = 3

# One figure 
# -----------
fig1 = plt.figure(1)
#fig2 = plt.figure(2)

# first figure with two plot-areas with axes 
# --------------------------------------------
ax1_1 = fig1.add_subplot(121)
ax1_2 = fig1.add_subplot(122)
fig1.canvas.draw()

 
The interesting statements are the first two. The statements for setting up the plot figures should be familiar to my readers already. The statement “fig1.canvas.draw()” leads to the display of two basic coordinate systems at the beginning of the run.

The real plotting is, however, done with the help of our second callback “EpochPlot()” (see the code given above for Jupyter cell 6). We initialize our class with the number of epochs for which our training run shall be performed. We also provide the addresses of the plot figures and their two coordinate systems (ax1_1 and ax1_2). At the beginning of the training we provide empty lists for some key data which we later (after each epoch) fill with calculated values.

Which key or metric data can the training loop of Keras provide us with?

Keras fills a dictionary “logs” with metric data and loss values. What the metric data are depends on the loss function. At the following link you find a discussion of the metrics you can use with respect to different kinds of ML problems: machinelearningmastery.com custom-metrics-deep-learning-keras-python/.

For categorical_crossentropy and our metrics list [‘accuracy’] we get the following keys

loss, accuracy, val_loss, val_accuracy

as we also provided validation data. We print this information out at epoch == 0. (You will note however, that epoch 0 is elsewhere printed out as epoch 1. So, the telling is a bit different inside Keras and outside).

Then have a look at the parameterization of the method on_epoch_end(self, epoch, logs={}). This should not disturb you; logs is only empty at the beginning, later on a filled logs-dictionary is automatically provided.

As you see form the code we retrieve the data from the logs-dictionary at the end of each epoch. The rest are plain matplot-commands The “clear()”-statements empty the coordinate areas; the statement “self.fig1.canvas.draw()” triggers a redrawing.

The new training function

To do things properly we now need to extend our training function which invokes the other already defined functions as needed:

Jupyter Cell 7:

# Training 2 - with test data integrated 
# *****************************************
def train( cnn, build, train_imgs, train_labels, 
           li_Conv, li_Pool, li_MLP, input_shape, 
           reset=True, epochs=5, batch_size=64, 
           my_loss='categorical_crossentropy', my_metrics=['accuracy'], 
           my_regularizer=None, 
           my_reg_param_l2=0.01, my_reg_param_l1=0.01, 
           my_optimizer='rmsprop', my_momentum=0.0, 
           my_lr_sched=None, 
           my_lr_init=0.001, my_lr_decay_steps=1, my_lr_decay_rate=0.00001,
           fig1=None, ax1_1=None, ax1_2=None
):
    
    if build:
        # build cnn layers - now with regularizer - 200603 by ralph
        cnn = build_cnn_simple( li_Conv, li_Pool, li_MLP, input_shape, 
                                my_regularizer = my_regularizer, 
                                my_reg_param_l2 = my_reg_param_l2, my_reg_param_l1 = my_reg_param_l1)
        
        # compile - now with lr_scheduler - 200603 by ralph
        cnn = my_compile(cnn=cnn, 
                         my_loss=my_loss, my_metrics=my_metrics, 
                         my_optimizer=my_optimizer, my_momentum=my_momentum, 
                         my_lr_sched=my_lr_sched,
                         my_lr_init=my_lr_init, my_lr_decay_steps=my_lr_decay_steps, 
                         my_lr_decay_rate=my_lr_decay_rate)        
        
        # save the inital (!) weights to be able to restore them  
        cnn.save_weights('cnn_weights.h5') # save the initial weights 
        
    # reset weights(standard)
    if reset and not build:
        cnn.load_weights('cnn_weights.h5')
 
    # Callback list 
    # ~~~~~~~~~~~~~
    use_scheduler = True
    if my_lr_sched == None:
        use_scheduler = False
    lr_history = LrHistory(use_scheduler)
    callbacks_list = [lr_history]
    if fig1 != None:
        epoch_plot = EpochPlot(epochs, fig1, ax1_1, ax1_2)
        callbacks_list.append(epoch_plot)
    
    start_t = time.perf_counter()
    if reset:
        history = cnn.fit(train_imgs, train_labels, initial_epoch=0, epochs=epochs, batch_size=batch_size, verbose=1, shuffle=True, 
                  validation_data=(test_imgs, test_labels), callbacks=callbacks_list) 
    else:
        history = cnn.fit(train_imgs, train_labels, epochs=epochs, batch_size=batch_size, verbose=1, shuffle=True, 
                validation_data=(test_imgs, test_labels), callbacks=callbacks_list ) 
    end_t = time.perf_counter()
    fit_t = end_t - start_t
    
    return cnn, fit_t, history, x_optimizer  # we return cnn to be able to use it by other functions in the Jupyter later on

 
Note, how big our interface became; there are a lot of parameters for the control of our training. Our set of configuration parameters is now very similar to what we used for MLP training runs before.

Note also how we provided a list of our callbacks to the “model.fit()”-function.

Setting up a training run

Now , we are only a small step away from testing our modified CNN-setup. We only need one further cell:

Jupyter Cell 8:

# Perform a training run 
# ********************

# Prepare the plotting 
# The really important command for interactive (=interediate) plot updating
%matplotlib notebook
plt.ion()

#sizing
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 8
fig_size[1] = 3

# One figure 
# -----------
fig1 = plt.figure(1)
#fig2 = plt.figure(2)

# first figure with two plot-areas with axes 
# --------------------------------------------
ax1_1 = fig1.add_subplot(121)
ax1_2 = fig1.add_subplot(122)
nfig1.canvas.draw()

# second figure with just one plot area with axes
# -------------------------------------------------
#ax2 = fig2.add_subplot(121)
#ax2_1 = fig2.add_subplot(121)
#ax2_2 = fig2.add_subplot(122)
#fig2.canvas.draw()

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Parameterization of the training run 

build = False
build = True
if cnn == None:
    build = True
    x_optimizer = None 
batch_size=64
epochs=80
reset = False # we want training to start again with the initial weights

my_loss    ='categorical_crossentropy'
my_metrics =['accuracy']

my_regularizer = None
my_regularizer = 'l2'
my_reg_param_l2 = 0.0009
my_reg_param_l1 = 0.01


my_optimizer      = 'rmsprop'       # Present alternatives:  rmsprop, nadam, adamax 
my_momentum       = 0.6             # momentum value 
my_lr_sched       = 'powerSched'    # Present alternatrives: None, powerSched, exponential 
#my_lr_sched       = None           # Present alternatrives: None, powerSched, exponential 
my_lr_init        = 0.001           # initial leaning rate  
my_lr_decay_steps = 1               # decay steps = 1 
my_lr_decay_rate  = 0.001           # decay rate 


li_conv_1   = [64, (3,3), 0] 
li_conv_2   = [64, (3,3), 0] 
li_conv_3   = [128, (3,3), 0] 
li_Conv     = [li_conv_1, li_conv_2, li_conv_3]
li_pool_1   = [(2,2)]
li_pool_2   = [(2,2)]
li_Pool     = [li_pool_1, li_pool_2]
li_dense_1  = [120, 0]
#li_dense_2  = [30, 0]
li_dense_3  = [10, 0]
li_MLP      = [li_dense_1, li_dense_2, li_dense_3]
li_MLP      = [li_dense_1, li_dense_3]
input_shape = (28,28,1)

try: 
    if gpu:
        with tf.device("/GPU:0"):
            cnn, fit_time, history, x_optimizer  = train( cnn, build, train_imgs, train_labels, 
                                            li_Conv, li_Pool, li_MLP, input_shape, 
                                            reset, epochs, batch_size, 
                                            my_loss=my_loss, my_metrics=my_metrics, 
                                            my_regularizer=my_regularizer, 
                                            my_reg_param_l2=my_reg_param_l2, my_reg_param_l1=my_reg_param_l1,  
                                            my_optimizer=my_optimizer, my_momentum = 0.8,  
                                            my_lr_sched=my_lr_sched, 
                                            my_lr_init=my_lr_init, my_lr_decay_steps=my_lr_decay_steps, 
                                            my_lr_decay_rate=my_lr_decay_rate,  
                                            fig1=fig1, ax1_1=ax1_1, ax1_2=ax1_2
                                            )
        print('Time_GPU: ', fit_time)  
    else:
        with tf.device("/CPU:0"):
            cnn, fit_time, history = train( cnn, build, train_imgs, train_labels, 
                                            li_Conv, li_Pool, li_MLP, input_shape, 
                                            reset, epochs, batch_size, 
                                            my_loss=my_loss, my_metrics=my_metrics, 
                                            my_regularizer=my_regularizer, 
                                            my_reg_param_l2=my_reg_param_l2, my_reg_param_l1=my_reg_param_l1,  
                                            my_optimizer=my_optimizer, my_momentum = 0.8, 
                                            my_lr_sched=my_lr_sched, 
                                            my_lr_init=my_lr_init, my_lr_decay_steps=my_lr_decay_steps, 
                                            my_lr_decay_rate=my_lr_decay_rate,  
                                            fig1=fig1, ax1_1=ax1_1, ax1_2=ax1_2
                                            )
        print('Time_CPU: ', fit_time)  
except SystemExit:
    print("stopped due to exception")

 

n

Results

Below I show screenshots taken during training for the parameters defined above.





The left plot shows the achieved accuracy; red for the validation set (around 99.32%). The right plot shows the decline of the loss function relative to the original value. The green lines are for the training data, the red for the validation data.

Besides the effect of seeing the data change and the plots evolve during data, we can also take the result with us that we have improved the accuracy already from 99.0% to 99.32%. When you play around with the available hyperparameters a bit you may find that 99.25% is a reproducible accuracy. But in some cases you may reach 99.4% as best values.

The following plots have best values around 99.4% with an average above 99.35%.

Epoch 13/80
935/938 [============================>.] - ETA: 0s - loss: 0.0096 - accuracy: 0.9986
present lr:  7.57920279e-05
present iteration: 12194
938/938 [==============================] - 6s 6ms/step - loss: 0.0096 - accuracy: 0.9986 - val_loss: 0.0238 - val_accuracy: 0.9944
Epoch 19/80
937/938 [============================>.] - ETA: 0s - loss: 0.0064 - accuracy: 0.9993
present lr:  5.3129319e-05
present iteration: 17822
938/938 [==============================] - 6s 6ms/step - loss: 0.0064 - accuracy: 0.9993 - val_loss: 0.0245 - val_accuracy: 0.9942
Epoch 22/80
930/938 [============================>.] - ETA: 0s - loss: 0.0056 - accuracy: 0.9994
present lr:  4.6219262e-05
present iteration: 20636
938/938 [==============================] - 6s 6ms/step - loss: 0.0056 - accuracy: 0.9994 - val_loss: 0.0238 - val_accuracy: 0.9942
Epoch 30/80
937/938 [============================>.] - ETA: 0s - loss: 0.0043 - accuracy: 0.9996
present lr:  3.43170905e-05
present iteration: 28140
938/938 [==============================] - 6s 6ms/step - loss: 0.0043 - accuracy: 0.9997 - val_loss: 0.0239 - val_accuracy: 0.9941
Epoch 55/80
937/938 [============================>.] - ETA: 0s - loss: 0.0028 - accuracy: 0.9998
present lr:  1.
90150222e-05
present iteration: 51590
938/938 [==============================] - 6s 6ms/step - loss: 0.0028 - accuracy: 0.9998 - val_loss: 0.0240 - val_accuracy: 0.9940
Epoch 69/80
936/938 [============================>.] - ETA: 0s - loss: 0.0025 - accuracy: 0.9999
present lr:  1.52156063e-05
present iteration: 64722
938/938 [==============================] - 6s 6ms/step - loss: 0.0025 - accuracy: 0.9999 - val_loss: 0.0245 - val_accuracy: 0.9940
Epoch 70/80
933/938 [============================>.] - ETA: 0s - loss: 0.0024 - accuracy: 0.9999
present lr:  1.50015e-05
present iteration: 65660
938/938 [==============================] - 6s 6ms/step - loss: 0.0024 - accuracy: 0.9999 - val_loss: 0.0245 - val_accuracy: 0.9940
Epoch 76/80
937/938 [============================>.] - ETA: 0s - loss: 0.0024 - accuracy: 0.9999
present lr:  1.38335554e-05
present iteration: 71288
938/938 [==============================] - 6s 6ms/step - loss: 0.0024 - accuracy: 0.9999 - val_loss: 0.0237 - val_accuracy: 0.9943

 

Parameters of the last run were 

build = True
if cnn == None:
    build = True
    x_optimizer = None 

batch_size=64
epochs=80
reset = True 

my_loss    ='categorical_crossentropy'
my_metrics =['accuracy']
my_regularizer = None
my_regularizer = 'l2'
my_reg_param_l2 = 0.0008
my_reg_param_l1 = 0.01

my_optimizer      = 'rmsprop'       # Present alternatives:  rmsprop, nadam, adamax 
my_momentum       = 0.5            # momentum value 
my_lr_sched       = 'powerSched'    # Present alternatrives: None, powerSched, exponential 
#my_lr_sched       = None           # Present alternatrives: None, powerSched, exponential 
my_lr_init        = 0.001           # initial leaning rate  
my_lr_decay_steps = 1               # decay steps = 1 
my_lr_decay_rate  = 0.001           # decay rate 

li_conv_1   = [64, (3,3), 0] 
li_conv_2   = [64, (3,3), 0] 
li_conv_3   = [128, (3,3), 0] 
li_Conv     = [li_conv_1, li_conv_2, li_conv_3]
li_pool_1   = [(2,2)]
li_pool_2   = [(2,2)]
li_Pool     = [li_pool_1, li_pool_2]
li_dense_1  = [120, 0]
#li_dense_2  = [30, 0]
li_dense_3  = [10, 0]
li_MLP      = [li_dense_1, li_dense_2, li_dense_3]
li_MLP      = [li_dense_1, li_dense_3]
input_shape = (28,28,1)

 
It is interesting that RMSProp only requires small values of the L2-regularizer for a sufficient stabilization – such that the loss curve for the validation data does not rise again substantially. Instead we observe an oscillation around a minimum after the learning rate as decreased sufficiently.

Addendum, 15.06.2020

An attentive reader has found out that I have cheated a bit: I have used 64 maps at the first convolutional layer instead of 32 according to the setup in the previous article. Yes, sorry. To compensate for this I include the plot of a run with 32 maps at the first convolution. The blue line marks an accuracy of 99.35%. You see that we can get above it with 32 maps, too.

The parameters were: 

build = True
if cnn == None:
    build = True
    x_optimizer = None 
batch_size=64
epochs=80
reset = False # reset the initial weight values to those saved?  

my_loss    ='categorical_crossentropy'
my_metrics =['accuracy']

my_regularizer = None
my_regularizer = 'l2'
my_reg_param_l2 = 0.001
#my_reg_param_l2 = 0.01
my_reg_param_l1 = 0.01

my_
optimizer      = 'rmsprop'       # Present alternatives:  rmsprop, nadam, adamax 
my_momentum       = 0.9           # momentum value 
my_lr_sched       = 'powerSched'    # Present alternatrives: None, powerSched, exponential 
#my_lr_sched       = None           # Present alternatrives: None, powerSched, exponential 
my_lr_init        = 0.001           # initial leaning rate  
my_lr_decay_steps = 1               # decay steps = 1 
my_lr_decay_rate  = 0.001           # decay rate 

li_conv_1    = [32, (3,3), 0] 
li_conv_2    = [64, (3,3), 0] 
li_conv_3    = [128, (3,3), 0] 
li_Conv      = [li_conv_1, li_conv_2, li_conv_3]
li_Conv_Name = ["Conv2D_1", "Conv2D_2", "Conv2D_3"]
li_pool_1    = [(2,2)]
li_pool_2    = [(2,2)]
li_Pool      = [li_pool_1, li_pool_2]
li_Pool_Name = ["Max_Pool_1", "Max_Pool_2", "Max_Pool_3"]
li_dense_1   = [120, 0]
#li_dense_2  = [30, 0]
li_dense_3   = [10, 0]
li_MLP       = [li_dense_1, li_dense_2, li_dense_3]
li_MLP       = [li_dense_1, li_dense_3]
input_shape  = (28,28,1)

  

Epoch 15/80
926/938 [============================>.] - ETA: 0s - loss: 0.0095 - accuracy: 0.9988
present lr:  6.6357e-05
present iteration: 14070
938/938 [==============================] - 4s 5ms/step - loss: 0.0095 - accuracy: 0.9988 - val_loss: 0.0268 - val_accuracy: 0.9940
Epoch 23/80
935/938 [============================>.] - ETA: 0s - loss: 0.0067 - accuracy: 0.9995
present lr:  4.42987512e-05
present iteration: 21574
938/938 [==============================] - 4s 5ms/step - loss: 0.0066 - accuracy: 0.9995 - val_loss: 0.0254 - val_accuracy: 0.9943
Epoch 35/80
936/938 [============================>.] - ETA: 0s - loss: 0.0049 - accuracy: 0.9996
present lr:  2.95595619e-05
present iteration: 32830
938/938 [==============================] - 4s 5ms/step - loss: 0.0049 - accuracy: 0.9996 - val_loss: 0.0251 - val_accuracy: 0.9945

Another stupid thing, which I should have mentioned:
I have not yet found a simple way of how to explicitly reset the number of iterations, only,

  • without a recompilation
  • or reloading a fully saved model at iteration 0 instead of reloading only the weights
  • or writing my own scheduler class based on epochs or batches.

With the present code you would have to (re-) build the model to avoid starting with a large number of “iterations” – and thus a small learning-rate in a second training run. My “reset”-parameter alone does not help.

By the way: Shape of the weight matrices

Some of my readers may be tempted to have a look at the weight tensors. This is possible via the optimizer and a callback. Then they may wonder about the dimensions, which are logically a bit different from the weight matrices I have used in my code for MLPs in another article series.

In the last article of this CNN series I have mentioned that Keras takes care of configuring the weight matrices itself as soon as it got all information about the layers and layers’ nodes (or units). Now, this still leaves some open degrees of handling the enumeration of layers and matrices; it is plausible that the logic of the layer and weight association must follow a convention. E.g., you can associate a weight matrix with the receiving layer in forward propagation direction. This is what Keras and TF2 do and it is different from what I did in my MLP code. Even if you have fixed this point, then you still can discuss the row/column-layout (shape) of the weight matrix:

Keras and TF2 require the “input_shape” i.e. the shape of the tensor which is fed into the present layer. Regarding the weight matrix it is the number of nodes (units) of the previous layer (in FW direction) which is interesting – normally this is given by the 2nd dimension number of the tensors shape. This has to be combined with the number of units in the present
layer. The systematics is that TF2 and Keras define the weight matrix between two layers according to the following convention for two adjacent layers L_N and L_(N+1) in FW direction:

L_N with A nodes (6 units) and L_(N+1) with B nodes (4 units) give a shape of the weight matrix as (A,B). [(6,4)]

See the references at the bottom of the article.

Note that this is NOT what we would expect from our handling of the MLP. However, the amount of information kept in the matrix is, of course, the same. It is just a matter of convention and array transposition for the required math operations during forward and error backward propagation. If you (for reasons of handling gradients) concentrate on BW propagation then the TF2/Keras convention is quite reasonable.

Conclusion

The Keras framework gives you access to a variety of hyperparameters you can use to control the use of a regularizer, the optimizer, the decline of the learning rate with batches/epochs, the momentum. The handling is a bit unconventional, but one gets pretty fast used to it.

We once again learned that it pays to invest a bit in a variation of such parameters, when you want to fight for the last tenth of percents beyond 99.0% accuracy. In the case of MNIST we can drive accuracy to 99.35%.

With the next article of this series

A simple CNN for the MNIST dataset – IV – Visualizing the activation output of convolutional layers and maps

we turn more to the question of visualizing some data at the convolutional layers to better understand their working.

Links

Layers and the shape of the weight matrices
https://keras.io/guides/sequential_model/
https://stackoverflow.com/questions/ 44791014/ understanding-keras-weight-matrix-of-each-layers
https://keras.io/guides/ making new layers and models via subclassing/

Learning rate and adaptive optimizers
https://towardsdatascience.com/learning-rate-schedules-and-adaptive-learning-rate-methods-for-deep-learning-2c8f433990d1
https://machinelearningmastery.com/ understand-the-dynamics-of-learning-rate-on-deep-learning-neural-networks/
https://faroit.com/keras-docs/2.0.8/optimizers/
https://keras.io/api/optimizers/
https://keras.io/api/optimizers/ learning_rate_schedules/
https://www.jeremyjordan.me/nn-learning-rate/

Keras Callbacks
https://keras.io/api/callbacks/
https://keras.io/guides/ writing_your_own_callbacks/
https://keras.io/api/callbacks/ learning_rate_scheduler/

Interactive plotting
https://stackoverflow.com/questions/ 39428347/ interactive-matplotlib-plots-in-jupyter-notebook
https://matplotlib.org/3.2.1/api/ _as_gen/ matplotlib.pyplot.ion.html
https://matplotlib.org/3.1.3/tutorials/ introductory/ usage.html

Further articles in this series

A simple CNN for the MNIST dataset – XI – Python code for filter visualization and OIP detection
A simple CNN for the MNIST dataset – X – filling some gaps in filter visualization
A simple CNN for the MNIST dataset – IX – filter visualization at a convolutional layer
A simple CNN for the MNIST dataset – VIII – filters and features – Python code to visualize patterns which activate a map strongly
A simple CNN for the MNIST dataset – VII – outline of steps to visualize image patterns which trigger filter maps
A simple CNN for the MNIST dataset – VI – classification by activation patterns and the role of the CNN’s MLP part
A simple CNN for the MNIST dataset – V – about the difference of activation patterns and features