A single neuron perceptron with sigmoid activation function – I – failure of gradient descent due to saturation

Readers who follow my series on a Python program for a "Multilayer Perceptron" [MLP] have noticed that I emphasized the importance of a normalization of input data in my last article:

A simple Python program for an ANN to cover the MNIST dataset – XII – accuracy evolution, learning rate, normalization

Our MLP "learned" by searching for the global minimum of the loss function via the "gradient descent" method. Normalization seemed to improve a smooth path to convergence whilst our algorithm moved into the direction of a global minimum on the surface of the loss or cost functions hyperplane over the weight parameter space. At least in our case where we used the sigmoid function as the activation function of the artificial neurons. I indicated that the reason for our observations had to do with properties of this function - especially for neurons in the first hidden layer of an MLP.

In case of the 4-layer MLP, which I used on the MNIST dataset, I applied a special form of normalization namely "standardization". I did this by using the StandarScaler of SciKit-Learn. See the following link for a description: Sklearn preprocessing StandardScaler

We saw the smoothing and helpful impact of normalization on the general convergence of gradient descent by the help of a few numerical experiments. The interaction of the normalization of 784 features with mini-batches and with a complicated 4 layer-MLP structure, which requires the determination of several hundreds of thousands weight values is however difficult to grasp or analyze. One understands that there is a basic relation to the properties of the activation function, but the sheer number of the dimensions of the feature and weight spaces and statistics make a thorough understanding difficult.

Since then I have thought a bit about how to set up a really small and comprehensible experiment which makes the positive impact of normalization visible in a direct and visual form. I have come up with the following most simple scenario, namely the training of a simple perceptron with only one computing neuron and two additional "stupid" neurons in an input layer which just feed input data to our computing neuron:

Input values K1 and K2 are multiplied by weights w1 and w2 and added by the central solitary "neuron". We use the sigmoid function as the "activation function" of this neuron - well anticipating that the properties of this function may lead to trouble.

The perceptron has only one task: Discriminate between two different types of input data by assigning them two distinct output values.

  • For K1 > 100 and K2 < 25 we want an output of A=0.3.
  • For K1 &l; 25 and K2 > 100 we, instead, want an output of A=0.7

We shall feed the perceptron only 14 different pairs of input values K1[i], K2[i] (i =0,1,..13), e.g. in form of lists:

li_K1 = [200.0,   1.0, 160.0,  11.0, 220.0,  11.0, 120.0,  22.0, 195.0,  15.0, 130.0,   5.0, 185.0,  16.0]
li_K2 = [ 14.0, 107.0,  10.0, 193.0,  32.0, 178.0,   2.0, 210.0,  12.0, 134.0,  15.0, 167.0,  10.0, 229.0] 

(The careful reader detects one dirty example li_K2[4] = 32 (> 25), in contrast to our setting. Just to see how much of an impact such a deviation has ...)

We call each pair (K1, K2)=(li_K1[i], li_K2[i]) for a give "i" a "sample". Each sample contains values for two "features": K1 and K2. So, our solitary computing neuron has to solve a typical classification problem - it shall distinguish between two groups of input samples. In a way it must learn the difference between small and big numbers for 2 situations appearing at its input channels.

Off topic: This morning I listened to a series of comments of Mr. Trump during the last weeks on the development of the Corona virus crisis in the USA. Afterwards, I decided to dedicate this mini-series of articles on a perceptron to him - a person who claims to "maybe" be "natural talent" on complicated things as epidemic developments. Enjoy (?) his own words via an audio compilation in the following news article:

Two well separable clusters

In the 2-dim feature space {K1, K2} we have just two clusters of input data:

Each cluster has a long diameter along one of the feature axes - but overall the clusters are well separable. A rather good separation surface would e.g. be a diagonal line.

For a given input sample with K-values K1 and K2 we define the output of our computing neuron to be

A(w1, w2) = expit( w1*K1 + w2*K2 ) ,

where expit() symbolizes the sigmoid function. See the next section for more details.

Corresponding target-values for the output A are (with at1 = 0.3 and at2 = 0.7):

li_a_tgt = [at1,  at2,  at1,  at2,  at1,  at2,  at1,  at2,  at1,  at2,  at1,  at2,   at1,  at2]

With the help of these target values our poor neuron shall learn from the 14 input samples to which cluster a given future sample probably belongs to. We shall use the "gradient descent" method to train the neuron for this classification task. o solve the task our neuron must find a reasonable separation surface - a line - in the {K1,K2}-plane; but it got the additional task to associate two distinct output values "A" with the two clusters:

A=0.7 for data points with a large K1-value and A=0.3 for data points with a small K1-value.

So, the separation surface has to fulfill some side conditions.

Readers with a background in Machine Learning and MLPs will now ask: Why did you pick the special values 0.3 and 0.7? A good question - I will come back to it during our experiments. Another even more critical question could be: What about a bias neuron in the input layer? Don't we need it? Again, a very good question! A bias neuron allows for a shift of a separation surface in the feature space. But due to the almost symmetrical nature of our input data (see the positions and orientations of the clusters!) and the target values the impact of a bias neuron on the end result would probably only be small - but we shall come back to the topic of a bias neuron in a later article. But you are free to extend the codes given below to account for a bias neuron in the input layer. You will notice a significant impact if you change either the relative symmetry of the input or of the output data. But lets keep things simple for the next hours ...

The sigmoid function and its saturation for big arguments

You can import the sigmoid function under the name "expit" from the "scipy" library into your Python code. The sigmoid function is a smooth one - but it quickly saturates for big negative or positive values:

So, output values get almost indistinguishable if the absolute values of the arguments are bigger than 15.

What is interesting about our input data? What is the relation to MNIST data?

The special situation about the features in our example is the following: For one and the same feature we have a big number and a small number to work with - depending on the sample. Which feature value - K1 or K2 - is big depends on the sample.

This is something that also happens with the input "features" (i.e. pixels) coming from a MNIST-image:
For a certain feature (= a pixel at a defined position in the 28x28 picture) in a given sample (= a distinct image) we may get a big number as 255. For another feature (= pixel) the situation may be totally different and we may find a value of 0. In another sample (= picture) we may get the reverse situation for the same two pixels.

What happens in such a situation at a specific neuron in the first hidden neuron layer of a MLP when we start gradient descent with a statistical distribution of weight values? If we are unlucky then the initial statistical weight constellation for a sample may lead to a big number of the total input to our selected hidden neuron with the effect of a very small gradient at this node - due to saturation of the sigmoid function.

To give you a feeling: Assume that you have statistical weights in the range of [-0.025, 0.025]. Assume further that only 4 pixels of a MNIST picture with a value of 200 contribute with a local maximum weight of 0.25; then we get a a minimum input at our node of 4*0.25*200 = 20. The gradient of expit(20) has a value of 2e-9. Even if we multiply by the required factor of 200 for a weight correction at one of the contributing input nodes we would would arrive at 4e-7. Quite hopeless. Of course, the situation is not that bad for all weights and image-samples, but you get an idea about the basic problem ....

Our simple scenario breaks the MNIST situation down to just two features and just one working neuron - and therefore makes the correction situation for gradient descent really extreme - but interesting, too. And we can analyze the situation far better than for a MLP because we deal with an only 2-dimensional feature-space {K1, K2} and a 2-dimensional weight-space {w1, w2}.

A simple quadratic cost function for our neuron

For given values w1 and w2, i.e. a tuple (w1, w2), we define a quadratic cost or loss function C_sgl for one single input sample (K1, K2) as follows:

C_sgl = 0.5 * ( li_a_tgt(i) - expit(z_i) )**2, with z_i = li_K[i]*w1 + li_K2[i]*w2

The total cost-function for the batch of all 14 samples is just the sum of all these terms for the individual samples.

Existence of a solution for our posed problem?

From all we theoretically know about the properties of a simple perceptron it should be able to find a reasonable solution! But, how do we know that a reasonable solution for a (w1, w2)-pair does exist at all? One line of reasoning goes like follows:

For two samples - each a member of either cluster - you can plot the hyperplanes of the related outputs "A(K1, K2) = expit(w1*K1+w2*K2)" over the (w1, w2)-space. These hyperplanes are almost orthogonal to each other. If you project the curves of a cut with the A=0.3-planes and the A=0.7-planes down to the (w1, w2)-plane at the center you get straight lines - almost orthogonally oriented against each other. So, such 2 specific lines cut each other in a well defined point - somewhere off the center. As the expit()-function is a relatively steep one for our big input values the crossing point is relatively close to the center. If we choose other samples we would get slightly different lines and different crossing points of the projected lines - but not too far from each other.

The next plot shows the expit()-functions close to the center of the (w1, w2)-plane for two specific samples of either cluster. We have in addition displayed the surfaces for A=0.7 and A=0.3.

The following plot shows the projections of the cuts of the surfaces for 7 samples of each cluster with the A=0.3-plane and the A=0.7, respectively.

The area of crossings is not too big on the chosen scale of w1, w2. Looking at the graphics we would expect an optimal point around (w1=-0.005, w2=+0.005) - for the original, unnormalized input data.

By the way: There is also a solution for at1=0.3 and at2=0.3, but a strange one. Such a setup would not allow for discrimination. We expect a rather strange behavior then. A first guess could be: The resulting separation curve in the (K1, K2)-plane would move out of the area between the two clusters.

Code for a mesh based display of the costs over the weight-parameter space

Below you find the code suited for a Jupyter cell to get a mesh display of the cost values

import numpy as np
import numpy as np
import random
import math 
import sys

from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import Normalizer 
from sklearn.preprocessing import MinMaxScaler
from scipy.special import expit  

import matplotlib as mpl
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
import matplotlib.patches as mpat 
from mpl_toolkits import mplot3d
from mpl_toolkits.mplot3d import Axes3D

# total cost functions for overview 
# *********************************
def costs_mesh(num_samples, W1, W2, li_K1, li_K2, li_a_tgt):
    zshape = W1.shape
    li_Z_sgl = []
    li_C_sgl = []
    C = np.zeros(zshape) 
    rg_idx = range(num_samples)
    for idx in rg_idx:
        Z_idx      = W1 * li_K1[idx] + W2 * li_K2[idx]
        A_tgt_idx  = li_a_tgt[idx] * np.ones(zshape) 
        C_idx = 0.5 * ( A_tgt_idx - expit(Z_idx) )**2
        li_C_sgl.append( C_idx )  
        C += C_idx 
    C /= np.float(num_samples)
    return C, li_C_sgl

# ******************
# Simple Perceptron  

# input at 2 nodes => 2 features K1 and K2 => there will be just one output neuron 
li_K1 = [200.0,   1.0, 160.0,  11.0, 220.0,  11.0, 120.0,  14.0, 195.0,  15.0, 130.0,   5.0, 185.0,  16.0  ]
li_K2 = [ 14.0, 107.0,  10.0, 193.0,  32.0, 178.0,   2.0, 210.0,  12.0, 134.0,  7.0, 167.0,  10.0, 229.0 ]

# target values 
at1 = 0.3; at2 = 0.7
li_a_tgt  = [at1,  at2,  at1,  at2,   at1,   at2,  at1,   at2,  at1,  at2,  at1,  at2,   at1,  at2 ]

# Change to np floats 
li_K1 = np.array(li_K1)
li_K2 = np.array(li_K2)
li_a_tgt = np.array(li_a_tgt)

num_samples = len(li_K1)

# Get overview over costs on mesh
# *******************************
# Mesh of weight values  
wm1 = np.arange(-0.2,0.4,0.002)
wm2 = np.arange(-0.2,0.2,0.002)
W1, W2 = np.meshgrid(wm1, wm2) 
# costs 
C, li_C_sgl  = costs_mesh(num_samples=num_samples, W1=W1, W2=W2, \
                          li_K1=li_K1, li_K2=li_K2, li_a_tgt = li_a_tgt)

# Mesh-Plots
# ********
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 19
fig_size[1] = 19

fig1 = plt.figure(1)
fig2 = plt.figure(2)

ax1 = fig1.gca(projection='3d')
ax1.get_proj = lambda: np.dot(Axes3D.get_proj(ax1), np.diag([1.0, 1.0, 1, 1]))
ax1.set_xlabel('w1', fontsize=16)
ax1.set_ylabel('w2', fontsize=16)
ax1.set_zlabel('single costs', fontsize=16)

#ax1.plot_wireframe(W1, W2, li_C_sgl[0], colors=('blue'))
#ax1.plot_wireframe(W1, W2, li_C_sgl[1], colors=('orange'))
ax1.plot_wireframe(W1, W2, li_C_sgl[6], colors=('orange'))
ax1.plot_wireframe(W1, W2, li_C_sgl[5], colors=('green'))
#ax1.plot_wireframe(W1, W2, li_C_sgl[9], colors=('orange'))
#ax1.plot_wireframe(W1, W2, li_C_sgl[6], colors=('magenta'))

ax2 = fig2.gca(projection='3d')
ax2.get_proj = lambda: np.dot(Axes3D.get_proj(ax2), np.diag([1.0, 1.0, 1, 1]))
ax2.set_xlabel('w1', fontsize=16)
ax2.set_ylabel('w2', fontsize=16)
ax2.set_zlabel('Total costs', fontsize=16)
ax2.plot_wireframe(W1, W2, 1.2*C, colors=('green'))


The cost landscape for individual samples without normalization

Gradient descent tries to find a minimum of a cost function by varying the weight values systematically in the cost gradient's direction. To get an overview about the cost hyperplane over the 2-dim (w1, w2)-space we make some plots. Let us first plot 2 the individual costs for the input samples i=0 and i=5.

Actually the cost functions for the different samples do show some systematic, though small differences. Try it out yourself ... Here is the plot for samples 1,5,9 (counted from 0!).

You see different orientation angles in the (w1, w2)-plane?

Total cost landscape without normalization

Now let us look at the total costs; to arrive at a comparable level with the other plots I divided the sum by 14:

All of the above cost plots look like big trouble for both the "stochastic gradient descent" and the "batch gradient descent" methods for "Machine Learning" [ML]:

We clearly see the effect of the sigmoid saturation. We get almost flat areas beyond certain relatively small w1- and w2-values (|w1| > 0.02, |w2| > 0.02). The gradients in this areas will be very, very close to zero. So, if we have a starting point as e.g. (w1=0.3, w2=0.2) our gradient descent would get stuck. Due to the big input values of at least one feature.

In the center of the {w1, w2}-plane, however, we detect a steep slope to a global minimum.

But how to get there? Let us say, we start with w1=0.04, w2=0.04. The learning-rate "η" is used to correct the weight values by

w1 = w1 - η*grad_w1
w2 = w2 - η*grad_w2

where grad_w1 and grad_w2 describe the relevant components of the cost-function's gradient.

In the beginning you would need a big "η" to get closer to the center due to small gradient values. However, if you choose it too big you may pass the tiny area of the minimum and just hop to an area of a different cost level with again a gradient close to zero. But you cannot decrease the learning rate fast as a remedy, either, to avoid getting stuck again.

A view at the center of the loss function hyperplane

Let us take a closer look at the center of our disastrous total cost function. We can get there by reducing our mesh to a region defined by "-0.05 < w1, w2 < 0.05". We get :

This looks actually much better - on such a surface we could probably work with gradient descent. There is a clear minimum visible - and on this scale of the (w1, w2)-values we also recognize reasonably paths into this minimum. An analysis of the meshdata to get values for the minimum is possible by the following statements:

print("min =", C.min())
pt_min = np.argwhere(C==C.min())
print("w1 = ", w1)
print("w2 = ", w2)

The result is:

min = 0.0006446277000906343
w1 = -0.004999999999999963
w2 = 0.005000000000000046

But to approach this minimum by a smooth gradient descent we would have had to know in advance at what tiny values of (w1, w2) to start with gradient descent - and choose our η suitably small in addition. This is easy in our most simplistic one neuron case, but you almost never can fulfill the first condition when dealing with real artificial neural networks for complex scenarios.

And a naive gradient descent with a standard choice of a (w1, w2)-starting point would have lead us to nowhere in our one-neuron case - as we shall see in a minute ..

Let us keep one question in mind for a while: Is there a chance that we could get the hyperplane surface to look similar to the one at the center - but for much bigger weight values?

Some Python code for gradient descent for our one neuron scenario

Here are some useful functions, which we shall use later on to perform a gradient descent:

# ****************************************
# Functions for stochastic GRADIENT DESCENT  
# *****************************************
import random
import pandas as pd

# derivative of expit 
def d_expit(z): 
    exz = expit(z)
    dex = exz * (1.0 - exz)
    return dex

# single costs for stochastic descent 
# ************************************
def dcost_sgl(w1, w2, idx, li_K1, li_K2, li_a_tgt):
    z_in  = w1 * li_K1[idx] + w2 * li_K2[idx] 
    a_tgt = li_a_tgt[idx] 
    c = 0.5 * ( a_tgt - expit(z_in))**2
    return c

# Gradients
# *********
def grad_sgl(w1, w2, idx, li_K1, li_K2, li_a_tgt):
    z_in  = w1 * li_K1[idx] + w2 * li_K2[idx] 
    a_tgt = li_a_tgt[idx] 
    gradw1 = 0.5 * 2.0 * (a_tgt - expit(z_in)) * (-d_expit(z_in)) * li_K1[idx]
    gradw2 = 0.5 * 2.0 * (a_tgt - expit(z_in)) * (-d_expit(z_in)) * li_K2[idx]
    grad = (gradw1, gradw2)
    return grad

def grad_tot(num_samples, w1, w2, li_K1, li_K2, li_a_tgt):
    gradw1 = 0 
    gradw2 = 0 
    rg_idx = range(num_samples)
    for idx in rg_idx:
        z_in  = w1 * li_K1[idx] + w2 * li_K2[idx] 
        a_tgt = li_a_tgt[idx] 
        gradw1_idx = 0.5 * 2.0 * (a_tgt - expit(z_in)) * (-d_expit(z_in)) * li_K1[idx]
        gradw2_idx = 0.5 * 2.0 * (a_tgt - expit(z_in)) * (-d_expit(z_in)) * li_K2[idx]
        gradw1 += gradw1_idx
        gradw2 += gradw2_idx
    #gradw1 /= float(num_samples) 
    #gradw2 /= float(num_samples) 
    grad = (gradw1, gradw2)
    return grad

# total costs at given point 
# ************************************
def dcost_tot(num_samples, w1, w2,li_K1, li_K2, li_a_tgt):
    c_tot  = 0
    rg_idx = range(num_samples)
    for idx in rg_idx:
        #z_in  = w1 * li_K1[idx] + w2 * li_K2[idx] 
        a_tgt = li_a_tgt[idx] 
        c_idx = dcost_sgl(w1, w2, idx, li_K1, li_K2, li_a_tgt) 
        c_tot += c_idx
    ctot = 1.0/num_samples * c_tot
    return c_tot

# Prediction function 
# ********************
def predict_batch(num_samples, w1, w2,ay_k_1, ay_k_2, li_a_tgt):
    shape_res = (num_samples, 5)
    ResData = np.zeros(shape_res)  
    rg_idx = range(num_samples)
    err = 0.0
    for idx in rg_idx:
        z_in  = w1 * ay_k_1[idx] + w2 * ay_k_2[idx] 
        a_out = expit(z_in)
        a_tgt = li_a_tgt[idx]
        err_idx = np.absolute(a_out - a_tgt) / a_tgt 
        err += err_idx
        ResData[idx][0] = ay_k_1[idx] 
        ResData[idx][1] = ay_k_2[idx] 
        ResData[idx][2] = a_tgt
        ResData[idx][3] = a_out
        ResData[idx][4] = err_idx
    err /= float(num_samples)
    return err, ResData    

def predict_sgl(k1, k2, w1, w2):
    z_in  = w1 * k1 + w2 * k2 
    a_out = expit(z_in)
    return a_out

def create_df(ResData):
    ''' ResData: Array with result values K1, K2, Tgt, A, rel.err 
    cols=["K1", "K2", "Tgt", "Res", "Err"]
    df = pd.DataFrame(ResData, columns=cols)
    return df    


With these functions a quick and dirty "gradient descent" can be achieved by the following code:

# **********************************
# Quick and dirty Gradient Descent  
# **********************************
b_scale_2 = False
if b_scale_2:
    ay_k_1 = ay_K1
    ay_k_2 = ay_K2
    ay_k_1 = li_K1
    ay_k_2 = li_K2

li_w1_st = []
li_w2_st = []
li_c_sgl_st = []
li_c_tot_st = []

li_w1_ba = []
li_w2_ba = []
li_c_sgl_ba = []
li_c_tot_ba = []

idxc = 2    
# Starting point
w1_start = -0.04
w2_start = -0.0455
#w1_start = 0.5
#w2_start = -0.5

# learn rate 
# **********
eta = 0.0001
decrease_rate = 0.000000001
num_steps = 2500 

# norm = 1
#eta = 0.75
#decrease_rate = 0.000000001
#num_steps = 100 

# Gradient descent loop
# *********************
rg_j = range(num_steps) 
rg_i = range(num_samples) 
w1d_st = w1_start
w2d_st = w2_start 
w1d_ba = w1_start
w2d_ba = w2_start 

for j in rg_j:
    eta = eta / (1.0 + float(j) * decrease_rate)
    gradw1 = 0
    gradw2 = 0
    # loop over samples and individ. corrs 
    ns = num_samples
    rg = range(ns)
    rg_idx = random.sample(rg, num_samples)
    for idx in rg_idx:  
        #print("idx = ", idx) 
        grad_st = grad_sgl(w1d_st, w2d_st, idx, ay_k_1, ay_k_2, li_a_tgt) 
        gradw1_st = grad_st[0]
        gradw2_st = grad_st[1]
        w1d_st -= gradw1_st * eta
        w2d_st -= gradw2_st * eta
        # costs for special sample 
        cd_sgl_st = dcost_sgl(w1d_st, w2d_st, idx, ay_k_1, ay_k_2, li_a_tgt) 

        # total costs for special sample 
        cd_tot_st = dcost_tot(num_samples, w1d_st, w2d_st, ay_k_1, ay_k_2, li_a_tgt) 
        #print("j:", j, " li_c_tot[j] = ", li_c_tot[j] )            

    # work with total costs and total gradient 
    grad_ba = grad_tot(num_samples, w1d_ba, w2d_ba, ay_k_1, ay_k_2, li_a_tgt)
    gradw1_ba = grad_ba[0]
    gradw2_ba = grad_ba[1]
    w1d_ba -= gradw1_ba * eta
    w2d_ba -= gradw2_ba * eta
    co_ba = dcost_tot(num_samples, w1d_ba, w2d_ba, ay_k_1, ay_k_2, li_a_tgt)    

# Printed Output
# ***************
num_end = len(li_w1_st)    
err_sgl, ResData_sgl = predict_batch(num_samples, li_w1_st[num_end-1], li_w2_st[num_end-1], ay_k_1, ay_k_2, li_a_tgt)
err_ba,  ResData_ba = predict_batch(num_samples, li_w1_ba[num_steps-1], li_w2_ba[num_steps-1], ay_k_1, ay_k_2, li_a_tgt)
df_sgl = create_df(ResData_sgl)
df_ba  = create_df(ResData_ba)
print("\n", df_sgl)
print("\n", df_ba)
print("\nTotal error stoch descent: ", err_sgl )            
print("Total error batch descent: ", err_ba )  

# Styled Pandas Output 
# *******************


Those readers who followed my series on a Multilayer Perceptron should have no difficulties to understand the code: I used two methods in parallel - one for a "stochastic descent" and one for a "batch descent":

  • During "stochastic descent" we correct the weights by a stepwise application of the cost-gradients of single samples. (We shuffle the order of the samples statistically during epochs to avoid cyclic effects.) This is done for all samples during an epoch.
  • During batch gradient we apply the gradient of the total costs of all samples once during each epoch.

And here is also some code to perform some plotting after training runs:

# Plots for Single neuron Gradient Descent        
# ****************************************
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 14
fig_size[1] = 5

fig1 = plt.figure(1)
fig2 = plt.figure(2)
fig3 = plt.figure(3)
fig4 = plt.figure(4)

ax1_1 = fig1.add_subplot(121)
ax1_2 = fig1.add_subplot(122)

ax1_1.plot(range(len(li_c_sgl_st)), li_c_sgl_st)
#ax1_1.set_xlim (0, num_tot+5)
#ax1_1.set_ylim (0, 0.4)
ax1_1.set_xlabel("epochs * batches (" + str(num_steps) + " * " + str(num_samples) + " )")
ax1_1.set_ylabel("sgl costs")

ax1_2.plot(range(len(li_c_tot_st)), li_c_tot_st)
#ax1_2.set_xlim (0, num_tot+5)
#ax1_2.set_ylim (0, y_max_err)
ax1_2.set_xlabel("epochs * batches (" + str(num_steps) + " * " + str(num_samples) + " )")
ax1_2.set_ylabel("total costs ")

ax2_1 = fig2.add_subplot(121)
ax2_2 = fig2.add_subplot(122)

ax2_1.plot(range(len(li_w1_st)), li_w1_st)
#ax1_1.set_xlim (0, num_tot+5)
#ax1_1.set_ylim (0, y_max_costs)
ax2_1.set_xlabel("epochs * batches (" + str(num_steps) + " * " + str(num_samples) + " )")

ax2_2.plot(range(len(li_w2_st)), li_w2_st)
#ax1_2.set_xlim (0, num_to_t+5)
#ax1_2.set_ylim (0, y_max_err)
ax2_2.set_xlabel("epochs * batches (" + str(num_steps) + " * " + str(num_samples) + " )")

ax3_1 = fig3.add_subplot(121)
ax3_2 = fig3.add_subplot(122)

ax3_1.plot(range(len(li_c_tot_ba)), li_c_tot_ba)
#ax3_1.set_xlim (0, num_tot+5)
#ax3_1.set_ylim (0, 0.4)
ax3_1.set_xlabel("epochs (" + str(num_steps) + " )")
ax3_1.set_ylabel("batch costs")

ax4_1 = fig4.add_subplot(121)
ax4_2 = fig4.add_subplot(122)

ax4_1.plot(range(len(li_w1_ba)), li_w1_ba)
#ax4_1.set_xlim (0, num_tot+5)
#ax4_1.set_ylim (0, y_max_costs)
ax4_1.set_xlabel("epochs (" + str(num_steps) + " )")

ax4_2.plot(range(len(li_w2_ba)), li_w2_ba)
#ax4_2.set_xlim (0, num_to_t+5)
#ax4_2.set_ylim (0, y_max_err)
ax4_2.set_xlabel("epochs (" + str(num_steps) + " )")


You can put these codes into suitable cells of a Jupyter environment and start doing experiments on your PC.

Frustration WITHOUT data normalization ...

Let us get the worst behind us:
Let us use un-normalized input data, set a standard starting point for the weights and try a gradient descent with 2500 epochs.
Well, what are standard initial weight values? We can follow LeCun's advice on bigger networks: a uniform distribution between - sqrt(1/2) and +srt(1/2) = 0.7 should be helpful. Well, we take such values. The parameters of our trial run are:

w1_start = -0.1, w2_start = 0.1, eta = 0.01, decrease_rate = 0.000000001, num_steps = 12500

You, unfortunately, get nowhere:

        K1     K2  Tgt           Res       Err
0   200.0   14.0  0.3  3.124346e-15  1.000000
1     1.0  107.0  0.7  9.999996e-01  0.428571
2   160.0   10.0  0.3  2.104822e-12  1.000000
3    11.0  193.0  0.7  1.000000e+00  0.428571
4   220.0   32.0  0.3  1.117954e-15  1.000000
5    11.0  178.0  0.7  1.000000e+00  0.428571
6   120.0    2.0  0.3  8.122661e-10  1.000000
7    14.0  210.0  0.7  1.000000e+00  0.428571
8   195.0   12.0  0.3  5.722374e-15  1.000000
9    15.0  134.0  0.7  9.999999e-01  0.428571
10  130.0    7.0  0.3  2.783284e-10  1.000000
11    5.0  167.0  0.7  1.000000e+00  0.428571
12  185.0   10.0  0.3  2.536279e-14  1.000000
13   16.0  229.0  0.7  1.000000e+00  0.428571

        K1     K2  Tgt           Res       Err
0   200.0   14.0  0.3  7.567897e-24  1.000000
1     1.0  107.0  0.7  1.000000e+00  0.428571
2   160.0   10.0  0.3  1.485593e-19  1.000000
3    11.0  193.0  0.7  1.000000e+00  0.428571
4   220.0   32.0  0.3  1.411189e-21  1.000000
5    11.0  178.0  0.7  1.000000e+00  0.428571
6   120.0    2.0  0.3  2.293804e-16  1.000000
7    14.0  210.0  0.7  1.000000e+00  0.428571
8   195.0   12.0  0.3  1.003437e-23  1.000000
9    15.0  134.0  0.7  1.000000e+00  0.428571
10  130.0    7.0  0.3  2.463730e-16  1.000000
11    5.0  167.0  0.7  1.000000e+00  0.428571
12  185.0   10.0  0.3  6.290055e-23  1.000000
13   16.0  229.0  0.7  1.000000e+00  0.428571

Total error stoch descent:  0.7142856616691734
Total error batch descent:  0.7142857142857143

A parameter setting like

w1_start = -0.1, w2_start = 0.1, eta = 0.0001, decrease_rate = 0.000000001, num_steps = 25000

does not bring us any further:

        K1     K2  Tgt           Res       Err
0   200.0   14.0  0.3  9.837323e-09  1.000000
1     1.0  107.0  0.7  9.999663e-01  0.428523
2   160.0   10.0  0.3  3.496673e-07  0.999999
3    11.0  193.0  0.7  1.000000e+00  0.428571
4   220.0   32.0  0.3  7.812207e-09  1.000000
5    11.0  178.0  0.7  9.999999e-01  0.428571
6   120.0    2.0  0.3  8.425742e-06  0.999972
7    14.0  210.0  0.7  1.000000e+00  0.428571
8   195.0   12.0  0.3  1.328667e-08  1.000000
9    15.0  134.0  0.7  9.999902e-01  0.428557
10  130.0    7.0  0.3  5.090220e-06  0.999983
11    5.0  167.0  0.7  9.999999e-01  0.428571
12  185.0   10.0  0.3  2.943780e-08  1.000000
13   16.0  229.0  0.7  1.000000e+00  0.428571

        K1     K2  Tgt           Res       Err
0   200.0   14.0  0.3  9.837323e-09  1.000000
1     1.0  107.0  0.7  9.999663e-01  0.428523
2   160.0   10.0  0.3  3.496672e-07  0.999999
3    11.0  193.0  0.7  1.000000e+00  0.428571
4   220.0   32.0  0.3  7.812208e-09  1.000000
5    11.0  178.0  0.7  9.999999e-01  0.428571
6   120.0    2.0  0.3  8.425741e-06  0.999972
7    14.0  210.0  0.7  1.000000e+00  0.428571
8   195.0   12.0  0.3  1.328667e-08  1.000000
9    15.0  134.0  0.7  9.999902e-01  0.428557
10  130.0    7.0  0.3  5.090220e-06  0.999983
11    5.0  167.0  0.7  9.999999e-01  0.428571
12  185.0   10.0  0.3  2.943780e-08  1.000000
13   16.0  229.0  0.7  1.000000e+00  0.428571

Total error stoch descent:  0.7142779420120247
Total error batch descent:  0.7142779420164836

For the following parameters we do get something:

w1_start = -0.1, w2_start = 0.1, eta = 0.001, decrease_rate = 0.000000001, num_steps = 25000

        K1     K2  Tgt       Res       Err
0   200.0   14.0  0.3  0.298207  0.005976
1     1.0  107.0  0.7  0.603422  0.137969
2   160.0   10.0  0.3  0.334158  0.113860
3    11.0  193.0  0.7  0.671549  0.040644
4   220.0   32.0  0.3  0.294089  0.019705
5    11.0  178.0  0.7  0.658298  0.059574
6   120.0    2.0  0.3  0.368446  0.228154
7    14.0  210.0  0.7  0.683292  0.023869
8   195.0   12.0  0.3  0.301325  0.004417
9    15.0  134.0  0.7  0.613729  0.123244
10  130.0    7.0  0.3  0.362477  0.208256
11    5.0  167.0  0.7  0.654627  0.064819
12  185.0   10.0  0.3  0.309307  0.031025
13   16.0  229.0  0.7  0.697447  0.003647

        K1     K2  Tgt       Res       Err
0   200.0   14.0  0.3  0.000012  0.999961
1     1.0  107.0  0.7  0.997210  0.424586
2   160.0   10.0  0.3  0.000106  0.999646
3    11.0  193.0  0.7  0.999957  0.428510
4   220.0   32.0  0.3  0.000009  0.999968
5    11.0  178.0  0.7  0.999900  0.428429
6   120.0    2.0  0.3  0.000771  0.997429
7    14.0  210.0  0.7  0.999980  0.428543
8   195.0   12.0  0.3  0.000014  0.999953
9    15.0  134.0  0.7  0.998541  0.426487
10  130.0    7.0  0.3  0.000555  0.998150
11    5.0  167.0  0.7  0.999872  0.428389
12  185.0   10.0  0.3  0.000023  0.999922
13   16.0  229.0  0.7  0.999992  0.428560

Total error single:  0.07608269490258893
Total error batch:  0.7134665897677123

By pure chance we found a combination of starting point and learning-rate for which we - by hopping around on the flat cost areas - we accidentally arrived at the slope area of one sample and started a gradient descent. This did however not (yet) happen for the total costs.
We get a minimum around (w1=-0.005,w2=0.005) but with a big spread of 0.0025 for each of the weight values.

Intermediate Conclusion

We looked at a simple perceptron scenario with one computing neuron. Our solitary neuron should learn to distinguish between input data of two distinct and separate data clusters in a 2-dimensional feature space. The feature data change between big and small values for different samples. The neuron used the sigmoid-function as activation and output function. The cost function for all samples shows a minimum at a tiny area in the weight space. We found this minimum with the help of a fine grained and mesh-based analysis of the cost values. However, such an analysis is not applicable to general ML-scenarios.

The problem we face is that due to the saturation properties of the sigmoid function the minimum cannot be detected automatically via gradient descent without knowing already precise details about the solution. Gradient descent does not work - we either get stuck on areas of nearly constant costs or we hop around between different plateaus of the cost function - missing a tiny location in the (w1, w2)-parameter space for a real descent into the existing minimum.

We need to find a way out of this dilemma. In the next article
A single neuron perceptron with sigmoid activation function – II – normalization to overcome saturation
I shall show that normalization opens such a way.

A simple Python program for an ANN to cover the MNIST dataset – XII – accuracy evolution, learning rate, normalization

We continue our article series on building a Python program for a MLP and training it to recognize handwritten digits on images of the MNIST dataset.

A simple Python program for an ANN to cover the MNIST dataset – XI – confusion matrix
A simple Python program for an ANN to cover the MNIST dataset – X – mini-batch-shuffling and some more tests
A simple Python program for an ANN to cover the MNIST dataset – IX – First Tests
A simple Python program for an ANN to cover the MNIST dataset – VIII – coding Error Backward Propagation
A simple Python program for an ANN to cover the MNIST dataset – VII – EBP related topics and obstacles
A simple Python program for an ANN to cover the MNIST dataset – VI – the math behind the „error back-propagation“
A simple Python program for an ANN to cover the MNIST dataset – V – coding the loss function
A simple Python program for an ANN to cover the MNIST dataset – IV – the concept of a cost or loss function
A simple Python program for an ANN to cover the MNIST dataset – III – forward propagation
A simple Python program for an ANN to cover the MNIST dataset – II - initial random weight values
A simple Python program for an ANN to cover the MNIST dataset – I - a starting point

In the last article we used our prediction data to build a so called "confusion matrix" after training. With its help we got an overview about the "false negative" and "false positive" cases, i.e. cases of digit-images for which the algorithm made wrong predictions. We also displayed related critical MNIST images for the digit "4".

In this article we first want to extend the ability of our class "ANN" such that we can measure the level of accuracy (more precisely: the recall) on the full test and the training data sets during training. We shall see that the resulting curves will trigger some new insights. We shall e.g. get an answer to the question at which epoch the accuracy on the test data set does no longer change, but the accuracy on the training set still improves. Meaning: We can find out after which epoch we spend CPU time on overfitting.

In addition we want to investigate the efficiency of our present approach a bit. So far we have used a relatively small learning rate of 0.0001 with a decrease rate of 0.000001. This gave us relatively smooth curves during convergence. However, it took us a lot of epochs and thus computational time to arrive at a cost minimum. The question is:

Is a small learning rate really required? What happens if we use bigger initial learning rates? Can we reduce the number of epochs until learning converges?

Regarding the last point we should not forget that a bigger learning rate may help to move out of local minima on our way to the vicinity of a global minimum. Some of our experiments will indeed indicate that one may get stuck somewhere before moving deep into a minimum valley. However, our forthcoming experiments will also show that we have to take care about the weight initialization. And this in turn will lead us to a major deficit of our present code. Resolving it will help us with bigger learning rates, too.

Class interface changes

We introduce some new parameters, whose usage will become clear later on. They are briefly documented within the code. In addition we do no longer call the method _fit() automatically when we create a Python object instance of the class. This means the you have to call "_fit()" on your own in your Jupyter cells in the future.

To be able to use some additional features later on we first need some more import statements.

New import statements of the class MyANN

import numpy as np
import math 
import sys
import time
import tensorflow
# from sklearn.datasets import fetch_mldata
from sklearn.datasets import fetch_openml
from sklearn.metrics import confusion_matrix
from sklearn.preprocessing import StandardScaler

from sklearn.cluster import KMeans
from sklearn.cluster import MiniBatchKMeans

from keras.datasets import mnist as kmnist
from scipy.special import expit  
from matplotlib import pyplot as plt
from symbol import except_clause
from IPython.core.tests.simpleerr import sysexit
from math import floor


Extended My_ANN interface
We extend our interface substantially - although we shall not use every new parameter, yet. Most of the parameters are documented shortly; but to really understand what they control you have to look into some other changed parts of the class's code, which we present later on. You can, however, safely ignore parameters on "clustering" and "PCA" in this article. We shall yet neither use the option to import MNIST X- and y-data (X_import, y_import) instead of loading them internally.

    def __init__(self, 
                 my_data_set = "mnist",
                 X_import = None, # imported X dataset 
                 y_import = None, # imported y dataset 
                 num_test_records = 10000, # number of test data 
                 # parameter for normalization of input data 
                 b_normalize_X = False, # True: apply StandardScaler on X input data
                 # parameters for clustering of input data 
                 b_perform_clustering = False,  # shall we cluster the X_data before learning? 
                 my_clustering_method = "MiniBatchKMeans", # Choice between 2 methods: MiniBatchKMeans, KMeans  
                 cl_n_clusters = 200,      # number of clusters (often "k" in literature)
                 cl_max_iter = 600,        # number of iterations for centroid movement
                 cl_n_init = 100,          # number of different initial centroid positions tried 
                 cl_n_jobs = 4,            # number of CPU cores (jobs to start for investigating n_init variations
                 cl_batch_size = 500,      # batch size, only used for MiniBatchKMeans
                 #parameters for PCA of input data 
                 b_perform_pca = False,
                 num_pca_categories = 155, 
                 # parameters for MLP structure
                 n_hidden_layers = 1, 
                 ay_nodes_layers = [0, 100, 0], # array which should have as much elements as n_hidden + 2
                 n_nodes_layer_out = 10,  # expected number of nodes in output layer 
                 my_activation_function = "sigmoid", 
                 my_out_function        = "sigmoid",   
                 my_loss_function       = "LogLoss",   
                 n_size_mini_batch = 50,  # number of data elements in a mini-batch 
                 n_epochs      = 1,
                 n_max_batches = -1,  # number of mini-batches to use during epochs - > 0 only for testing 
                                      # a negative value uses all mini-batches 
                 lambda2_reg = 0.1,     # factor for quadratic regularization term 
                 lambda1_reg = 0.0,     # factor for linear regularization term 
                 vect_mode = 'cols', 
                 init_weight_meth_L0 = "sqrt_nodes",  # method to init weights => "sqrt_nodes", "const"
                 init_weight_meth_Ln = "sqrt_nodes",  # sqrt_nodes", "const"
                 init_weight_intervals = [(-0.5, 0.5), (-0.5, 0.5), (-0.5, 0.5)],   # size must fit number of hidden layers
                 init_weight_fact = 2.0,                # extends the interval 
                 learn_rate = 0.001,          # the learning rate (often called epsilon in textbooks) 
                 decrease_const = 0.00001,    # a factor for decreasing the learning rate with epochs
                 learn_rate_limit = 2.0e-05,  # a lower limit for the learn rate 
                 adapt_with_acc = False,      # adapt learning rate with additional factor depending on rate of acc change
                 reduction_fact = 0.001,      # small reduction factor - should be around 0.001 because of an exponential reduction
                 mom_rate   = 0.0005,         # a factor for momentum learning
                 b_shuffle_batches = True,    # True: we mix the data for mini-batches in the X-train set at the start of each epoch
                 b_predictions_train = False, # True: At the end of periodic epochs the code performs predictions on the train data set
                 b_predictions_test  = False, # True: At the end of periodic epochs the code performs predictions on the test data set
                 prediction_test_period  = 1, # Period of epochs for which we perform predictions
                 prediction_train_period = 1, # Period of epochs for which we perform predictions
                 print_period = 20,         # number of epochs for which to print the costs and the averaged error
                 figs_x1=12.0, figs_x2=8.0, 
                 legend_loc='upper right',
                 b_print_test_data = True
        Initialization of MyANN
            data_set: type of dataset; so far only the "mnist", "mnist_784" datsets are known 
                      We use this information to prepare the input data and learn about the feature dimension. 
                      This info is used in preparing the size of the input layer.     
            X_import: external X dataset to import  
            y_import: external y dataset to import - must fit in dimension to X_import 
            num_test_records: number of test data
            b_normalize_X: True => Invoke the StandardScaler of Scikit-Learn 
                                   to center and normalize the input data X
            Preprocessing of input data treatment before learning 
            b_perform_clustering   # True => Cluster the X_data before learning? 
            my_clustering_method   # string: 2 methods: MiniBatchKMeans, KMeans 
            cl_n_clusters = 200       # number of clusters (often "k" in literature)
            cl_max_iter = 600      # number of iterations for centroid movement
            cl_n_init = 100        # number of different initial centroid positions tried 
            cl_n_jobs = 4,         # number of CPU cores => jobs - only used for "KMeans"
            cl_batch_size = 500    # batch size used for "MiniBatchKMeans"
            b_perform_pca: True => perform a pca analysis 
            num_pca_categories: 155 - choose a reasonable number 
            n_hidden_layers = number of hidden layers => between input layer 0 and output layer n 
            ay_nodes_layers = [0, 100, 0 ] : We set the number of nodes in input layer_0 and the output_layer to zero 
                              Will be set to real number afterwards by infos from the input dataset. 
                              All other numbers are used for the node numbers of the hidden layers.
            n_nodes_out_layer = expected number of nodes in the output layer (is checked); 
                                this number corresponds to the number of categories NC = number of labels to be distinguished
            my_activation_function : name of the activation function to use 
            my_out_function : name of the "activation" function of the last layer which produces the output values 
            my_loss_function : name of the "cost" or "loss" function used for optimization 
            n_size_mini_batch : Number of elements/samples in a mini-batch of training data 
                                The number of mini-batches will be calculated from this
            n_epochs : number of epochs to calculate during training
            n_max_batches : > 0: maximum of mini-batches to use during training 
                            < 0: use all mini-batches  
            lambda_reg2:    The factor for the quadartic regularization term 
            lambda_reg1:    The factor for the linear regularization term 
            vect_mode: Are 1-dim data arrays (vctors) ordered by columns or rows ?
            init_weight_meth_L0: Method to calculate the initial weights at layer L0: "sqrt_nodes" => sqrt(number of nodes) /  "const" => interval borders 
            init_weight_meth_Ln: Method to calculate the initial weights at hidden layers 
            init_weight_intervals: list of tuples with interval limits [(-0.5, 0.5), (-0.5, 0.5), (-0.5, 0.5)],   
                                   size must fit number of hidden layers
            init_weight_fact:  interval limits get scald by this factor, e.g. 2* (0,5, 0.5)

            learn rate :     Learning rate - definies by how much we correct weights in the indicated direction of the gradient on the cost hyperplane.
            decrease_const:  Controls a systematic decrease of the learning rate with epoch number 
            learn_rate_limit = 2.0e-05,  # a lowee limit for the learning rate 
            adapt_with_acc:  True => adapt learning rate with additional factor depending on rate of acc change
            reduction_fact:  around 0.001  => almost exponential reduction during the first 500 epochs   
            mom_const:       Momentum rate. Controls a mixture of the last with the present weight corrections (momentum learning)
            b_shuffle_batches: True => vary composition of mini-batches with each epoch
            # The next two parameters enable the measurement of accuracy and total cost function 
            # by making predictions on the train and test datasets 
            b_predictions_train: True => perform a prediction run on the full training data set => get accuracy 
            b_predictions_test:  True => perform a prediction run on the full test data set => get accuracy 
            prediction_test_period: period of epochs for which to perform predictions
            prediction_train_period: period of epochs for which to perform predictions
            print_period:    number of periods between printing out some intermediate data 
                             on costs and the averaged error of the last mini-batch   
            figs_x1=12.0, figs_x2=8.0 : Standard sizing of plots , 
            legend_loc='upper right': Position of legends in the plots
            b_print_test_data: Boolean variable to control the print out of some tests data 
        # Array (Python list) of known input data sets 
        self._input_data_sets = ["mnist", "mnist_784", "mnist_keras", "imported"]  
        self._my_data_set = my_data_set
        # X_import, y_import, X, y, X_train, y_train, X_test, y_test  
            # will be set by method handle_input_data() 
            # X: Input array (2D) - at present status of MNIST image data, only.    
            # y: result (=classification data) [digits represent categories in the case of Mnist]
        self._X_import = X_import 
        self._y_import = y_import 
        # number of test data 
        self._num_test_records = num_test_records
        self._X       = None 
        self._y       = None 
        self._X_train = None 
        self._y_train = None 
        self._X_test  = None   
        self._y_test  = None
        # perform a normalization of the input data
        self._b_normalize_X = b_normalize_X
        # relevant dimensions 
        # from input data information;  will be set in handle_input_data()
        self._dim_X        = 0  # total num of records in the X,y input sets
        self._dim_sets     = 0  # num of records in the TRAINING sets X_train, y_train
        self._dim_features = 0  
        self._n_labels     = 0   # number of unique labels - will be extracted from y-data 
        # Img sizes 
        self._dim_img      = 0 # should be sqrt(dim_features) - we assume square like images  
        self._img_h        = 0 
        self._img_w        = 0 
        # Preprocessing of input data 
        # ---------------------------
        self._b_perform_clustering = b_perform_clustering
        self._my_clustering_method = my_clustering_method # for the related dictionary see below 
        self._kmeans        = None   # pointer to object used for clustering  
        self._cl_n_clusters = cl_n_clusters       # number of clusters (often "k" in literature)
        self._cl_max_iter   = cl_max_iter      # number of iterations for centroid movement
        self._cl_n_init     = cl_n_init        # number of different initial centroid positions tried 
        self._cl_batch_size = cl_batch_size    # batch size used for MiniBatchKMeans
        self._cl_n_jobs     = cl_n_jobs        # number of parallel jobs (on CPU-cores) - only used for KMeans
        # Layers
        # ------
        # number of hidden layers 
        self._n_hidden_layers = n_hidden_layers
        # Number of total layers 
        self._n_total_layers = 2 + self._n_hidden_layers  
        # Nodes for hidden layers 
        self._ay_nodes_layers = np.array(ay_nodes_layers)
        # Number of nodes in output layer - will be checked against information from target arrays
        self._n_nodes_layer_out = n_nodes_layer_out
        # Weights 
        # --------
        # empty List for all weight-matrices for all layer-connections
        # Numbering : 
        # w[0] contains the weight matrix which connects layer 0 (input layer ) to hidden layer 1 
        # w[1] contains the weight matrix which connects layer 1 (input layer ) to (hidden?) layer 2 
        self._li_w = []
        # Arrays for encoded output labels - will be set in _encode_all_mnist_labels()
        # -------------------------------
        self._ay_onehot = None
        self._ay_oneval = None
        # Known Randomizer methods ( 0: np.random.randint, 1: np.random.uniform )  
        # ------------------
        self.__ay_known_randomizers = [0, 1]

        # Types of activation functions and output functions 
        # ------------------
        self.__ay_activation_functions = ["sigmoid"] # later also relu 
        self.__ay_output_functions     = ["sigmoid"] # later also softmax 
        # Types of cost functions 
        # ------------------
        self.__ay_loss_functions = ["LogLoss", "MSE" ] # later also other types of cost/loss functions  

        # dictionaries for indirect function calls 
        self.__d_activation_funcs = {
            'sigmoid': self._sigmoid, 
            'relu':    self._relu
        self.__d_output_funcs = { 
            'sigmoid': self._sigmoid, 
            'softmax': self._softmax
        self.__d_loss_funcs = { 
            'LogLoss': self._loss_LogLoss, 
            'MSE': self._loss_MSE
        # Derivative functions 
        self.__d_D_activation_funcs = {
            'sigmoid': self._D_sigmoid, 
            'relu':    self._D_relu
        self.__d_D_output_funcs = { 
            'sigmoid': self._D_sigmoid, 
            'softmax': self._D_softmax
        self.__d_D_loss_funcs = { 
            'LogLoss': self._D_loss_LogLoss, 
            'MSE': self._D_loss_MSE
        self.__d_clustering_functions = {
            'MiniBatchKMeans': self._Mini_Batch_KMeans, 
            'KMeans': self._KMeans
        # The following variables will later be set by _check_and set_activation_and_out_functions()            
        self._my_act_func  = my_activation_function
        self._my_out_func  = my_out_function
        self._my_loss_func = my_loss_function
        self._act_func      = None    
        self._out_func      = None    
        self._loss_func     = None    
        self._cluster_func  = None    
        # number of data samples in a mini-batch 
        self._n_size_mini_batch = n_size_mini_batch
        self._n_mini_batches = None  # will be determined by _get_number_of_mini_batches()

        # maximum number of epochs - we set this number to an assumed maximum 
        # - as we shall build a backup and reload functionality for training, this should not be a major problem 
        self._n_epochs = n_epochs
        # maximum number of batches to handle ( if < 0 => all!) 
        self._n_max_batches = n_max_batches
        # actual number of batches 
        self._n_batches = None

        # regularization parameters
        self._lambda2_reg = lambda2_reg
        self._lambda1_reg = lambda1_reg
        # parameters to control the initialization of the weights (see _create_WM_Input(), create_WM_Hidden())
        self._init_weight_meth_L0 = init_weight_meth_L0
        self._init_weight_meth_Ln = init_weight_meth_Ln
        self._init_weight_intervals = init_weight_intervals # list of lists with interval borders
        self._init_weight_fact = init_weight_fact           # extends weight intervals 
        # parameters for adaption of the learning rate
        self._learn_rate = learn_rate
        self._decrease_const = decrease_const
        self._learn_rate_limit = learn_rate_limit
        self._adapt_with_acc  = adapt_with_acc
        self._reduction_fact  = reduction_fact
        # parameters for momentum learning 
        self._mom_rate   = mom_rate
        self._li_mom = [None] *  self._n_total_layers
        # shuffle data in X_train? 
        self._b_shuffle_batches = b_shuffle_batches
        # perform predictions on train and test data set and related analysis 
        self._b_predictions_train = b_predictions_train
        self._b_predictions_test  = b_predictions_test
        self._prediction_test_period  = prediction_test_period
        self._prediction_train_period = prediction_train_period
        # epoch period for printing 
        self._print_period = print_period
        # book-keeping for epochs and mini-batches 
        # -------------------------------
        # range for epochs - will be set by _prepare-epochs_and_batches() 
        self._rg_idx_epochs = None
        # range for mini-batches 
        self._rg_idx_batches = None
        # dimension of the numpy arrays for book-keeping - will be set in _prepare_epochs_and_batches() 
        self._shape_epochs_batches = None    # (n_epochs, n_batches, 1) 

        # training evolution:
        # +++++++++++++++++++ 
        # List for error values at outermost layer for minibatches and epochs during training
        # we use a numpy array here because we can redimension it
        self._ay_theta = None
        # List for cost values of mini-batches during training - The list will later be split into sections for epochs 
        self._ay_costs = None
        # List for test accuracy values and error values at epoch periods 
        self._ay_period_test_epoch = None # x-axis for plots of the following 2 quantities 
        self._ay_acc_test_epoch = None  
        self._ay_err_test_epoch = None  
        # List for train accuracy values and error values at epoch periods  
        self._ay_period_train_epoch = None # x-axis for plots of the following 2 quantities 
        self._ay_acc_train_epoch = None  
        self._ay_err_train_epoch = None  
        # Data elements for back propagation
        # ----------------------------------
        # 2-dim array of partial derivatives of the elements of an additive cost function 
        # The derivative is taken with respect to the output results a_j = ay_ANN_out[j]
        # The array dimensions account for nodes and sampls of a mini_batch. The array will be set in function 
        # self._initiate_bw_propagation()
        self._ay_delta_out_batch = None

        # parameter to allow printing of some test data 
        self._b_print_test_data = b_print_test_data

        # Plot handling 
        # --------------
        # Alternatives to resize plots 
        # 1: just resize figure  2: resize plus create subplots() [figure + axes] 
        self._plot_resize_alternative = 1 
        # Plot-sizing
        self._figs_x1 = figs_x1
        self._figs_x2 = figs_x2
        self._fig = None
        self._ax  = None 
        # alternative 2 does resizing and (!) subplots() 
        # ***********
        # operations 
        # ***********
        # check and handle input data 
        # set the ANN structure 
        # Prepare epoch and batch-handling - sets ranges, limits num of mini-batches and initializes book-keeping arrays
        self._rg_idx_epochs, self._rg_idx_batches = self._prepare_epochs_and_batches()


Code modifications to create precise accuracy information on the full test and training sets during training

You certainly noticed the following set of control parameters in the class's new interface:

  • b_predictions_train = False, # True: At the end of periodic epochs the code performs predictions on the train data set
  • b_predictions_test = False, # True: At the end of periodic epochs the code performs predictions on the test data set
  • prediction_test_period = 1, # Period of epochs for which we perform predictions
  • prediction_train_period = 1, # Period of epochs for which we perform predictions

These parameters control whether we perform predictions during training for the full test dataset and/or the full training dataset - and if so, at which epoch period. Actually, during all of the following experiments we shall evaluate the accuracy data after each single period.

We need an array to gather accuracy information. We therefore modify the method "_prepare_epochs_and_batches()", where we fill some additional Numpy arrays with initialization values. Thus we avoid a costly "append()" later on; we just overwrite the array entries successively. This overwriting happens in our method _fit(); see below.

Changes to function "_prepare_epochs_and_batches()":

    ''' -- Main Method to prepare epochs, batches and book-keeping arrays -- '''
    def _prepare_epochs_and_batches(self, b_print = True):
        # range of epochs
        ay_idx_epochs  = range(0, self._n_epochs)
        # set number of mini-batches and array with indices of input data sets belonging to a batch 
        # limit the number of mini-batches
        self._n_batches = min(self._n_max_batches, self._n_mini_batches)
        ay_idx_batches = range(0, self._n_batches)
        if (b_print):
            if  self._n_batches < self._n_mini_batches :
                print("\nWARNING: The number of batches has been limited from " + 
                      str(self._n_mini_batches) + " to " + str(self._n_max_batches) )
        # Set the book-keeping arrays 
        self._shape_epochs_batches = (self._n_epochs, self._n_batches)
        self._ay_theta = -1 * np.ones(self._shape_epochs_batches) # float64 numbers as default
        self._ay_costs = -1 * np.ones(self._shape_epochs_batches) # float64 numbers as default
        shape_test_epochs  = ( floor(self._n_epochs / self._prediction_test_period), )
        shape_train_epochs = ( floor(self._n_epochs / self._prediction_train_period), )
        self._ay_period_test_epoch  = -1 * np.ones(shape_test_epochs) # float64 numbers as default
        self._ay_acc_test_epoch     = -1 * np.ones(shape_test_epochs) # float64 numbers as default
        self._ay_err_test_epoch     = -1 * np.ones(shape_test_epochs) # float64 numbers as default
        self._ay_period_train_epoch = -1 * np.ones(shape_train_epochs) # float64 numbers as default
        self._ay_acc_train_epoch    = -1 * np.ones(shape_train_epochs) # float64 numbers as default
        self._ay_err_train_epoch    = -1 * np.ones(shape_train_epochs) # float64 numbers as default
        return ay_idx_epochs, ay_idx_batches 


We then create two new methods to calculate accuracy values by predicting results on all records of both the training dataset and the test dataset. The attentive reader certainly recognizes the methods' structure from a previous article where we used similar code in a Jupyter cell:

New functions "_predict_all_test_data()" and "_predict_all_train_data()":

    ''' Method to predict values for the full set of test data '''
    def _predict_all_test_data(self): 
        size_set = self._X_test.shape[0]
        li_Z_in_layer_test  = [None] * self._n_total_layers
        li_Z_in_layer_test[0] = self._X_test
        # Transpose input data matrix  
        ay_Z_in_0T       = li_Z_in_layer_test[0].T
        li_Z_in_layer_test[0] = ay_Z_in_0T
        li_A_out_layer_test  = [None] * self._n_total_layers
        # prediction by forward propagation of the whole test set 
        self._fw_propagation(li_Z_in = li_Z_in_layer_test, li_A_out = li_A_out_layer_test, b_print = False) 
        ay_predictions_test = np.argmax(li_A_out_layer_test[self._n_total_layers-1], axis=0)
        # accuracy 
        ay_errors_test = self._y_test - ay_predictions_test 
        acc_test = (np.sum(ay_errors_test == 0)) / size_set
        # print ("total acc for test data = ", acc)
        # return acc, ay_predictions_test
        return acc_test
    ''' Method to predict values for the full set of training data '''
    def _predict_all_train_data(self): 
        size_set = self._X_train.shape[0]
        li_Z_in_layer_train  = [None] * self._n_total_layers
        li_Z_in_layer_train[0] = self._X_train
        # Transpose 
        ay_Z_in_0T       = li_Z_in_layer_train[0].T
        li_Z_in_layer_train[0] = ay_Z_in_0T
        li_A_out_layer_train  = [None] * self._n_total_layers
        self._fw_propagation(li_Z_in = li_Z_in_layer_train, li_A_out = li_A_out_layer_train, b_print = False) 
        ay_predictions_train = np.argmax(li_A_out_layer_train[self._n_total_layers-1], axis=0)
        ay_errors_train = self._y_train - ay_predictions_train 
        acc_train = (np.sum(ay_errors_train == 0)) / size_set
        #print ("total acc for train data = ", acc)    
        return acc_train


Eventually, we modify our method "_fit()" with a series of statements on the level of the epoch loop. You may ignore most of the statements for learning rate adaption; we only use the "simple" adaption methods. The really important changes are those regarding predictions.

Modifications of function "_fit()":

    ''' -- Method to perform training in epochs for defined mini-batches -- '''
    def _fit(self, b_print = False, b_measure_epoch_time = True, b_measure_batch_time = False):
            b_print:                 Do we print intermediate results of the training at all? 
            b_print_period:          For which period of epochs do we print? 
            b_measure_epoch_time:    Measure CPU-Time for an epoch
            b_measure_batch_time:    Measure CPU-Time for a batch
        rg_idx_epochs  = self._rg_idx_epochs 
        rg_idx_batches = self._rg_idx_batches
        if (b_print):    
            print("\nnumber of epochs = " + str(len(rg_idx_epochs)))
            print("max number of batches = " + str(len(rg_idx_batches)))
        # Some intial parameters 
        acc_old = 0.0000001
        acc_test = 0.001
        orig_rate = self._learn_rate
        adapt_fact = 1.0
        n_predict_test  = 0
        n_predict_train = 0
        # loop over epochs
        # ****************
        start_train = time.perf_counter()
        for idxe in rg_idx_epochs:
            if b_print and (idxe % self._print_period == 0):
                if b_measure_epoch_time:
                    start_0_e = time.perf_counter()
                print("\n ---------")
                print("Starting epoch " + str(idxe+1))
            # simple adaption of the learning rate 
            # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            orig_rate /= (1.0 + self._decrease_const * idxe)
            self._learn_rate /= (1.0 + self._decrease_const * idxe)
            if self._learn_rate < self._learn_rate_limit:
                self._learn_rate = self._learn_rate_limit
            # adapt wit acc. - not working well, yet 
            acc_change_rate = math.fabs((acc_test - acc_old) / acc_old)
            if b_print and (idxe % self._print_period == 0):
                print("acc_chg_rate = ", acc_change_rate)
            ratio = self._learn_rate / orig_rate 
            if ratio > 0.33 and acc_change_rate < 1/3 and self._adapt_with_acc:
                if acc_change_rate < 0.001:
                    acc_change_rate = 0.001
                #adapt_fact = 2.0 * acc_change_rate / (1.0 - acc_change_rate)
                adapt_fact = 1.0 - 0.001 * (1.0 - acc_change_rate / (1.0 - acc_change_rate))
                if b_print and (idxe % self._print_period == 0):
                    print("adapt_fact = ", adapt_fact)
                self._learn_rate *= adapt_fact
            acc_old = acc_test # for adaption of learning rate 
            # shuffle indices for a variation of the mini-batches with each epoch
            # ******************************************************************
            if self._b_shuffle_batches:
                shuffled_index = np.random.permutation(self._dim_sets)
                self._X_train, self._y_train, self._ay_onehot = self._X_train[shuffled_index], self._y_train[shuffled_index], self._ay_onehot[:, shuffled_index]
            # loop over mini-batches
            # **********************
            for idxb in rg_idx_batches:
                if b_measure_batch_time: 
                    start_0_b = time.perf_counter()
                # deal with a mini-batch
                self._handle_mini_batch(num_batch = idxb, num_epoch=idxe, b_print_y_vals = False, b_print = False)
                if b_measure_batch_time: 
                    end_0_b = time.perf_counter()
                    print('Time_CPU for batch ' + str(idxb+1), end_0_b - start_0_b) 
            # predictions
            # ***********
            # Control and perform predictions on the full test data set 
            if self._b_predictions_test and idxe % self._prediction_test_period == 0:
                self._ay_period_test_epoch[n_predict_test] = idxe
                acc_test = self._predict_all_test_data()
                self._ay_acc_test_epoch[n_predict_test] = acc_test
                n_predict_test += 1
            # Control and perform predictions on the full training training data set 
            if self._b_predictions_train and idxe % self._prediction_train_period == 0:
                self._ay_period_train_epoch[n_predict_train] = idxe
                acc_train = self._predict_all_train_data()
                self._ay_acc_train_epoch[n_predict_train] = acc_train
                n_predict_train += 1
            # printing some evolution and epoch information
            if b_print and (idxe % self._print_period == 0):
                if b_measure_epoch_time:
                    end_0_e = time.perf_counter()
                    print('Time_CPU for epoch' + str(idxe+1), end_0_e - start_0_e) 
                print("learning rate = ", self._learn_rate)
                print("orig learn rate = ", orig_rate)
                print("\ntotal costs of last mini_batch = ", self._ay_costs[idxe, idxb])
                print("avg total error of last mini_batch = ", self._ay_theta[idxe, idxb])
                # print presently reached accuracy values on the test and training sets 
                print("presently reached train accuracy =<div style="width: 95%; overflow: auto; height: 400px;">
<pre style="width: 1000px;">
 ", acc_train)
                print("presently reached test accuracy = ", acc_test)
        # print out required secs for training
        # **************************************
        end_train = time.perf_counter()
        print('\n\n ------') 
        print('Total training Time_CPU: ', end_train - start_train) 
        print("\nStopping program regularily")

        return None

The method we apply in the above code to reduce the learning rate with every epoch is by the way called "power scheduling". The book of Aurelien Geron ["Hands on Machine learning ....", 2019, 2nd edition, O'Reilly], quoted already in previous articles, lists a bunch of other methods, e.g. "exponential scheduling", where we multiply the learning rate with a constant factor < 1 at every epoch.

A further change of code happens in the functions _create_WM_input() and _ceate_WM_hidden" to initiate weight values.

Modifications of functions _create_WM_input() and _create_WM_hidden":
Addendum 25.03.2020: Changed _create_WM_hidden() because of errors in the code

    '''-- Method to create the weight matrix between L0/L1 --'''
    def _create_WM_Input(self):
        Method to create the input layer 
        The dimension will be taken from the structure of the input data 
        We need to fill self._w[0] with a matrix for conections of all nodes in L0 with all nodes in L1
        We fill the matrix with random numbers between [-1, 1] 
        # the num_nodes of layer 0 should already include the bias node 
        num_nodes_layer_0 = self._ay_nodes_layers[0]
        num_nodes_with_bias_layer_0 = num_nodes_layer_0 + 1 
        num_nodes_layer_1 = self._ay_nodes_layers[1] 
        # Set interval borders for randomizer 
        if self._init_weight_meth_L0 == "sqrt_nodes":  # sqrtr(nodes) - rule of Prof. J. Frochte
            rand_high = self._init_weight_fact / math.sqrt(float(num_nodes_layer_0))
            rand_low = - rand_high 
            rand_low  = self._init_weight_intervals[0][0]
            rand_high = self._init_weight_intervals[0][1]
        print("\nL0: weight range [" + str(rand_low) + ", " + str(rand_high) + "]" )
        # fill the weight matrix at layer L0 with random values 
        randomizer = 1 # method np.random.uniform   
        rand_size = num_nodes_layer_1 * (num_nodes_with_bias_layer_0) 
        w0 = self._create_vector_with_random_values(rand_low, rand_high, rand_size, randomizer)
        w0 = w0.reshape(num_nodes_layer_1, num_nodes_with_bias_layer_0)
        # put the weight matrix into array of matrices 
        print("\nShape of weight matrix between layers 0 and 1 " + str(self._li_w[0].shape))
    '''-- Method to create the weight-matrices for hidden layers--''' 
    def _create_WM_Hidden(self):
        Method to create the weights of the hidden layers, i.e. between [L1, L2] and so on ... [L_n, L_out] 
        We fill the matrix with random numbers between [-1, 1] 
        # The "+1" is required due to range properties ! 
        rg_hidden_layers = range(1, self._n_hidden_layers + 1, 1)
        # Check parameter input fro weight intervals 
        if self._init_weight_meth_Ln == "const":
            if len(self._init_weight_intervals) != (self._n_hidden_layers + 1):
                print("\nError: we shall initialize weights with values from intervals, but wrong number of intervals provided!") 
        for i in rg_hidden_layers: 
            print ("\nCreating weight matrix for layer " + str(i) + " to layer " + str(i+1) )
            num_nodes_layer = self._ay_nodes_layers[i] 
            num_nodes_with_bias_layer = num_nodes_layer + 1 

            # Set interval borders for randomizer 
            if self._init_weight_meth_Ln == "sqrt_nodes":  # sqrtr(nodes) - rule of Prof. J. Frochte
                rand_high = self._init_weight_fact / math.sqrt(float(num_nodes_layer))
                rand_low = - rand_high 
                rand_low  = self._init_weight_intervals[i][0]
                rand_high = self._init_weight_intervals[i][1]
            print("L" + str(i) + ": weight range [" + str(rand_low) + ", " + str(rand_high) + "]" )
            # the number of the next layer is taken without the bias node!
            num_nodes_layer_next = self._ay_nodes_layers[i+1]
            # ill the weight matrices at the hidden layer with random values  
            rand_size = num_nodes_layer_next * num_nodes_with_bias_layer   
            randomizer = 1 # np.random.uniform
            w_i_next = self._create_vector_with_random_values(rand_low, rand_high, rand_size, randomizer)   
            w_i_next = w_i_next.reshape(num_nodes_layer_next, num_nodes_with_bias_layer)
            # put the weight matrix into our array of matrices 
            print("Shape of weight matrix between layers " + str(i) + " and " + str(i+1) + " = " + str(self._li_w[i].shape))

As you see, we distinguish between different cases depending on the parameters "init_weight_meth_L0" and "init_weight_meth_Ln". There, obviously, happens a choice regarding the borders of the intervals from which we randomly pick our initial weight values. In case of the method "sqrt_nodes" the interval borders are determined by the number of nodes of the neighboring layer. Otherwise we can read the interval borders from the parameter list "init_weight_intervals". You will better understand these options later on.

Plotting accuracys

We use a very simple code in a Jupyter cell to get a plot for the accuracy values on the training and the test datasets. The orange line will show the accuracy reached at each epoch for the training dataset when we apply the weights evaluated given at the epoch. The blue line shows the accuracy reached for the test dataset. In the text below we shall use the following abbreviations:

acc_train = accuracy reached for the X_train dataset of MNIST
acc_test   = accuracy reached for the X_test dataset of MNIST

Code for plotting

plt.ylim(y_min, y_max)




Experiment 1: Accuracy plot for a Reference Run

The next test run will be used as a reference run for comparisons later on. It shows where we stand right now.

Test 1:
Parameters: learn_rate = 0.0001, decrease_rate = 0.000001, mom_rate = 0.00005, n_size_mini_batch = 500, Lambda2 = 0.2, n_epochs = 1800, initial weights for all layers in [-0.5, +0.5]:
Results: acc_train: 0.996 , acc_test: 0.961, convergence after ca. 1150 epochs

Note that we use a very small learn_rate, but an even smaller decrease rate. The evolution of the accuracy values looks like follows:

The x-axis measures the number of epochs during training. The y-axis the degree of accuracy - given as a fraction; multiply by 100 to get percentage values. By the way, applying the reached weight set on the full training and test datasets in each epoch cost us at least 20% rise in CPU time (45 minutes).

What does our new way of representing the "learning" of our MLP by the evolution of the accuracy levels tell us?

Noise: There is substantial noise visible along the lines. If you go back to previous articles you may detect the same type of noise in the plots of the evolution of the cost function. Obviously, our mini-batches and the constant change of their composition with each epoch lead to wiggles around the visible trends.

Tendencies: Note that there is a tendency of linear rise of the accuracy acc_train between periods 350 and 900. And, actually, the accuracy even decreases a bit around epoch 1550. This is a warning that the very last epoch of a run may not reveal the optimal solution.

Overfitting and a typical related splitting of the evolution of the two accuracys: One clearly sees that after a certain epoch (here: around epoch 300) the accuracy on the training dataset deviates systematically from the accuracy on the test dataset. In the end the gap is bigger than 3.5 percent. And in our case the accuracy on the test dataset reaches its final level of 0.96 significantly earlier - at around epoch 750 - and remains there, while the accuracy on the training set still rises up to epoch 1000.

However, I would like to add a warning:
Warning: Later on we shall see that there are cases for which both curves turn into a convergence at almost the same epoch. So, yes, there almost always occurs some overfitting during training of a MLP. However, we cannot set up a rule which says that convergence of the accuracy on the test dataset always occurs much earlier than for the training set. You always have to watch the evolution of both during your training experiments!

Experiment 2: Increasing the learning rate - better efficiency?

Let us now be brave and increase the learning rate by a factor of 10:

Test 2:
Parameters: learn_rate = 0.001, decrease_rate = 0.000001, mom_rate = 0.00005, n_size_mini_batch = 500, Lambda2 = 0.2, n_epochs = 1800, initial weights for all layers in [-0.5, +0.5]:
Results: acc_train: 0.971 , acc_test: 0.959, no convergence after ca. 1800 epochs, yet

Ooops! Our algorithm ran into real difficulties! We seem to hop in and out of a minimum area until epoch 400 and despite a following systematic linear improvementthere is no sign of a real convergence - yet!

The learning rate seems to big to lead to a consistent quick path into a minimum of all mini-batches! This may have to do with the size of the mini-batches, too - see below. The increase of the learning rate did not do us any good.

Experiment 3: Increased learning rate - but a higher decrease rate, too

As the larger learning rate seems to be a problem after period 50, we may think of a faster reduction of the learning rate.

Test 3:
Parameters: learn_rate = 0.001, decrease_rate = 0.00001, mom_rate = 0.00005, n_size_mini_batch = 500, Lambda2 = 0.2, n_epochs = 2000, initial weights for all layers in [-0.5, +0.5]:
Results: acc_train: 0.9909, acc_test: 0.9646, convergence after ca. 800 epochs

The evolution looks strange, too, but better than experiment 2! We see a real convergence again after some rather linear development! As a lesson learned I would say: Yes we can work with an initially bigger learning rate - but we need a stronger decrease of it, too, to really run into a global minimum eventually.

Experiment 4: Increased learning rate, higher decrease rate and smaller initial weights

Maybe the weight initialization has some impact? According to a rule published by Prof. Frochte in his book "Maschinelles Lernen" [2019, 2. Auflage, Carl Hanser Verlag] I limited the initial random weight values to a range between [-1.0/sqrt(784), +1.0/sqrt(784)] - instead of [-0.5, 0.5] for all layers.

Test 4:
Parameters: learn_rate = 0.001, decrease_rate = 0.00001, mom_rate = 0.00005, n_size_mini_batch = 500, Lambda2 = 0.2, n_epochs = 2000, initial weights for all layers within [-0.36 0.36]:
Results: acc_train: 0.987 , acc_test: 0.967, convergence after ca. 900 epochs

The interesting part in this case happens below and at epoch 200: There we see a clear indication that something has "trapped" us for a while before we could descend into some minimum with the typical split of the accuracy for the training set and the accuracy for the test set. Remember that smaller initial weights also mean an initially smaller contribution of the regularization term to the cost function!

Did we run into a side minimum? Or walk around the edge between two minima? Too complex to analyze in a space with 7000 dimensions!, But, I think this gives you some impression of what might happen on the surface of a varying, bumpy hyperplane ...

Experiment 5: Reduced weights only between the L0/L1 layers

The next test shows the same as the last experiment, but with the initial weights only reduced for the L0/L1 matrix.

Test 5:
Parameters: learn_rate = 0.001, decrease_rate = 0.00001, mom_rate = 0.00005, n_size_mini_batch = 500, Lambda2 = 0.2, n_epochs = 2000, initial weights for the matrix of the first layers L0/L1 within [-0.36 0.36], otherwise in [-0.5, 0.5]:
Results: acc_train: 0.988 , acc_test: 0.967, convergence after ca. 900 epochs

All in all - the trouble the code has with finding a way into a global minimum got even more pronounced around epoch 100. It seems as if the algorithm has to find a new path direction there. The lesson learned is: Weight initialization is important!

Experiment 6: Enlarged mini-batch-size - do we get a smoother evolution?

Now we keep the parameters of experiment 5, but we enlarge the batch size - could be helpful to align and deepen the different minima for the different mini-batches - and thus maybe lead to a smoothing. We choose a batch-size of 1200 (i.e. 50 batches instead of 120 in the training set):

Test 6:
Parameters: learn_rate = 0.001, decrease_rate = 0.00001, mom_rate = 0.00005, n_size_mini_batch = 1200, Lambda2 = 0.2, n_epochs = 2000, initial weights for the matrix first layers L0/L1 [-0.36 0.36], otherwise in [-0.5, 0.5]:
Results: acc_train: 0.959 , acc_test: 0.946, not yet converged after ca. 750 epochs

Would you say that enlarging the mini-batch-siz really helped us? I would say: Bigger batch-sizes do not help an algorithm on the verge of trouble! Nope, the structural problems do not disappear.

Experiment 7: Reduced learn-rate, increased decrease-rate

Let us face it: For our present state of the MLP-algorithm and the MNIST X-data values directly fed into the input nodes the learn-rate must be chosen significantly smaller to overcome the initial problems of restructuring the weight matrices. So, we give up our trials to work with larger learn-rates - but only for a moment. Let us for confirmation now reduce the initial learning-rate again, but increase the "decrease rate". At the same time we also decrease the values of the weights.

Test 7:
Parameters: learn_rate = 0.0002, decrease_rate = 0.00001, mom_rate = 0.00005, n_size_mini_batch = 500, Lambda2 = 0.2, n_epochs = 1200,initial weights for the matrix first layers L0/L1 [-0.36 0.36] and for the next layers L1/L2 + L2/L3 in [0.08, 0.08]:
Results: acc_train: 0.9943 , acc_test: 0.9655, convergence after ca. 600 epochs

OK, nice again! There is some trouble, but we only need 600 epochs to come to a pretty good accuracy value for the test data set!

Intermediate conclusion

Quite often you may read in literature that a bigger learning rate (often abbreviated with a greek eta) can save computational time in terms of required epochs - as long as convergence is guaranteed. Hmmm - we saw in the tests above that this may not be true under certain conditions. It is better to say that - depending on the data, the depth of the network and the size of the mini-batches - you may have to control a delicate balance of an initial rate and a rate decline to find an optimum in terms of epochs.

Initial learning rates which are chosen too big together with a too small decrease rate may lead into trouble: the algorithm may get trapped after a few hundred epochs or even stay a long time in some side minimum until it finds a deepening which it really can descent into.

With a smaller learning rate, however, you may find a reasonable path much faster and descent into the minimum much more steadfast and smoothly - in the end requiring remarkably fewer epochs until convergence!

But as we saw with our experiment 4: Even a wiggled start can end up in a pretty good minimum with a really good accuracy. Reducing the learning rate too fast may lead to a circle path with some distance to the minimum. We are talking here about the last < 0.5 percent.

Which minimum level you reach in the end depends on many parameters, but in particular also on the initial weight values. In general setting the initial weight values small enough with respect to the number of nodes on the lower neighbor layer seems to be reasonable.

The sigmoid function - and a major problem

It is time to think a bit deeper before we start more experiments. All in all one does not get rid of the feeling that something profound is wrong with our algorithm or our setup of the experiments. In my youth I have seen similar things in simulations on non-linear physics - and almost always a basic concept was missing or wrongly applied. Time to care about the math.

An important ingredient in the whole learning via back-propagation was the activation function, which due to its non-linearity has an impact on the gradients which we need to calculate. The sigmoid function is a smooth function; but it has some properties which obviously can lead to trouble.

One is that it produces function values pretty close to 1 for arguments x > 15.

sig(10) = 0.9999546021312976
sig(12) = 0.9999938558253978
sig(15) = 0.9999998874648379
sig(20) = 0.9999999979388463
sig(25) = 0.9999999999948910
sig(30) = 0.9999999999999065

So, function values for bigger arguments can almost not be distinguished and resulting gradients during backward propagation will get extremely small. Keeping this in mind we turn towards the initial steps of forward propagation. What happens to our input data there?

We directly present the feature values of the MNIST data images at 784 input nodes in layer L0. The following sketch only shows the basic architecture ofa a MLP; the node numbers do NOT correspond to our present MLP.

Then we multiply by the weights (randomly chosen initially from some interval) and accumulate 784 contributions at each of the 70 nodes of layer L1. Even if we choose the initial weight values to be in range of [-0.5, +0.5] this will potentially lead to big input values at layer L1 due to summing up all contributions. Thus at the output side of layer L1 our sigmoid function will produce many almost indistinguishable values and pretty small gradients in the first steps. This is certainly not good for a clear adjustment of weights during backward propagation.

There are two remedies, one can think about:

  • We should adapt the initial weight values to the number of nodes of the lower layer in forward propagation direction. A first guess would be something in the range 1.0e-3 for weights between layer L0 and L1 - assuming that ca. 10% of the 784 input features show values around 220. Weights between layers L1 and L2 should be in the range of [-0.05, 0.05] and between layer L2 and L3 in the range [-0.1, 0.1] to prevent maximum values above 5.
  • We should scale down the input data, i.e. we should normalize them such that they cover a reasonable value range which leads to distinguishable output values of the sigmoid function.

A plot for the first option with a reasonably small learn-rate as in experiment 7 and weights following the 1/sqrt(num_nodes) at every layer (!) is the following :

Quite OK, but not a breakthrough. So, let us look at normalization.

Normalization - Standardization

There are different methods how one can normalize values for a bunch of instances in a set. One basic method is to subtract the minimum value "x_min" of all instances from the value of each instance followed by a division of the difference between the max value (x_max) and the minimum value (x_max - x_min): x => (x - x_min) / (x_max - x_min).

A more clever version - which is called "standardization" - subtracts the mean value "x_mean" of all instances and divides by the standard deviation of the set. The resulting values have a mean of zero and a variance of 1. The advantage of this normalization approach is that it does not react strongly to extreme data values in the set; still it reduces big values to a very moderate scale.

SciKit-Learn provides the second normalization variant as a function with the name "StandardScaler" - this is the reason why we introduced an import statement for this function at the top of this article.

Code modifications to address standardization of the input data

Let us include standardization in our method to handle the input data:

Modifications to function "_method _handle_input_data()":

    ''' -- Method to handle different types of input data sets --''' 
    def _handle_input_data(self):    
        Method to deal with the input data: 
        - check if we have a known data set ("mnist" so far)
        - reshape as required 
        - analyze dimensions and extract the feature dimension(s) 
        # check for known dataset 
            if (self._my_data_set not in self._input_data_sets ): 
                raise ValueError
        except ValueError:
            print("The requested input data" + self._my_data_set + " is not known!" )
        # MNIST datasets 
        # **************
        # handle the mnist original dataset - is not supported any more 
        #if ( self._my_data_set == "mnist"): 
        #    mnist = fetch_mldata('MNIST original')
        #    self._X, self._y = mnist["data"], mnist["target"]
        #    print("Input data for dataset " + self._my_data_set + " : \n" + "Original shape of X = " + str(self._X.shape) +
        #      "\n" + "Original shape of y = " + str(self._y.shape))
        # handle the mnist_784 dataset 
        if ( self._my_data_set == "mnist_784"): 
            mnist2 = fetch_openml('mnist_784', version=1, cache=True, data_home='~/scikit_learn_data') 
            self._X, self._y = mnist2["data"], mnist2["target"]
            print ("data fetched")
            # the target categories are given as strings not integers 
            self._y = np.array([int(i) for i in self._y])
            print ("data modified")
            print("Input data for dataset " + self._my_data_set + " : \n" + "Original shape of X = " + str(self._X.shape) +
              "\n" + "Original shape of y = " + str(self._y.shape))
        # handle the mnist_keras dataset 
        if ( self._my_data_set == "mnist_keras"): 
            (X_train, y_train), (X_test, y_test) = kmnist.load_data()
            len_train =  X_train.shape[0]
            len_test  =  X_test.shape[0]
            X_train = X_train.reshape(len_train, 28*28) 
            X_test  = X_test.reshape(len_test, 28*28) 
            # Concatenation required due to possible later normalization of all data
            self._X = np.concatenate((X_train, X_test), axis=0)
            self._y = np.concatenate((y_train, y_test), axis=0)
            print("Input data for dataset " + self._my_data_set + " : \n" + "Original shape of X = " + str(self._X.shape) +
              "\n" + "Original shape of y = " + str(self._y.shape))
        # common MNIST handling 
        if ( self._my_data_set == "mnist" or self._my_data_set == "mnist_784" or self._my_data_set == "mnist_keras" ): 
        # handle IMPORTED datasets 
        # **************************+
        if ( self._my_data_set == "imported"): 
            if (self._X_import is not None) and (self._y_import is not None):
                self._X = self._X_import
                self._y = self._y_import
                print("Shall handle imported datasets - but they are not defined")
        # number of total records in X, y
        # *******************************
        self._dim_X = self._X.shape[0]
        # Give control to preprocessing - has to happen before normalizing and splitting 
        # ****************************
        # Common dataset handling 
        # ************************
        # normalization 
        if self._b_normalize_X: 
            # normalization by sklearn.preprocessing.StandardScaler
            scaler = StandardScaler()
            self._X = scaler.fit_transform(self._X)
        # mixing the training indices - MUST happen BEFORE encoding 
        shuffled_index = np.random.permutation(self._dim_X)
        self._X, self._y = self._X[shuffled_index], self._y[shuffled_index]

        # Splitting into training and test datasets 
        if self._num_test_records > 0.25 * self._dim_X:
            print("\nNumber of test records bigger than 25% of available data. Too big, we stop." )
            num_sep = self._dim_X - self._num_test_records
            self._X_train, self._X_test, self._y_train, self._y_test = self._X[:num_sep], self._X[num_sep:], self._y[:num_sep], self._y[num_sep:] 
        # numbers, dimensions
        self._dim_sets = self._y_train.shape[0]
        self._dim_features = self._X_train.shape[1] 
        print("\nFinal dimensions of training and test datasets of type " + self._my_data_set + 
              " : \n" + "Shape of X_train = " + str(self._X_train.shape) + 
              "\n" + "Shape of y_train = " + str(self._y_train.shape) + 
              "\n" + "Shape of X_test = " + str(self._X_test.shape) + 
              "\n" + "Shape of y_test = " + str(self._y_test.shape) 
        print("\nWe have " + str(self._dim_sets) + " data records for training") 
        print("Feature dimension is " + str(self._dim_features)) 
        # encoding the y-values = categories // MUST happen AFTER encoding 
        ''' Remark: Other input data sets can not yet be handled ''' 
        return None

Well, this looks a bit different compared to our original function. Actually, we perform normalization twice. Once inside the new function "_preprocess_input_data()" and once afterwards.

New function "_preprocess_data()":

    Method to preprocess the input data 
    ----------------------------------- '''
    def _preprocess_input_data(self):
        # normalization ahead
        if self._b_normalize_X: 
            # normalization by sklearn.preprocessing.StandardScaler
            scaler = StandardScaler()
            self._X = scaler.fit_transform(self._X)

        # Clustering 
        if self._b_perform_clustering:
            print("\nClustering started")
            print("\nNo Clustering requested")
        return None

The reason is that we have to take into account other transformations of the input data by other methods, too. One of these methods will be clustering, which we shall investigate in a forthcoming article. (For the nervous ones among the readers: The StandardScaler is intelligent enough to avoid divisions by zero means at the second time it is called!)

Experiment 8: Standardizes input data, reduced learn-rate, increased decrease-rate and "1/sqrt(nodes)-rule for the initial weights of all layers

We shall call our class My_ANN now with the parameter "b_normalize_X = True", i.e. we standardize the whole MNIST input data set X before we split it into a training and a test data set.

In addition we apply the rule to set the interval-borders for initial weights to [-1.0/sqrt(num_nodes_layer), 1.0/sqrt(num_nodes_layer)], with "num_nodes_layer" being the number of nodes in the lower layer which the weights act upon during forward propagation.

Test 8:
Parameters: learn_rate = 0.0002, decrease_rate = 0.00001, mom_rate = 0.00005, n_size_mini_batch = 500, Lambda2 = 0.2, n_epochs = 2000, weights at all layers in [- 1.0/sqrt(num_nodes_layer), 1.0/sqrt(num_nodes_layer)]
Results: acc_train: 0.9913 , acc_test: 0.9689, convergence after ca. 650 epochs

Wow, extremely smooth curves now - and we got the highest accuracy so far!

Experiment 9: Standardized input, bigger initial learning, enlarged intervals for weight initialization

We get brave again! we enlarge the learning-rate back to 0.001. In addition we enlarge the intervals for a random distribution of initial weights for each layer by a factor of 2 =>- [-2*1.0/sqrt(num_nodes_layer), 2*1.0/sqrt(num_nodes_layer)].

Test 9:
Parameters: learn_rate = 0.001, decrease_rate = 0.00001, mom_rate = 0.00005, n_size_mini_batch = 500, n_epochs = 1200, weights at all layers in [-2*1.0/sqrt(num_nodes_layer), 2*1.0/sqrt(num_nodes_layer)]
Results: acc_train: 0.9949 , acc_test: 0.9754, convergence after ca. 550-600 epochs

Not such smooth curves as in the previous plot. But WoW again - now we broke the 0.97-threshold - already at an period as small as 100!

I admit that a very balanced initial statistical distribution of digit images across the training and the test datasets helped in this specific test run, but only a bit. You will easily and regularly pass a value of 0.972 for the accuracy on the test dataset during multiple consecutive runs. Compared to our reference value of 0.96 this is a solid improvement!

But what is really convincing is the fact the even with a relatively high initial learning rate we see no major trouble on our way to the minimum! I would call this a breakthrough!


We learned today that working with mini-batch training can be tricky. In some cases we may need to control a balance between a sufficiently small initial learning rate and a reasonable reduction rate during training. We also saw that it is helpful to get some control over the weight initialization. The rule to create randomly distributed initial weight values initialization within intervals given by [n*1/sqrt(num_nodes), n*1/sqrt(num_nodes)] appears to be useful.

However, the real lesson of our experiments was that we do our MLP learning algorithm a major favor by normalizing and centering the input data.

At least if the sigmoid function is applied as the activation function at the MLP's nodes a initial standardization of the input data should always be tested and compared to training runs without standardization.

In the next article of this series we shall have a look at a transformation of the input data called "clustering". The objective of such a step is the reduction of input channels and such a significant reduction of CPU time. Stay tuned!