A simple program for an ANN to cover the Mnist dataset – III – forward propagation

A simple program for an ANN to cover the Mnist dataset – II

A simple program for an ANN to cover the Mnist dataset – I

we came so far that we could apply the "Feed Forward Propagation" algorithm [FFPA] to multiple data records of a mini-batch of training data in parallel. We spoke of a so called *vectorized* form of the FFPA; we used special Linear Algebra matrix operations of Numpy to achieve the parallel operations. In the last article

A simple program for an ANN to cover the Mnist dataset – IV – the concept of a cost or loss function

I commented on the necessity of a so called "*loss function*" for the MLP. Although not required for a proper training algorithm we will nevertheless encode a class method to calculate cost values for mini-batches. The behavior of such cost values with training epochs will give us an impression of how good the training algorithm works and whether it actually converges into a minimum of the loss function. As explained in the last article this minimum should correspond to an overall minimum distance of the FFPA results for *all* training data records from their known correct target values in the result vector space of the MLP.

Before we do the coding for two specific cost or loss functions - namely the "**Log Loss**"-function and the "**MSE**"-function, I will briefly point out the difference between standard "*"-operations between multidimensional Numpy arrays and real "dot"-matrix-operations in the sense of Linear Algebra. The latte one follows special rules in multiplying specific elements of both matrices and summing up over the results.

As in all the other articles of this series: This is for beginners; experts will not learn anything new - especially not of the first section.

I would like to point out some aspects of combining two multidimensional Numpy arrays which may be confusing for Python beginners. At least they were for me . As a former physicist I automatically expected a "*"-like operation for two multidimensional arrays to perform a matrix operation in the sense of linear algebra. This lead to problems when I tried to understand Python code of others.

Let us assume we have two 2-dimensional arrays **A** and **B**. **A** and **B** shall be similar in the sense that their *shape* is identical, i.e. A.shape = B.shape - e.g (784, 60000):

The two matrices each have the same specific number of elements in their different dimensions.

Whenever we operate with multidimensional Numpy arrays with the same *same shape* we can use the standard operators "+", "-", "*", "/". These operators then are applied between corresponding elements of the matrices. I.e., the mathematical operation is applied between elements with the same position along the different dimensional axes in **A** and **B**. We speak of an element-wise operation. See the example below.

This means (**A** ***** **B**) is **not** equivalent to the **C** = numpy.dot(**A**, **B**) operation - which appears in Linear Algebra; e.g. for vector and operator transformations!

The"dot()"-operation implies a special operation: Let us assume that the shape of **A[i,j,v]** is

A.shape = (p,q,y)

and the shape of **B[k,w,m]** is

B.shape = (r,z,s)

with

y = z .

Then in the "dot()"-operation all elements of a dimension "v" of **A[i,j,v]** are multiplied with corresponding elements of the dimension "w" of **B[k,w,m]** and then the results summed up.

dot(**A**, **B**)[i,j,k,m] = sum(**A**[i,j,:] * **B**[k,:,m])

The "*" operation in the formula above is to be interpreted as a standard multiplication of array elements.

In the case of A being a 2-dim array and B being a 1-dimensional vector we just get an operation which could - under certain conditions - be interpreted as a typical vector transformation in a 2-dim vector space.

So, when we define two Numpy arrays there may exist two different methods to deal with array-multiplication: If we have two arrays with the same shape, then the "*"-operation means an element-wise multiplication of the elements of both matrices. In the context of ANNs such an operation may be useful - even if real linear algebra matrix operations dominate the required calculations. The first "*"-operation will, however, not work if the array-shapes deviate.

The "**numpy.dot(A, B)**"-operation instead requires a correspondence of the last dimension of matrix **A** with the second to last dimension of matrix **B**. Ooops - I realize I just used the expression "*matrix*" for a multidimensional Numpy array without much thinking. As said: "matrix" in linear algebra has a connotation of a transformation operator on vectors of a vector space. Is there a difference in Numpy?

Yes, there is, indeed - which may even lead to more confusion: We can apply the function numpy.matrix()

A = numpy.matrix(A),

B = numpy.matrix(B)

then the "*"-operator will get a different meaning - namely that of numpy.dot(A,B):

**A** * **B** = numpy.dot(**A**, **B**)

So, better read Python code dealing with multidimensional arrays rather carefully ....

To understand this better let us execute the following operations on some simple examples in a Jupyter cell:

A1 = np.ones((5,3)) A1[:,1] *= 2 A1[:,2] *= 4 print("\nMatrix A1:\n") print(A1) A2= np.random.randint(1, 10, 5*3) A2 = A2.reshape(5,3) # A2 = A2.reshape(3,5) print("\n Matrix A2 :\n") print(A2) A3 = A1 * A2 print("\n\nA3:\n") print(A3) A4 = np.dot(A1, A2.T) print("\n\nA4:\n") print(A4) A5 = np.matrix(A1) A6 = np.matrix(A2) A7 = A5 * A6.T print("\n\nA7:\n") print(A7) A8 = A5 * A6

We get the following output:

Matrix A1: [[1. 2. 4.] [1. 2. 4.] [1. 2. 4.] [1. 2. 4.] [1. 2. 4.]] Matrix A2 : [[6 8 9] [9 1 6] [8 8 9] [2 8 3] [5 8 8]] A3: [[ 6. 16. 36.] [ 9. 2. 24.] [ 8. 16. 36.] [ 2. 16. 12.] [ 5. 16. 32.]] A4: [[58. 35. 60. 30. 53.] [58. 35. 60. 30. 53.] [58. 35. 60. 30. 53.] [58. 35. 60. 30. 53.] [58. 35. 60. 30. 53.]] A7: [[58. 35. 60. 30. 53.] [58. 35. 60. 30. 53.] [58. 35. 60. 30. 53.] [58. 35. 60. 30. 53.] [58. 35. 60. 30. 53.]] --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-10-4ea2dbdf6272> in <module> 28 print(A7) 29 ---> 30 A8 = A5 * A6 31 /projekte/GIT/ai/ml1/lib/python3.6/site-packages/numpy/matrixlib/defmatrix.py in __mul__(self, other) 218 if isinstance(other, (N.ndarray, list, tuple)) : 219 # This promotes 1-D vectors to row vectors --> 220 return N.dot(self, asmatrix(other)) 221 if isscalar(other) or not hasattr(other, '__rmul__') : 222 return N.dot(self, other) <__array_function__ internals> in dot(*args, **kwargs) ValueError: shapes (5,3) and (5,3) not aligned: 3 (dim 1) != 5 (dim 0)

This example obviously demonstrates the difference of an multiplication operation on multidimensional arrays and a real matrix "dot"-operation. Note especially how the "*" operator changed when we calculated **A7**.

If we instead execute the following code

A1 = np.ones((5,3)) A1[:,1] *= 2 A1[:,2] *= 4 print("\nMatrix A1:\n") print(A1) A2= np.random.randint(1, 10, 5*3) #A2 = A2.reshape(5,3) A2 = A2.reshape(3,5) print("\n Matrix A2 :\n") print(A2) A3 = A1 * A2 print("\n\nA3:\n") print(A3)

we directly get an error:

Matrix A1: [[1. 2. 4.] [1. 2. 4.] [1. 2. 4.] [1. 2. 4.] [1. 2. 4.]] Matrix A2 : [[5 8 7 3 8] [4 4 8 4 5] [8 1 9 4 8]] --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-12-c4d3ffb1e683> in <module> 13 14 ---> 15 A3 = A1 * A2 16 print("\n\nA3:\n") 17 print(A3) ValueError: operands could not be broadcast together with shapes (5,3) (3,5)

As expected!

As we want to be able to use different types of cost/loss functions we have to introduce new corresponding parameters in the class's interface. So we update the "__init__()"-function:

def __init__(self, my_data_set = "mnist", n_hidden_layers = 1, ay_nodes_layers = [0, 100, 0], # array which should have as much elements as n_hidden + 2 n_nodes_layer_out = 10, # expected number of nodes in output layer my_activation_function = "sigmoid", my_out_function = "sigmoid", my_loss_function = "LogLoss", n_size_mini_batch = 50, # number of data elements in a mini-batch n_epochs = 1, n_max_batches = -1, # number of mini-batches to use during epochs - > 0 only for testing # a negative value uses all mini-batches lambda2_reg = 0.1, # factor for quadratic regularization term lambda1_reg = 0.0, # factor for linear regularization term vect_mode = 'cols', figs_x1=12.0, figs_x2=8.0, legend_loc='upper right', b_print_test_data = True ): ''' Initialization of MyANN Input: data_set: type of dataset; so far only the "mnist", "mnist_784" datsets are known We use this information to prepare the input data and learn about the feature dimension. This info is used in preparing the size of the input layer. n_hidden_layers = number of hidden layers => between input layer 0 and output layer n ay_nodes_layers = [0, 100, 0 ] : We set the number of nodes in input layer_0 and the output_layer to zero Will be set to real number afterwards by infos from the input dataset. All other numbers are used for the node numbers of the hidden layers. n_nodes_out_layer = expected number of nodes in the output layer (is checked); this number corresponds to the number of categories NC = number of labels to be distinguished my_activation_function : name of the activation function to use my_out_function : name of the "activation" function of the last layer which produces the output values my_loss_function : name of the "cost" or "loss" function used for optimization n_size_mini_batch : Number of elements/samples in a mini-batch of training data The number of mini-batches will be calculated from this n_epochs : number of epochs to calculate during training n_max_batches : > 0: maximum of mini-batches to use during training < 0: use all mini-batches lambda_reg2: The factor for the quadartic regularization term lambda_reg1: The factor for the linear regularization term vect_mode: Are 1-dim data arrays (vctors) ordered by columns or rows ? figs_x1=12.0, figs_x2=8.0 : Standard sizing of plots , legend_loc='upper right': Position of legends in the plots b_print_test_data: Boolean variable to control the print out of some tests data ''' # Array (Python list) of known input data sets self._input_data_sets = ["mnist", "mnist_784", "mnist_keras"] self._my_data_set = my_data_set # X, y, X_train, y_train, X_test, y_test # will be set by analyze_input_data # X: Input array (2D) - at present status of MNIST image data, only. # y: result (=classification data) [digits represent categories in the case of Mnist] self._X = None self._X_train = None self._X_test = None self._y = None self._y_train = None self._y_test = None # relevant dimensions # from input data information; will be set in handle_input_data() self._dim_sets = 0 self._dim_features = 0 self._n_labels = 0 # number of unique labels - will be extracted from y-data # Img sizes self._dim_img = 0 # should be sqrt(dim_features) - we assume square like images self._img_h = 0 self._img_w = 0 # Layers # ------ # number of hidden layers self._n_hidden_layers = n_hidden_layers # Number of total layers self._n_total_layers = 2 + self._n_hidden_layers # Nodes for hidden layers self._ay_nodes_layers = np.array(ay_nodes_layers) # Number of nodes in output layer - will be checked against information from target arrays self._n_nodes_layer_out = n_nodes_layer_out # Weights # -------- # empty List for all weight-matrices for all layer-connections # Numbering : # w[0] contains the weight matrix which connects layer 0 (input layer ) to hidden layer 1 # w[1] contains the weight matrix which connects layer 1 (input layer ) to (hidden?) layer 2 self._ay_w = [] # Arrays for encoded output labels - will be set in _encode_all_mnist_labels() # ------------------------------- self._ay_onehot = None self._ay_oneval = None # Known Randomizer methods ( 0: np.random.randint, 1: np.random.uniform ) # ------------------ self.__ay_known_randomizers = [0, 1] # Types of activation functions and output functions # ------------------ self.__ay_activation_functions = ["sigmoid"] # later also relu self.__ay_output_functions = ["sigmoid"] # later also softmax # Types of cost functions # ------------------ self.__ay_loss_functions = ["LogLoss", "MSE" ] # later also othr types of cost/loss functions # the following dictionaries will be used for indirect function calls self.__d_activation_funcs = { 'sigmoid': self._sigmoid, 'relu': self._relu } self.__d_output_funcs = { 'sigmoid': self._sigmoid, 'softmax': self._softmax } self.__d_loss_funcs = { 'LogLoss': self._loss_LogLoss, 'MSE': self._loss_MSE } # The following variables will later be set by _check_and set_activation_and_out_functions() self._my_act_func = my_activation_function self._my_out_func = my_out_function self._my_loss_func = my_loss_function self._act_func = None self._out_func = None self._loss_func = None # list for cost values of mini-batches during training # The list will later be split into sections for epochs self._ay_cost_vals = [] # number of data samples in a mini-batch self._n_size_mini_batch = n_size_mini_batch self._n_mini_batches = None # will be determined by _get_number_of_mini_batches() # number of epochs self._n_epochs = n_epochs # maximum number of batches to handle (<0 => all!) self._n_max_batches = n_max_batches # regularization parameters self._lambda2_reg = lambda2_reg self._lambda1_reg = lambda1_reg # paramter to allow printing of some test data self._b_print_test_data = b_print_test_data # Plot handling # -------------- # Alternatives to resize plots # 1: just resize figure 2: resize plus create subplots() [figure + axes] self._plot_resize_alternative = 1 # Plot-sizing self._figs_x1 = figs_x1 self._figs_x2 = figs_x2 self._fig = None self._ax = None # alternative 2 does resizing and (!) subplots() self.initiate_and_resize_plot(self._plot_resize_alternative) # *********** # operations # *********** # check and handle input data self._handle_input_data() # set the ANN structure self._set_ANN_structure() # Prepare epoch and batch-handling - sets mini-batch index array, too self._prepare_epochs_and_batches() # perform training start_c = time.perf_counter() self._fit(b_print=False, b_measure_batch_time=False) end_c = time.perf_counter() print('\n\n ------') print('Total training Time_CPU: ', end_c - start_c) print("\nStopping program regularily") sys.exit() #

The way of accessing a method/function by a parameterized "name"-string should already be familiar from other methods. The method with the given name must of course exist in the Python module; otherwise already Eclipse#s PyDev we display errors.

'''-- Method to set the loss function--''' def _check_and_set_loss_function(self): # check for known loss functions try: if (self._my_loss_func not in self.__d_loss_funcs ): raise ValueError except ValueError: print("\nThe requested loss function " + self._my_loss_func + " is not known!" ) sys.exit() # set the function to variables for indirect addressing self._loss_func = self.__d_loss_funcs[self._my_loss_func] if self._b_print_test_data: z = 2.0 print("\nThe loss function of the ANN/MLP was defined as \"" + self._my_loss_func + '"') ''' ''' return None #

The "LogLoss"-function has a special form. If "**a_i**" characterizes the FFPA result for a special training record and "**y_i**" the real known value for this record then we calculate its contribution to the costs as:

Loss = SUM_i [- **y_i** * log(**a**_i) - (1 - **y_i**)*log(1 - **a_i**)]

This loss function has its justification in statistical considerations - for which we assume that our output function produces a kind of probability distribution. Please see the literature for more information.

Now, due to the encoded result representation over 10 different output dimensions in the MNIST case, corresponding to 10 nodes in the output layer; see the second article of this series, we know that **a_i** and **y_i** would be 1-dimensional arrays for each training data record. However, if we vectorize this by treating all records of a mini-batch in parallel we get 2-dim arrays. Actually, we have already calculated the respective arrays in the second to last article.

The rows (1st dim) of **a** represent the output nodes (training data records, the columns (2nd dim) of **a** represent the results of the FFPA-result values, which due to our output function have values in the interval ]0.0, 1.0].

The same holds for **y** - with the difference, that 9 of the values in the rows are 0 and exactly one is 1 for a training record.

The "*" multiplication thus can be done via a normal element-wise array "multiplication" on the given 2-dim arrays of our code.

**a** = ay_ANN_out**y** = ay_y_enc

Numpy offers a function "numpy.sum(M)" for a multidimensional array **M**, which just sums up all element values. The result is of course a simple scalar.

This information should be enough to understand the following new method:

''' method to calculate the logistic regression loss function ''' def _loss_LogLoss(self, ay_y_enc, ay_ANN_out, b_print = False): ''' Method which calculates LogReg loss function in a vectorized form on multidimensional Numpy arrays ''' b_test = False if b_print: print("From LogLoss: shape of ay_y_enc = " + str(ay_y_enc.shape)) print("From LogLoss: shape of ay_ANN_out = " + str(ay_ANN_out.shape)) print("LogLoss: ay_y_enc = ", ay_y_enc) print("LogLoss: ANN_out = \n", ay_ANN_out) print("LogLoss: log(ay_ANN_out) = \n", np.log(ay_ANN_out) ) # The following means an element-wise (!) operation between matrices of the same shape! Log1 = -ay_y_enc * (np.log(ay_ANN_out)) # The following means an element-wise (!) operation between matrices of the same shape! Log2 = (1 - ay_y_enc) * np.log(1 - ay_ANN_out) # the next operation calculates the sum over all matrix elements # - thus getting the total costs for all mini-batch elements cost = np.sum(Log1 - Log2) #if b_print and b_test: # Log1_x = -ay_y_enc.dot((np.log(ay_ANN_out)).T) # print("From LogLoss: L1 = " + str(L1)) # print("From LogLoss: L1X = " + str(L1X)) if b_print: print("From LogLoss: cost = " + str(cost)) # The total costs is just a number (scalar) return cost

Although not often used for classification tasks (but more for regression problems) this loss function is so simple that we encode it on the fly. Here we just calculate something like a mean quadratic error:

Loss = 9.5 * SUM_i [ (**y_i** - **a_i**)**2 ]

This loss function is convex by definition and leads to the following method code:

''' method to calculate the MSE loss function ''' def _loss_MSE(self, ay_y_enc, ay_ANN_out, b_print = False): ''' Method which calculates LogReg loss function in a vectorized form on multidimensional Numpy arrays ''' if b_print: print("From loss_MSE: shape of ay_y_enc = " + str(ay_y_enc.shape)) print("From loss_MSE: shape of ay_ANN_out = " + str(ay_ANN_out.shape)) #print("LogReg: ay_y_enc = ", ay_y_enc) #print("LogReg: ANN_out = \n", ay_ANN_out) #print("LogReg: log(ay_ANN_out) = \n", np.log(ay_ANN_out) ) cost = 0.5 * np.sum( np.square( ay_y_enc - ay_ANN_out ) ) if b_print: print("From loss_MSE: cost = " + str(cost)) return cost #

Regularization is a means against overfitting during training. The trick is that the cost function is enhanced by terms which include sums of linear or quadratic terms of all weights of all layers. This enforces that the weights themselves get minimized, too, in the search for a minimum of the loss function. The less degrees of freedom there are the less the chance of overfitting ...

In the literature (see the book hints in the last article) you find 2 methods for regularization - one with quadratic terms of the weights - the so called "Ridge-Regression" - and one based on a sum of absolute values of the weights - the so called "Lasso regression". See the books of Geron and Rashka for more information.

Loss = SUM_i [- **y_i** * log(**a**_i) - (1 - **y_i**)*log(1 - **a_i**)] **+** lambda_2 * SUM_layer [ SUM_nodes [ (w_layer_nodes)**2 ] ]**+** lambda_1 * SUM_layer [ SUM_nodes [ |w_layer_nodes| ] ]

Note that we included already two factors "lambda_2" and "lamda_1" by which the regularization terms are multiplied and added to the cost/loss function in the "__init__"-method.

The two related methods are easy to understand:

''' method do calculate the quadratic regularization term for the loss function ''' def _regularize_by_L2(self, b_print=False): ''' The L2 regularization term sums up all quadratic weights (without the weight for the bias) over the input and all hidden layers (but not the output layer The weight for the bias is in the first column (index 0) of the weight matrix - as the bias node's output is in the first row of the output vector of the layer ''' ilayer = range(0, self._n_total_layers-1) # this excludes the last layer L2 = 0.0 for idx in ilayer: L2 += (np.sum( np.square(self._ay_w[idx][:, 1:])) ) L2 *= 0.5 * self._lambda2_reg if b_print: print("\nL2: total L2 = " + str(L2) ) return L2 #

''' method do calculate the linear regularization term for the loss function ''' def _regularize_by_L1(self, b_print=False): ''' The L1 regularization term sums up all weights (without the weight for the bias) over the input and all hidden layers (but not the output layer The weight for the bias is in the first column (index 0) of the weight matrix - as the bias node's output is in the first row of the output vector of the layer ''' ilayer = range(0, self._n_total_layers-1) # this excludes the last layer L1 = 0.0 for idx in ilayer: L1 += (np.sum( self._ay_w[idx][:, 1:])) L1 *= 0.5 * self._lambda1_reg if b_print: print("\nL1: total L1 = " + str(L1)) return L1 #

Why do we not start with index "0" in the weight arrays - self._ay_w[idx][:, 1:]?

The reason is that we do not include the Bias-node in these terms. The weight at the bias nodes of the layers is not varied there during optimization!

**Note:** Normally we would expect a factor of **1/m**, with "**m**" being the number of records in a mini-batch, for all the terms discussed above. Such a constant factor does not hamper the principal procedure - if we omit it consistently also for for the regularization terms discussed below. It can be taken care of by choosing smaller "lambda"s and a smaller step size during optimization.

For our approach with mini-batches (i.e. an approach between pure stochastic and full batch handling) we have to include the cost calculation in our method "_handle_mini_batch()" to handle mini-batches. Method "_handle_mini_batch()" is modified accordingly:

''' -- Method to deal with a batch -- ''' def _handle_mini_batch(self, num_batch = 0, b_print_y_vals = False, b_print = False): ''' For each batch we keep the input data array Z and the output data A (output of activation function!) for all layers in Python lists We can use this as input variables in function calls - mutable variables are handled by reference values ! We receive the A and Z data from propagation functions and proceed them to cost and gradient calculation functions As an initial step we define the Python lists ay_Z_in_layer and ay_A_out_layer and fill in the first input elements for layer L0 ''' ay_Z_in_layer = [] # Input vector in layer L0; result of a matrix operation in L1,... ay_A_out_layer = [] # Result of activation function #print("num_batch = " + str(num_batch)) #print("len of ay_mini_batches = " + str(len(self._ay_mini_batches))) #print("_ay_mini_batches[0] = ") #print(self._ay_mini_batches[num_batch]) # Step 1: Special treatment of the ANN's input Layer L0 # Layer L0: Fill in the input vector for the ANN's input layer L0 ay_idx_batch = self._ay_mini_batches[num_batch] ay_Z_in_layer.append( self._X_train[ay_idx_batch] ) # numpy arrays can be indexed by an array of integers #print("\nPropagation : Shape of X_in = ay_Z_in_layer = " + str(ay_Z_in_layer[0].shape)) if b_print_y_vals: print("\n idx, expected y_value of Layer L0-input :") for idx in self._ay_mini_batches[num_batch]: print(str(idx) + ', ' + str(self._y_train[idx]) ) # Step 2: Layer L0: We need to transpose the data of the input layer ay_Z_in_0T = ay_Z_in_layer[0].T ay_Z_in_layer[0] = ay_Z_in_0T # Step 3: Call the forward propagation method for the mini-batch data samples self._fw_propagation(ay_Z_in = ay_Z_in_layer, ay_A_out = ay_A_out_layer, b_print = b_print) if b_print: # index range of layers ilayer = range(0, self._n_total_layers) print("\n ---- ") print("\nAfter propagation through all " + str(self._n_total_layers) + " layers: ") for il in ilayer: print("Shape of Z_in of layer L" + str(il) + " = " + str(ay_Z_in_layer[il].shape)) print("Shape of A_out of layer L" + str(il) + " = " + str(ay_A_out_layer[il].shape)) # Step 4: To be done: cost calculation for the batch ay_y_enc = self._ay_onehot[:, ay_idx_batch] ay_ANN_out = ay_A_out_layer[self._n_total_layers-1] # print("Shape of ay_ANN_out = " + str(ay_ANN_out.shape)) total_costs_batch = self._calculate_loss_for_batch(ay_y_enc, ay_ANN_out, b_print = False) self._ay_cost_vals.append(total_costs_batch) # Step 5: To be done: gradient calculation via back propagation of errors # Step 6: Adjustment of weights # try to accelerate garbage handling if len(ay_Z_in_layer) > 0: del ay_Z_in_layer if len(ay_A_out_layer) > 0: del ay_A_out_layer return None #

Note that we save the cost values of every batch in the 1-dim array "self._ay_cost_vals". This array can later on easily be split into arrays for epochs.

The whole process must be supplemented by a method which does the real cost value calculation:

''' -- Main Method to calculate costs -- ''' def _calculate_loss_for_batch(self, ay_y_enc, ay_ANN_out, b_print = False, b_print_details = False ): ''' Method which calculates the costs including regularization terms The cost function is called according to an input parameter of the class ''' pure_costs_batch = self._loss_func(ay_y_enc, ay_ANN_out, b_print = False) if ( b_print and b_print_details ): print("Calc_Costs: Shape of ay_ANN_out = " + str(ay_ANN_out.shape)) print("Calc_Costs: Shape of ay_y_enc = " + str(ay_y_enc.shape)) if b_print: print("From Calc_Costs: pure costs of a batch = " + str(pure_costs_batch)) # Add regularitzation terms - L1: linear reg. term, L2: quadratic reg. term # the sums over the weights (squared) have to be performed for each batch again due to intermediate corrections L1_cost_contrib = 0.0 L2_cost_contrib = 0.0 if self._lambda1_reg > 0: L1_cost_contrib = self._regularize_by_L1( b_print=False ) if self._lambda2_reg > 0: L2_cost_contrib = self._regularize_by_L2( b_print=False ) total_costs_batch = pure_costs_batch + L1_cost_contrib + L2_cost_contrib return total_costs_batch #

By the steps discussed above we completed the inclusion of a cost value calculation in our class for every step dealing with a mini-batch during training. All cost values are saved in a Python list for later evaluation. The list can later be split with respect to epochs.

In contrast to the FFP-algorithm all array-operations required in this step were simple element-wise operations and summations over all array-elements.

Cost value calculation obviously is simple and pretty fast regarding CPU-consumption! Just test it yourself!

In the next article we shall analyze the mathematics behind the calculation of the partial derivatives of our cost-function with respect to the many weights at all nodes of the different layers. We shall see that the gradient calculation reduces to remarkable simple formulas describing a kind of back-propagation of the error terms [**y_i** - **a_i**] through the network.

We will not be surprised that we need to involve some real matrix operations again as in the FFPA !

]]>

kamarada: how-to-upgrade-from-opensuse-leap-150-to-151/ .

The only problems I got had to do with the Optimus-design of my old laptop. When I followed my own description how to install and use Bumblebee as described in

Installation Opensuse Leap 15 auf Laptop – Grafik Probleme, Optimus

I always got an error message when trying to load the nvidia kernel module:

mytux:~ # sudo modprobe nvidia modprobe: ERROR: could not insert 'nvidia': No such device

The reason was given by the command "dmesg"; the Nvidia device was no longer available on the PCI bus:

NVRM: The NVIDIA GPU 0000:01:00.0 NVRM: (PCI ID: 10de:134d) installed in this system has NVRM: fallen off the bus and is not responding to commands. [ 3.312435] nvidia: probe of 0000:01:00.0 failed with error -1

The bbswitch-module could, however, be loaded without any problems.

A workaround is described in here :

After having started KDE or Gnome issue the following commands as root in a terminal:

mytux:~ # echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove

mytux:~ # echo 1 > /sys/bus/pci/devices/0000:00:02.0/rescan

Afterwards my Nvidia card (640M) was available on the bus again - and e.g. "primusrun glxgears" worked.

So, something with starting the dkms.service and the bumblebeed.service, switching off the Nvidia card by the bbswitch module during system startup and later on loading of the nvidia-module was failing. I suspected the bbswitch-module to be the cause ...

In my case I could solve the problem by installing the RPM packets

**bumblebee, dkms, bbswitch, bbswitch-kmp-default**

from the *standard* Update repository of Opensuse Leap 15.1

https://download.opensuse.org/update/leap/15.1/oss/

instead of installing them from the Bumblebee-repository

https://download.opensuse.org/repositories/X11:/Bumblebee/openSUSE_Leap_15.1

Otherwise I followed the instructions in

Installation Opensuse Leap 15 auf Laptop – Grafik Probleme, Optimus .

I.e.: I installed only the packets

nvidia-bumblebee, nvidia-bumblebee-32bit

from the Bumblebee repository.

Do not forget to issue a "**mkinitrd**" after you have successfully tested e.g. "optirun glxgears" and "tee /proc/acpi/bbswitch <<< OFF". Then reboot and use Optimus as you were used to.
I do not know what is wrong with the packets in the Bumblebee repository - but I hope this bug is fixed soon. It is a bit annoying when one has to play around with packets of different repositories.

A simple program for an ANN to cover the Mnist dataset – III – forward propagation

A simple program for an ANN to cover the Mnist dataset – II

A simple program for an ANN to cover the Mnist dataset – I

I have already explained

- what parts of an MLP setup we need to parameterize; e.g. the number of layers, the number of nodes per layer, the activation and output functions;
- how we create node layers and the corresponding weight arrays,
- how (and also a bit of why) we work with "mini-batches" of test data during training,
- how we can realize a "vectorized" form of the required "Feed Forward Propagation" algorithm [
**FFP**]. A vectorized form enables us to process all training data records of a mini-batch in parallel. We used Linear Algebra functions provided by Numpy for this purpose; these functions are supported by the the OpenBlas library on a Linux system.

We also set up a basic loop over a number of epochs during training. (Remember: An epoch corresponds to a training step over *all* training data records). The number of epochs is handled as a parameter to the class's interface. By artificially repeating the FFP algorithm up to a thousand times, we already got an impression of the code's performance and its dependence on the number of CPU cores and the size of a mini-batch.

A special method of our class MyANN controls the handling of a mini-batch of multiple input data records via two major steps so far:

- Step 1: Extract the data records for the mini-batch from the input data.
- Step 2: Apply FW-propagation to all data records of the mini-batch.

The next natural step would be to encode a training algorithm which optimizes the weight parameters of our MLP. However, in this article we shall not code anything. Instead, I shall discuss some aspects of the so called **"cost function"** of a MLP. I think this to be useful to get a basic understanding of what training of an ANN actually means and what the differences are in comparison to other ML-algorithms as e.g. the SVM approach. Understanding the cost function's role for the training of a MLP will also help to better understand the origin and the mathematical form of the back-propagation-algorithm used for training and discussed in a later article.

I simplify a lot below; more details can be found in the literature on machine Learning [ML]; see the section "Links" for some references. Note that if you know all about the theoretical concepts behind ANN training you will not learn anything new here. This is for beginners (and for later reference in this article series).

What do we mean by training an ANN? Training means to optimize the weights of the ANN such that the "Feed Forward Propagation" in the end delivers correct predictions for new datasets. A cost function is a central concept of the so called **"gradient descent method"** used for this optimization. By the way: A synonym for cost function is **"loss function"**. We use both terms alike below.

The relation between *ANN-training* based on a loss function and the *classification task*, which we want to solve with an ANN, is a subtle one. Let us first discuss what we understand by "classification":

Classification means to separate the input data into *categories*; i.e.: finding categorical separation surfaces in the multidimensional vector space of input data. In case of the MNIST dataset such separation interfaces should discriminate between 10 different clusters of data points.

I have discussed the problem of finding a separation surface for the case of the moons dataset example in previous articles in this blog. We then used SVM-algorithms to solve this particular problem. Actually, we determined parameters of (non-linear) polynomials to define a separation surface with a (soft) maximum distance from category related clusters of data points in an extended feature space (=input vector space). The extended feature space covered not only basic features of the input data but also powers of it.

All in all we worked directly in an multidimensional extension of the input vector space and optimized parameters describing *linear* separation interfaces there. If we had several categories instead of 2 we could use a so called "**one versus all"-strategy** to calculate 10 linear separation interfaces and determine the distance of any new data point towards the separation surfaces as a confidence measure (*score*) for a prediction. The separation with the highest score would be used to discriminate between the 10 possible solutions and choose the optimal one. Yes, working in an extended input vector space and with parameters of multiple linear separation surfaces was a bit difficult.

Actually, working with ANNs and cost functions corresponds to a more elegant way of optimizing; it starts with measuring distances in the *output vector space* of the ANN/MLP:

In the context of classification tasks (with known results for training data) a loss function provides a fictitious cost value which weighs the **deviations** (or distances) of calculated result values (of the ML-algorithm under training) from the already known correct result for training records. I.e. it measures the *errors* for the training data records in the output space. The optimization task then means to minimize the cost function and thereby minimize a kind of mean error for all input data records.

The hope is that the collection of resulting weight values allows for predictions of other unknown input data, too.

The result of an ANN/MLP for a training data record is the outcome of a complex transformation performed by the ANN. In case of an MLP the transformation of input into output data is done by the "**Feed Forward Propagation**" algorithm [FFP]. Thus a reasonably designed cost function becomes dependent on the parameters of the FFP-algorithm - predominantly on the *weights* given at the nodes of the MLP's layers. We concentrate on this type of parameter below; but note that in special ANN cases there may be additional other parameters to be varied for training and ANN optimization.

The MLP's **weights** can in principle be varied *continuously* during training. The parameter (vector) space thus can be described by multiple real value axes - one for each of the weights. The parameter space of a MLP is a multi-dimensional one with a dimension equal to or bigger than the space of input data - and of course also the result space. (That the dimension is bigger follows from the required node number in the input layer.)

With the help of a *suitable* cost function we can pose a mathematical optimization problem for the weight parameters:

Find a point in the ** weight vector space** for which the cost function gives us a minimum, which in turn corresponds to an overall minimum of the deviation distances.

A simple example for a cost function would be a sum of square values for the length of the difference vectors in the output space for all training data.

There are several things to mention:

- The result space is a multidimensional vector space (in case of MNIST a 10 dimensional one); so the distance between points there has to be defined via a mathematical
over components.**norm** - The result space in classification problems typically has a much smaller dimension "m" than the dimension "n" of the space of the input data (m < n).
- It makes almost no sense to display the cost function over the multidimensional space of input data - as a working ML-algorithm should deliver small cost values for
*all*input data. However, it makes a lot of sense to display the costs over the multi-dimensional vector space of continuous weight values. - We deal with batches of many training data records; it follows that a reasonable cost function in this case must
*combine*deviations of individual records from optimal values. This is very often done via some kind of**sum**over individual cost contributions from each training record.

In many MLP cases the cost function will be a function of the weight parameters only; this requires a reasonable node independent form of the activation functions. A loss function with a continuous dependency on all ANN parameters (as the weights) provides a multidimensional **hyperplane** in an (n+1)-dimensional space - with "n" being the number of FFP variables. The (n+1)-th dimension is for the cost values. As the the FFP-algorithm depends on a multitude of linear and non-linear operations we expect that the hyperplane-surface will have a rather complex form - with maxima and minima as well as so called saddle points.

However, if we construct the cost function cleverly the optimum values for the ANN's weights will lead to a *global minimum* of this hyperplane – which then in turn corresponds to a minimum of distances between the propagation results and the known values for the training data:

The task to find categorical separation surfaces in the vector space of input data is reformulated as an optimization task in the *cost-weight vector space:* There it means finding a (global) minimum of the cost hyperplane.

Let us assume we sit at some point on a yet unexplored hyperplane. A quite general way to find the (global) minimum of this hyperplane is to follow a path indicated by the (tangential) **gradient** vector at the local point: The gradient is vertically oriented with respect to contour lines of constant cost values on the hyperplane. It thus gives us the direction along which a maximum cost change occurs per unit change of some weights. Calculating corrections of the weights translates into following the gradient with small steps. Geometrically speaking:

We follow the direction the overall gradient points to - and translate the movement into to small components along each weight axis - which gives us the individual weight corrections. Our hope is that the overall gradient points into the direction of the global minimum. (In case of local minima or large planes of the hyperplane we would have to adopt the step size somehow.)

This is called the "**gradient descent method**". In one of the next contributions to this article series we shall see how this in turn efficiently translates into the backward propagation of errors through the network via matrix operations. Our optimization task is thus reduced to a systematic variation of the weights during gradient descent with a series of mathematical operations determining gradient components and resulting weight corrections.

The cost function absorbs complexity stemming from the large amount of all training data rather smoothly by summing up the individual contributions of training data records. Let us look a bit at the gradient: Normally we would have to calculate *partial derivatives* of ** all** cost contributions off all data sets with respect to all individual weights. For big training data sets this corresponds to a lot of mathematical operations - both matrix operations (linear algebra) and value calculations of nonlinear (activation and output) functions.

What happens if we took not all data records but concentrated on the contributions of selected input data, only? And corrected afterwards again for another disjunctive set of selected data points? I.e. what if we calculated the full required correction only piece-wise for different collections (mini-batches) of input data records?

Then the reduced gradient components would guide us into a direction on the hyperplane which deviates from the overall gradients direction. Taking the next data record would correct this movement a bit into another direction again. If we perform gradient correction for *batches* of different data records or in the extreme case for individual records we would move somewhat erratically around the overall gradient's direction; we speak of a "*stochastic* gradient descent" [SGD].

The erratic movement of SGD helps to overcome local overall minima. But all in all it may take more steps to come to a global minimum or at least close to it - as the a stochastic movement may never converge into the overall minimum's point in the weight space - but hop instead around it.

The question of how many input data we include in the cost function determining one single weight correction step during an epoch leads to the choice between the following cases:

- stochastic gradient descent (sequence of weight corrections during an epoch - each based on just
*one*training data record at a time and for all weights), - full batch gradient descent (one weight correction per epoch - based on
*all*training data records and for all weights), - mini-batch gradient descent (sequence of weight corrections during an epoch - each based on a
*batch*of multiple training data records and for all weights).

A stochastic or mini-bath based gradient descent may mean much *faster* corrections in terms of a reduced number of (vectorized) mathematical operations and CPU consumption - at least at the beginning of the descent. The CPU time of the training process for large amounts of input data may actually be reduced by factors!

In the case of mini-batches we can, therefore, optimize the performance by varying the mini-batch size. The required matrix operations can be performed vectorized over all data records of the batch; i.e. the operations can be performed "in parallel". Fortunately, we do not need to care about the necessary CPU register handling whilst coding - optimized libraries will take care of this. As we have seen already in this blog, also threading for a reasonable amount of CPU cores may influence the performance on a specific system a lot.

For our Python class we will therefore provide parameters for the size of a mini-batch - and adapt both the calculation of cost-contributions and respective weight corrections accordingly.

Note that we do not only hope for that the weights determined by gradient descent provide reasonable result values for the training data but also for any other data later on provided to the ANN/MLP. Solving the optimization problem in the end must provide reliable and complex separation surfaces in the multidimensional input vector space (for MNIST with a dimension of n=784). The mathematical equivalence of the problem of finding separation surfaces in the input vector space to the optimization problem in the result space can be proven for regression problems. (Actually, I do not know whether a mathematical equivalence has been proven for general problems. So, for some ML classification tasks gradient descent may not work sufficiently well.)

Cost functions should be designed carefully. A "cost function" must have certain properties for the so called "gradient descent method" to work successfully:

- For convenience the global extremum should be a minimum.
- The cost function must be continuous and differentiable with respect to the ANN's weights.
- The requirement of differentiability translates back to the requirement of differentiable activation and output functions - as we shall see in detail in a later article.
- It should expose a basic convex form in the surroundings of the global minimum (second partial derivatives > 0).
- The "cost function" must have certain properties for making use of an
way to calculate gradients, i.e. partial derivatives. We shall see that some reasonable cost functions turn this task into a back propagation of errors. The efficiency comes via similar matrix operations as those used in the forward propagation algorithm.**efficient**

Besides choosing a cost function carefully also the choice of the activation function is important for the success of gradient descent. The path to global minimum on a hyperplane may also depend on the starting point (defined by the statistically chosen initial weight values) as well as on an adaptive step size (called *learning rate*).

Most Machine Learning algorithms can incorporate a variety of reasonable "cost functions. For classification tasks often the following cost functions are used:

- Categorial Cross-Entropy
- Log Loss ( = Logistic Regression Loss )
- Relative Entropy,
- Exponential Loss
- MSE (Mean Square Error)

Each of these functions is more or less appropriate for a specific type of classification problem. See the literature for more information on each of these cost functions.

In our code for MNIST-problem we will only include two of these functions as a starting point - **Log Loss** and **MSE**. MSE is e.g. used by T. Rashid in his book (see section Links) on building an MLP with Python for the MNIST case. Information on the Log Loss function are provided by the book of Rashka and the book of Geron; see the references in the section "Links" below.

The training of an ANN - i.e. the optimization of weights - does not require the explicit calculation of cost values. The reason for this is of course that gradient descent first of all works with partial derivatives with respect to weights. To calculate them we must use the chain rule with respect to the activation function, the output of lower layers and so on. But the cost values themselves are nowhere required. As a consequence in all of the book of T. Rashid on "Make your Own Neural network" the calculation of costs is never encoded.

Nevertheless, in the next article of this series we shall discuss the code for cost calculations of mini-batches. The reason for this is that we can use the cost values to study the *progress* of training and the *convergence* into a minimum: The change of total "costs" provides a way to control and watch the success of training through its epochs.

The concept of a cost function is central to MLPs and classification tasks: Classification means to separate the input data into categories. The task to find categorical separation surfaces in the vector space of input data is reformulated as an optimization task. This in turn requires us to find a minimum of the cost/loss hyperplane over the multidimensional space of potential weight-parameters. Calculating corrections of the weights during following a gradient guided path to a minimum in turn efficiently translates into the backward propagation of errors through the network via matrix operations.

https://www.python-course.eu/matrix_arithmetic.php

**Gradient descent and cost functions**

towardsdatascience.com understanding-the-mathematics-behind-gradient-descent-dde5dc9be06e

ml-cheatsheet readthedocs - gradient-descent.html

page.mi.fu-berlin.de

neural chapter K7.pdf

**Regularization**

chunml.github.io tutorial on Regularization/

**Books**

"Neuronale Netze selbst programmieren", Tariq Rashid, 2017, O'Reilly Media Inc. + dpunkt.verlag GmbH

"Machine Learning mit SciKit-Learn & TensorFlow, Aurelien Geron, 2018, O'Reilly Media Inc. + dpunkt.verlag GmbH

"Python machine Learning", Seb. Raschka, 2016, Packt Publishing, Birmingham, UK

"Machine Learning mit Sckit-Learn & TensorFlow", A. Geron, 2018, O'REILLY, dpunkt.verlag GmbH, Heidelberg, Deutschland

Whenever a full upgrade of the server is required I, therefore, first test it on *copies* of the KVM host installation and each KVM instance. (The "dd" command is of good service during these tests.) One experiences some unwelcome surprises from time to time - and then you may need a quick restauration of a workings system.

When I switched everything to Opensuse Leap 15.1 some days ago I stumbled once again across small problems. It is interesting that one of the problems had to do with SSSD - again.

Some time ago 1 described a problem with PAM control files for **imap** and **smtp** services on the mail server when I upgraded to Leap 15.0. See:

Mail-server-upgrade to Opensuse Leap 15 – and some hours with authentication trouble

The PAM files included directives for SSSD. The file were unfortunately replaced (without backups) during upgrade from OS 42.3 to Leap 15.0. This hampered all authentication of mail clients via authentication requests from the imap and smtp services to the LDAP system. The cause of the resulting problems at the side of the email clients, namely authetication trouble, was not easy to identify.

This time I ran once again into authentication trouble - and suspected some mess with the PAM files again. Yet, this was not the case - the PAM files were all intact and correct. (SuSE learns!) However, after an hour of testing I saw that the SSSD service did not what it should. Checking the status of the service with "systemctl status sssd.service" I got a final status line saying **"Backend is offline"**.

What did this mean? I had no real clue. You naturally assume that LDAP would be my backend in my server configuration; this is reflected in the file /etc/sssd/sssd.conf:

[sssd] config_file_version = 2 services = nss,pam domains = default [nss] filter_groups = root filter_users = root [pam] [domain/default] ldap_uri = ldap://myldap.mydomain.de ldap_search_base = dc=mydc,dc=de ldap_schema = rfc2307bis id_provider = ldap ldap_user_uuid = entryuuid ldap_group_uuid = entryuuid ldap_id_use_start_tls = True enumerate = True cache_credentials = False ldap_tls_cacertdir = /etc/ssl/certs ldap_tls_cacert = /etc/ssl/certs/mydomainCA.pem chpass_provider = ldap auth_provider = ldap

I checked - the LDAP service was active in its KVM machine. Of course, NSS must also be working for SSSD to become functional. No problem there. I checked whether the LDAP service could be reached through the firewalls of the different KVM instances and their hosts. Yes, this worked, too. So, what the hack was wrong?

Eventually, I found some interesting contribution in a Fedora mailing list: See here. What if the problem had its origin really in some systemd glitch? Wouldn't be the first time.

So, I first made a copy of the original file "/usr/lib/systemd/system/sssd.service" and after that tried a modification of the original file linked by "sss.service" in "/etc/systemd/system/multi-user.target.wants". I simply added a line "After=network.service" to guarantee a full network setup before sssd was started.

[Unit] Description=System Security Services Daemon # SSSD must be running before we permit user sessions Before=systemd-user-sessions.service nss-user-lookup.target Wants=nss-user-lookup.target After=network.service [Service] Environment=DEBUG_LOGGER=--logger=files EnvironmentFile=-/etc/sysconfig/sssd ExecStart=/usr/sbin/sssd -i ${DEBUG_LOGGER} Type=notify NotifyAccess=main PIDFile=/var/run/sssd.pid [Install] WantedBy=multi-user.target

And guess what? This was successful! The reason being that at the point in time when the sssd.service starts name resolution (i.e. the evaluation of resolv.conf and access to DNS-servers ) may not yet be guaranteed!

**Hint:**

Note that there may be multiple reasons for such a delay; one you could think of is a firewall which is started at some point and requires time to establish all rules. Your server may not get access to any of the defined DNS-servers up to the point where the firewalls rules are working. Then, depending on when exactly you start your firewall service, you may have to use a different "After"-rule than mine.

**Important point: **

You should not permanently change the files in "/usr/lib/systemd". So, after such a test as described you should restore the original systemd file for a specific service in "/usr/lib/systemd/system/" with all its attributes! The correct mechanism to add modifications to systemd service configuration files is e.g. described here "askubuntu.com : how-do-i-override-or-configure-systemd-services".

So, in my case we need to execute "systemctl edit sssd" on the command line and then (in the editor window) add the lines

[Unit]

After=network.service

This leads to the creation of a directory "/etc/systemd/system/sssd.service" with a file "override.conf" which contains the required entries for service startup modification.

One of my anti-virus engines integrated with amavis is clamav. More precisely the daemon based variant, i.e. the "*clamd*" service. However, when I tested amavis for mail scanning I saw that it used to job instances of "clamscan" instead of "clamdscan". The impact of Amavis' using two parallel clamscan threads was an almost 100% CPU utilization for some time.

It took me a while to find out what the cause of this problem was: *clamd* requires time to start up. And due to whatever reasons this time is now a bit bigger on my mail system than the standard timeout of 90 secs systemd provides. This can be compensated by "systemctl edit sssd" and adding lines as

[Service]

TimeoutSec=3min

After this change clamd ran again as usual. Note however that clamav does not provide sufficient protection on professional mail servers, especially when your email clients are based on a Windows installations. Then you need at least one more advanced (and probably costly) antivirus solution.

how-to-troubleshoot-backend.html

fedora archive contribution

www.clearos.com community : clamd-start-up-times-out

unix.stackexchange.com : how-to-change-systemd-service-timeout-value

At some point in time during last week the hosting-provider changed his security policies on his (Norwegian) Apache servers. The provider seems to have at least changed settings of the "mod_security" module - and thereby started to eliminate old browsers by some rules. (Maybe they even introduced the use of the mod_security module for the first time ?). To implement mod-security with a reasonable set of rules basically is a good measure.

However, the effect was that our customer got a **406 error** whenever he tried to access his web-site with his Firefox browser. The "406 Not Acceptable" message indicates that a web server cannot or will not (due to some rules) satisfy some conditions in the HTTP GET- or POST-request. Our customer uses the latest version of Firefox. He tested whether he got something similar on a test installation of one of our hosted servers in Germany. Of course not.

A subsequent complaint of our customer was answered by his provider; the answer in a direct translation says:

Contact the Firefox technicians or use Chrome!

Very funny! Our customer asked us for help. We tested the web-servers response with multiple browsers from Linux and Windows desktops. The problem seemed to exist only for Firefox and only on desktop systems. This already indicated a strange server reaction to the HTTP "User-Agent" string.

But this was only part of the strange experience our customer got due to new security measures. In addition his provider enforced the usage of an Apache htaccess password (Basic HTTP user authentication) for all users who maintained their own WordPress installation on the hoster's web-servers. Our customer suddenly needed to provide a UserId and a password to get access to his WordPress installation's "wp-admin"-directory. We found out about this intentionally imposed restriction by having a look at the public web site of the provider. There, in a side column, we found a message regarding the new restriction. Customers were asked there to contact the hoster's specialists for required credentials. Our customer had not been directly informed by the provider about this new policy. So, we just sent the provider a mail and asked him to give us the authentication data to the admin folder of our customer's WP-installation. We got it one day later via email.

In my opinion these procedures indicate some mess we are facing with improperly handled IT-security activities these days.

**Comment 1:** It is, of course, OK to enforce a HTTP password access to directories of a web server. But this is only an effective protection measure if the provider at the same time enforces general TLS/SSL encryption for the access to the hosted web-sites. Otherwise the password would be sent in clear text over the Internet. However, you can still work with a WordPress installation or other CMS-installations on the provider's web-servers without any SSL certificate. Our customer has a SSL-certificate - but he had to pay for it. Here business interests of the provider obviously collide with real security procedures.

**Comment 2:** Personally, I regard it as a major mistake to set a *common* UserID and a fixed permanent password for customers and send these credentials to a web-admin via an *unencrypted* email. Ironically enough the provider asked the receiver in the mail to take note of the password and then to destroy the mail. So, mails on the customers mail system are dangerous, but the transfer of an unencrypted mail over at least partially unencrypted Internet lines is not?

Hey, we are not talking about a one time password here - but *permanent* credentials set and enforced by the provider. The CPanel admin tool offered by the hosting provider does NOT allow for the change of the fixed htaccess password set by the provider's admins.

Furthermore, why announce this policy on a public website and not inform the customers via a secure channel? Next question: How did they know that we were authorized to request the access data without contacting our customer first ???

Also interesting was the analysis of the Firefox problem. We can demonstrate the effect on the provider's own website. Here is what you presently (18.10.2019) get when opening the homepage of the provider with Firefox from a Linux desktop:

And here is what you get when you manipulate the User-Agent string a bit:

The blue rectangles have been added not to directly show the provider's name. Note the 406 error message in the FF developer tool at the bottom!

Well, well ... Our customer got the following when opening his own web-page:

Some analysis showed that we get a correct display of the web-site on the same browser if we manipulated the HTTP User-Agent-string for Firefox a bit. One way to do this is offered by the web developer tools of Firefox. However, there are also good plugins to fake the User-Agent string.

The next question was: What part in the User-Agent-string reacted the provider's Apache servers allergic to?

The standard User-Agent-string of Firefox in a HTTP-GET- or POST-request is defined to have the following structure:

Mozilla/5.0 (platform; rv:geckoversion) Gecko/geckotrail Firefox/firefoxversion

This can be learned from related explanations of mozilla.org:

Firefox User Agent string

"geckotrail" can be an indication of a version or a date. However - quotation:

On Desktop, geckotrail is the fixed string "20100101"

And when we check the User-Agent-string for Firefox on e.g. a Linux desktop we indeed get:

Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20190110 Firefox/68.0

Both were accepted by the provider's servers with a HTTP status code of 200 - and a complete correct web-page display.

To enable the customer to work with his FF until the provider corrects his server settings we recommended to install a plugin which allows for a manipulation of the User-Agent string. We in addition informed the provider about our findings.

What a mess a provider can produce with improper security measures! The only conclusion I get out of all this is: Security awareness is good. Education of the administrators is even more important. In Norway and everywhere else ...

]]>

A simple program for an ANN to cover the Mnist dataset – I

A simple program for an ANN to cover the Mnist dataset – II

I defined some code elements, which controlled the layers, their node numbers and built weight matrices. We succeeded in setting random initial values for the weights. This enables us to work on the **forward propagation algorithm** in this article.

As we later on need to define methods which cover "training epochs" and the handling of "mini-batches" comprising a defined number of training records we extend our set of methods already now by

An "**epoch**" characterizes a full training step comprising

- propagation, cost and derivative analysis and weight correction of all data records or samples in the set of training data, i.e. a loop over
**all**mini-batches.

Handling of a **mini-batch** comprises

- (vectorized) propagation of all training records of a mini-batch,
- cumulative cost analysis for all training records of a batch,
- cumulative, averaged gradient evaluation of the cost function by back-propagation of errors and summation over all records of a training batch,
- weight corrections for nodes in all layers based on averaged gradients over all records of the batch data.

Vectorized propagation means that we propagate all training records of a batch in parallel. This will be handled by Numpy matrix multiplications (see below).

We shall see in a forthcoming that we can also cover the cumulative gradient calculation over all batch samples by matrix-multiplications where we shift the central multiplication and summation operations to appropriate rows and columns.

However, we do not care for details of training epochs and complete batch-operations at the moment. We use the two methods "_fit()" and "_handle_mini_batch()" in this article only as envelopes to trigger the epoch loop and the matrix operations for propagation of a batch, respectively.

We change and extend our "__init_"-function of class MyANN a bit:

def __init__(self, my_data_set = "mnist", n_hidden_layers = 1, ay_nodes_layers = [0, 100, 0], # array which should have as much elements as n_hidden + 2 n_nodes_layer_out = 10, # expected number of nodes in output layer my_activation_function = "sigmoid", my_out_function = "sigmoid", n_size_mini_batch = 50, # number of data elements in a mini-batch n_epochs = 1, n_max_batches = -1, # number of mini-batches to use during epochs - > 0 only for testing # a negative value uses all mini-batches vect_mode = 'cols', figs_x1=12.0, figs_x2=8.0, legend_loc='upper right', b_print_test_data = True ): ''' Initialization of MyANN Input: data_set: type of dataset; so far only the "mnist", "mnist_784" datsets are known We use this information to prepare the input data and learn about the feature dimension. This info is used in preparing the size of the input layer. n_hidden_layers = number of hidden layers => between input layer 0 and output layer n ay_nodes_layers = [0, 100, 0 ] : We set the number of nodes in input layer_0 and the output_layer to zero Will be set to real number afterwards by infos from the input dataset. All other numbers are used for the node numbers of the hidden layers. n_nodes_out_layer = expected number of nodes in the output layer (is checked); this number corresponds to the number of categories NC = number of labels to be distinguished my_activation_function : name of the activation function to use my_out_function : name of the "activation" function of the last layer which produces the output values n_size_mini_batch : Number of elements/samples in a mini-batch of training data The number of mini-batches will be calculated from this n_epochs : number of epochs to calculate during training n_max_batches : > 0: maximum of mini-batches to use during training < 0: use all mini-batches vect_mode: Are 1-dim data arrays (vctors) ordered by columns or rows ? figs_x1=12.0, figs_x2=8.0 : Standard sizing of plots , legend_loc='upper right': Position of legends in the plots b_print_test_data: Boolean variable to control the print out of some tests data ''' # Array (Python list) of known input data sets self._input_data_sets = ["mnist", "mnist_784", "mnist_keras"] self._my_data_set = my_data_set # X, y, X_train, y_train, X_test, y_test # will be set by analyze_input_data # X: Input array (2D) - at present status of MNIST image data, only. # y: result (=classification data) [digits represent categories in the case of Mnist] self._X = None self._X_train = None self._X_test = None self._y = None self._y_train = None self._y_test = None # relevant dimensions # from input data information; will be set in handle_input_data() self._dim_sets = 0 self._dim_features = 0 self._n_labels = 0 # number of unique labels - will be extracted from y-data # Img sizes self._dim_img = 0 # should be sqrt(dim_features) - we assume square like images self._img_h = 0 self._img_w = 0 # Layers # ------ # number of hidden layers self._n_hidden_layers = n_hidden_layers # Number of total layers self._n_total_layers = 2 + self._n_hidden_layers # Nodes for hidden layers self._ay_nodes_layers = np.array(ay_nodes_layers) # Number of nodes in output layer - will be checked against information from target arrays self._n_nodes_layer_out = n_nodes_layer_out # Weights # -------- # empty List for all weight-matrices for all layer-connections # Numbering : # w[0] contains the weight matrix which connects layer 0 (input layer ) to hidden layer 1 # w[1] contains the weight matrix which connects layer 1 (input layer ) to (hidden?) layer 2 self._ay_w = [] # --- New ----- # Two lists for output of propagation # __ay_x_in : input data of mini-batches on the different layers; the contents is calculated by the propagation algorithm # __ay_a_out : output data of the activation function; the contents is calculated by the propagation algorithm # Note that the elements of these lists are numpy arrays self.__ay_X_in = [] self.__ay_a_out = [] # Known Randomizer methods ( 0: np.random.randint, 1: np.random.uniform ) # ------------------ self.__ay_known_randomizers = [0, 1] # Types of activation functions and output functions # ------------------ self.__ay_activation_functions = ["sigmoid"] # later also relu self.__ay_output_functions = ["sigmoid"] # later also softmax # the following dictionaries will be used for indirect function calls self.__d_activation_funcs = { 'sigmoid': self._sigmoid, 'relu': self._relu } self.__d_output_funcs = { 'sigmoid': self._sigmoid, 'softmax': self._softmax } # The following variables will later be set by _check_and set_activation_and_out_functions() self._my_act_func = my_activation_function self._my_out_func = my_out_function self._act_func = None self._out_func = None # number of data samples in a mini-batch self._n_size_mini_batch = n_size_mini_batch self._n_mini_batches = None # will be determined by _get_number_of_mini_batches() # number of epochs self._n_epochs = n_epochs # maximum number of batches to handle (<0 => all!) self._n_max_batches = n_max_batches # print some test data self._b_print_test_data = b_print_test_data # Plot handling # -------------- # Alternatives to resize plots # 1: just resize figure 2: resize plus create subplots() [figure + axes] self._plot_resize_alternative = 1 # Plot-sizing self._figs_x1 = figs_x1 self._figs_x2 = figs_x2 self._fig = None self._ax = None # alternative 2 does resizing and (!) subplots() self.initiate_and_resize_plot(self._plot_resize_alternative) # *********** # operations # *********** # check and handle input data self._handle_input_data() # set the ANN structure self._set_ANN_structure() # Prepare epoch and batch-handling - sets mini-batch index array, too self._prepare_epochs_and_batches() # perform training start_c = time.perf_counter() self._fit(b_print=True, b_measure_batch_time=False) end_c = time.perf_counter() print('\n\n ------') print('Total training Time_CPU: ', end_c - start_c) print("\nStopping program regularily") sys.exit()

Readers who have followed me so far will recognize that I renamed the parameter "n_mini_batch" to "n_size_mini_batch" to indicate its purpose a bit more clearly. We shall derive the number of required mini-batches form the value of this parameter.

I have added two new parameters:

- n_epochs = 1
- n_max_batches = -1

"n_epochs" will later receive the user's setting for the number of epochs to follow during training. "n_max_Batches" allows us to limit the number of mini-batches to analyze during tests.

The kind reader will also have noticed that I encapsulated the series of operations for preparing the weight-matrices for the ANN in a new method "**_set_ANN_structure()**"

'''-- Main method to set ANN structure --''' def _set_ANN_structure(self): # check consistency of the node-number list with the number of hidden layers (n_hidden) self._check_layer_and_node_numbers() # set node numbers for the input layer and the output layer self._set_nodes_for_input_output_layers() self._show_node_numbers() # create the weight matrix between input and first hidden layer self._create_WM_Input() # create weight matrices between the hidden layers and between tha last hidden and the output layer self._create_WM_Hidden() # check and set activation functions self._check_and_set_activation_and_out_functions() return None

The called functions have remained unchanged in comparison to the last article.

We can safely assume that some steps must be performed to prepare epoch- and batch handling. We, therefore, introduced a new function "_prepare_epochs_and_batches()". For the time being this method only calculates the number of mini-batches from the input parameter "n_size_mini_batch". We use the Numpy-function "array_split()" to split the full range of input data into batches.

''' -- Main Method to prepare epochs -- ''' def _prepare_epochs_and_batches(self): # set number of mini-batches and array with indices of input data sets belonging to a batch self._set_mini_batches() return None ## ''' -- Method to set the number of batches based on given batch size -- ''' def _set_mini_batches(self, variant=0): # number of mini-batches? self._n_mini_batches = math.ceil( self._y_train.shape[0] / self._n_size_mini_batch ) print("num of mini_batches = " + str(self._n_mini_batches)) # create list of arrays with indices of batch elements self._ay_mini_batches = np.array_split( range(self._y_train.shape[0]), self._n_mini_batches ) print("\nnumber of batches : " + str(len(self._ay_mini_batches))) print("length of first batch : " + str(len(self._ay_mini_batches[0]))) print("length of last batch : " + str(len(self._ay_mini_batches[self._n_mini_batches - 1]) )) return None

Note that the approach may lead to smaller batch sizes than requested by the user.

array_split() cuts out a series of sub-arrays of indices of the training data. I.e., "_ay_mini_batches" becomes a 1-dim array, whose elements are 1-dim arrays, too. Each of the latter contains a collection of indices for selected samples of the training data - namely the indices for those samples which shall be used in the related mini-batch.

For the time being method "_fit()" is used for looping over the number of epochs and the number of batches:

''' -- Method to set the number of batches based on given batch size -- ''' def _fit(self, b_print = False, b_measure_batch_time = False): # range of epochs ay_idx_epochs = range(0, self._n_epochs) # limit the number of mini-batches n_max_batches = min(self._n_max_batches, self._n_mini_batches) ay_idx_batches = range(0, n_max_batches) if (b_print): print("\nnumber of epochs = " + str(len(ay_idx_epochs))) print("max number of batches = " + str(len(ay_idx_batches))) # looping over epochs for idxe in ay_idx_epochs: if (b_print): print("\n ---------") print("\nStarting epoch " + str(idxe+1)) # loop over mini-batches for idxb in ay_idx_batches: if (b_print): print("\n ---------") print("\n Dealing with mini-batch " + str(idxb+1)) if b_measure_batch_time: start_0 = time.perf_counter() # deal with a mini-batch self._handle_mini_batch(num_batch = idxb, b_print_y_vals = False, b_print = b_print) if b_measure_batch_time: end_0 = time.perf_counter() print('Time_CPU for batch ' + str(idxb+1), end_0 - start_0) return None #

We limit the number of mini_batches. The double-loop-structure is typical. We tell function "_handle_mini_batch(num_batch = idxb,...)" which batch it should handle.

We shall build up the operations for batch handling over several articles. In this article we clarify the operations for feed forward propagation, only. Nevertheless, we have to think a step ahead: Gradient calculation will require that we keep the results of propagation layer-wise somewhere.

As the number of layers can be set by the user of the class we save the propagation results in two Python lists:

- ay_Z_in_layer = []
- ay_A_out_layer = []

The Z-values define a *collection* of input vectors which we normally get by a matrix multiplication from output data of the last layer and a suitable weight-matrix. The "collection" is our mini-batch. So, "ay_Z_in_layer" actually is a 2-dimensional array.

For the ANN's input layer "L0", however, we just fill in an excerpt of the "_X"-array-data corresponding to the present mini-batch.

Array "ay_A_out_layer[n]" contains the results of * activation function* applied onto the elements of "ay_Z_in_layer[n]" of Layer "Ln". (In addition we shall add a value for a bias neutron; see below).

Our method looks like:

''' -- Method to deal with a batch -- ''' def _handle_mini_batch(self, num_batch = 0, b_print_y_vals = False, b_print = False): ''' For each batch we keep the input data array Z and the output data A (output of activation function!) for all layers in Python lists We can use this as input variables in function calls - mutable variables are handled by reference values ! We receive the A and Z data from propagation functions and proceed them to cost and gradient calculation functions As an initial step we define the Python lists ay_Z_in_layer and ay_A_out_layer and fill in the first input elements for layer L0 ''' ay_Z_in_layer = [] # Input vector in layer L0; result of a matrix operation in L1,... ay_A_out_layer = [] # Result of activation function #print("num_batch = " + str(num_batch)) #print("len of ay_mini_batches = " + str(len(self._ay_mini_batches))) #print("_ay_mini_batches[0] = ") #print(self._ay_mini_batches[num_batch]) # Step 1: Special treatment of the ANN's input Layer L0 # Layer L0: Fill in the input vector for the ANN's input layer L0 ay_Z_in_layer.append( self._X_train[(self._ay_mini_batches[num_batch])] ) # numpy arrays can be indexed by an array of integers #print("\nPropagation : Shape of X_in = ay_Z_in_layer = " + str(ay_Z_in_layer[0].shape)) if b_print_y_vals: print("\n idx, expected y_value of Layer L0-input :") for idx in self._ay_mini_batches[num_batch]: print(str(idx) + ', ' + str(self._y_train[idx]) ) # Step 2: Layer L0: We need to transpose the data of the input layer ay_Z_in_0T = ay_Z_in_layer[0].T ay_Z_in_layer[0] = ay_Z_in_0T # Step 3: Call the forward propagation method for the mini-batch data samples self._fw_propagation(ay_Z_in = ay_Z_in_layer, ay_A_out = ay_A_out_layer, b_print = b_print) if b_print: # index range of layers ilayer = range(0, self._n_total_layers) print("\n ---- ") print("\nAfter propagation through all layers: ") for il in ilayer: print("Shape of Z_in of layer L" + str(il) + " = " + str(ay_Z_in_layer[il].shape)) print("Shape of A_out of layer L" + str(il) + " = " + str(ay_A_out_layer[il].shape)) # Step 4: To be done: cost calculation for the batch # Step 5: To be done: gradient calculation via back propagation of errors # Step 6: Adjustment of weights # try to accelerate garbage handling if len(ay_Z_in_layer) > 0: del ay_Z_in_layer if len(ay_A_out_layer) > 0: del ay_A_out_layer return None

Why do we need to transpose the Z-matrix for layer L0?

This has to do with the required matrix multiplication of the forward propagation (see below).

The function "_fw_propagation()" performs the forward propagation of a mini-batch through all of the ANN's layers - and saves the results in the lists defined above.

**Important note:** We transfer our lists (mutable Python objects) to "_fw_propagation()"! This has the effect that the array of the corresponding values is referenced from within "_fw_propagation()"; therefore will any elements added to the lists also be available outside the called function! Therefore we can use the calculated results also in further functions for e.g. gradient calculations which will later be called from within "_handle_mini_batch()".

Note also that this function leaves room for optimization: It is e.g. unnecessary to prepare ay_Z_in_0T again and again for each epoch. We will transfer the related steps to "_prepare_epochs_and_batches()" later on.

In one of my last articles in this blog I already showed how one can use Numpy's Linear Algebra features to cover propagation calculations required for information transport between two adjacent layers of a feed forward "Artificial Neural Network" [ANN]:

Numpy matrix multiplication for layers of simple feed forward ANNs

The result was that we can cover propagation between neighboring layers by a vectorized multiplication of two 2-dim matrices - one containing the weights and the other vectors of feature data for all mini-batch samples. In the named article I discussed in detail which rows and columns are used for the central multiplication with weights and summations - and that the last dimension of the input array should account for the mini-batch samples. This requires the transpose operation on the input array of Layer L0. All other intermediate layer results (arrays) do already get the right form for vectorizing.

"_fw_propagation()" takes the following form:

''' -- Method to handle FW propagation for a mini-batch --''' def _fw_propagation(self, ay_Z_in, ay_A_out, b_print= False): b_internal_timing = False # index range of layers ilayer = range(0, self._n_total_layers-1) # propagation loop for il in ilayer: if b_internal_timing: start_0 = time.perf_counter() if b_print: print("\nStarting propagation between L" + str(il) + " and L" + str(il+1)) print("Shape of Z_in of layer L" + str(il) + " (without bias) = " + str(ay_Z_in[il].shape)) # Step 1: Take input of last layer and apply activation function if il == 0: A_out_il = ay_Z_in[il] # L0: activation function is identity else: A_out_il = self._act_func( ay_Z_in[il] ) # use real activation function # Step 2: Add bias node A_out_il = self._add_bias_neuron_to_layer(A_out_il, 'row') # save in array ay_A_out.append(A_out_il) if b_print: print("Shape of A_out of layer L" + str(il) + " (with bias) = " + str(ay_A_out[il].shape)) # Step 3: Propagate by matrix operation Z_in_ilp1 = np.dot(self._ay_w[il], A_out_il) ay_Z_in.append(Z_in_ilp1) if b_internal_timing: end_0 = time.perf_counter() print('Time_CPU for layer propagation L' + str(il) + ' to L' + str(il+1), end_0 - start_0) # treatment of the last layer il = il + 1 if b_print: print("\nShape of Z_in of layer L" + str(il) + " = " + str(ay_Z_in[il].shape)) A_out_il = self._out_func( ay_Z_in[il] ) # use the output function ay_A_out.append(A_out_il) if b_print: print("Shape of A_out of last layer L" + str(il) + " = " + str(ay_A_out[il].shape)) return None #

First we set a range for a loop over the layers. Then we apply the activation function. In "step 2" we add a bias-node to the layer - compare this to the number of weights, which we used during the initialization of the weight matrices in the last article. In step 3 we apply the vectorized Numpy-matrix multiplication (np.dot-operation). Note that this is working for layer L0, too, because we already transposed the input array for this layer in "_handle_mini_batch()"!

Note that we need some special treatment for the last layer: here we call the out-function to get result values. And, of course, we do not add a bias neuron!

It remains to have a look at the function "_add_bias_neuron_to_layer(A_out_il, 'row')", which extends the A-data by a constant value of "1" for a bias neuron. The function is pretty simple:

''' Method to add values for a bias neuron to A_out ''' def _add_bias_neuron_to_layer(self, A, how='column'): if how == 'column': A_new = np.ones((A.shape[0], A.shape[1]+1)) A_new[:, 1:] = A elif how == 'row': A_new = np.ones((A.shape[0]+1, A.shape[1])) A_new[1:, :] = A return A_new

We let the program run in a Jupyter cell with the following parameters:

This produces the following output ( I omitted the output for initialization):

Input data for dataset mnist_keras : Original shape of X_train = (60000, 28, 28) Original Shape of y_train = (60000,) Original shape of X_test = (10000, 28, 28) Original Shape of y_test = (10000,) Final input data for dataset mnist_keras : Shape of X_train = (60000, 784) Shape of y_train = (60000,) Shape of X_test = (10000, 784) Shape of y_test = (10000,) We have 60000 data sets for training Feature dimension is 784 (= 28x28) The number of labels is 10 Shape of y_train = (60000,) Shape of ay_onehot = (10, 60000) Values of the enumerate structure for the first 12 elements : (0, 6) (1, 8) (2, 4) (3, 8) (4, 6) (5, 5) (6, 9) (7, 1) (8, 3) (9, 8) (10, 9) (11, 0) Labels for the first 12 datasets: Shape of ay_onehot = (10, 60000) [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.] [0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.] [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.] [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.] [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.] [1. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.] [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] [0. 1. 0. 1. 0. 0. 0. 0. 0. 1. 0. 0.] [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 1. 0.]] The node numbers for the 4 layers are : [784 100 50 10] Shape of weight matrix between layers 0 and 1 (100, 785) Creating weight matrix for layer 1 to layer 2 Shape of weight matrix between layers 1 and 2 = (50, 101) Creating weight matrix for layer 2 to layer 3 Shape of weight matrix between layers 2 and 3 = (10, 51) The activation function of the standard neurons was defined as "sigmoid" The activation function gives for z=2.0: 0.8807970779778823 The output function of the neurons in the output layer was defined as "sigmoid" The output function gives for z=2.0: 0.8807970779778823 num of mini_batches = 300 number of batches : 300 length of first batch : 200 length of last batch : 200 number of epochs = 1 max number of batches = 2 --------- Starting epoch 1 --------- Dealing with mini-batch 1 Starting propagation between L0 and L1 Shape of Z_in of layer L0 (without bias) = (784, 200) Shape of A_out of layer L0 (with bias) = (785, 200) Starting propagation between L1 and L2 Shape of Z_in of layer L1 (without bias) = (100, 200) Shape of A_out of layer L1 (with bias) = (101, 200) Starting propagation between L2 and L3 Shape of Z_in of layer L2 (without bias) = (50, 200) Shape of A_out of layer L2 (with bias) = (51, 200) Shape of Z_in of layer L3 = (10, 200) Shape of A_out of last layer L3 = (10, 200) ---- After propagation through all layers: Shape of Z_in of layer L0 = (784, 200) Shape of A_out of layer L0 = (785, 200) Shape of Z_in of layer L1 = (100, 200) Shape of A_out of layer L1 = (101, 200) Shape of Z_in of layer L2 = (50, 200) Shape of A_out of layer L2 = (51, 200) Shape of Z_in of layer L3 = (10, 200) Shape of A_out of layer L3 = (10, 200) --------- Dealing with mini-batch 2 Starting propagation between L0 and L1 Shape of Z_in of layer L0 (without bias) = (784, 200) Shape of A_out of layer L0 (with bias) = (785, 200) Starting propagation between L1 and L2 Shape of Z_in of layer L1 (without bias) = (100, 200) Shape of A_out of layer L1 (with bias) = (101, 200) Starting propagation between L2 and L3 Shape of Z_in of layer L2 (without bias) = (50, 200) Shape of A_out of layer L2 (with bias) = (51, 200) Shape of Z_in of layer L3 = (10, 200) Shape of A_out of last layer L3 = (10, 200) ---- After propagation through all layers: Shape of Z_in of layer L0 = (784, 200) Shape of A_out of layer L0 = (785, 200) Shape of Z_in of layer L1 = (100, 200) Shape of A_out of layer L1 = (101, 200) Shape of Z_in of layer L2 = (50, 200) Shape of A_out of layer L2 = (51, 200) Shape of Z_in of layer L3 = (10, 200) Shape of A_out of layer L3 = (10, 200) ------ Total training Time_CPU: 0.010270356000546599 Stopping program regularily stopped

We see that the dimensions of the Numpy arrays fit our expectations!

If you raise the number for batches and the number for epochs you will pretty soon realize that writing continuous output to a Jupyter cell costs CPU-time. You will also notice strange things regarding performance, multithreading and the use of the Linalg library OpenBlas on Linux system. I have discussed this extensively in a previous article in this blog:

Linux, OpenBlas and Numpy matrix multiplications – avoid using all processor cores

So, for another tests we set the following environment variable for the shell in which we start our Jupyter notebook:

**export OPENBLAS_NUM_THREADS=4**

This is appropriate for my Quad-core CPU with hyperthreading. You may choose a different parameter on your system!

We furthermore stop printing in the epoch loop by editing the call to function "_fit()":

self._fit(b_print=False, b_measure_batch_time=False)

We change our parameter setting to:

Then the last output lines become:

The node numbers for the 4 layers are : [784 100 50 10] Shape of weight matrix between layers 0 and 1 (100, 785) Creating weight matrix for layer 1 to layer 2 Shape of weight matrix between layers 1 and 2 = (50, 101) Creating weight matrix for layer 2 to layer 3 Shape of weight matrix between layers 2 and 3 = (10, 51) The activation function of the standard neurons was defined as "sigmoid" The activation function gives for z=2.0: 0.8807970779778823 The output function of the neurons in the output layer was defined as "sigmoid" The output function gives for z=2.0: 0.8807970779778823 num of mini_batches = 150 number of batches : 150 length of first batch : 400 length of last batch : 400 ------ Total training Time_CPU: 146.44446582399905 Stopping program regularily stopped

Good !

The time required to repeat this kind of forward propagation for a network with only one hidden layer with 50 neurons and 1000 epochs is around 160 secs. As backward propagation is not much more complex than forward propagation this already indicates that we should be able to train such a most simple MLP with 60000 28x28 images in less than 10 minutes on a standard CPU.

In this article we saw that coding forward propagation is a pretty straight-forward exercise with Numpy! The tricky thing is to understand the way **numpy.dot()** handles vectorizing of a matrix product and which structure of the matrices is required to get the expected numbers!

In the next article

A simple program for an ANN to cover the Mnist dataset – IV – the concept of a cost or loss function

we shall start working on cost and gradient calculation.

]]>

The problem had to do with the required vectorized matrix operations for forward propagation - in my case through an artificial neural network [ANN] with 4 layers. In a first approach I used 784, 100, 50, 10 neurons in 4 consecutive layers of the MLP. The weight matrices had corresponding dimensions.

The performance problem was caused by extensive multi-threading; it showed a strong dependency on mini-batch sizes and on basic matrix dimensions related to the neuron numbers per layer:

- For the given relatively small number of neurons per layer and for mini-batch sizes above a critical value (N > 255) OpenBlas suddenly occupied all processor cores with a 100% work load. This had a disastrous impact on performance.
- For neuron numbers as 784, 300, 140, 10 OpenBlas used all processor cores with a 100% work load right from the beginning, i.e. even for small batch sizes. With a seemingly bad performance over all batch sizes - but decreasing somewhat with large batch sizes.

This problem has been discussed elsewhere with respect to the matrix dimensions relevant for the core multiplication and summation operations - i.e. the neuron numbers per layer. However, the vectorizing aspect of matrix multiplications is interesting, too:

One can imagine that splitting the operations for multiple independent test samples is in principle ideal for multi-threading. So, using as many processor cores as possible (in my case 8) does not look like a wrong decision of OpenBlas at first.

Then I noticed that for mini-batch sizes "N" below a certain number (N < 250) the system only seemed to use up to 3-4 cores; so there remained plenty of CPU capacity left for other tasks. Performance for N < 250 was better by at least a factor of 2 compared to a situation with an only slightly bigger batch size (N ≥ 260). I got the impression that OpenBLAS under certain conditions just decides to use as many threads as possible - which no good outcome.

In the last years I sometimes had to work with optimizing multi-threaded database operations on Linux systems. I often got the impression that you have to be careful and leave some CPU resources free for other tasks and to avoid heavy context switching. In addition bottlenecks appeared due to the concurrent access of may processes to the CPU cache. (RAM limitations were an additional factor; but this should not be the case for my Python program.) Furthermore, one should not forget that Python/Numpy experiments on Jupyter notebooks require additional resources to handle the web page output and page update on the browser. And Linux itself also requires some free resources.

So, I wanted to find out whether reducing the number of threads - or available cores - for Numpy and OpenBlas would be helpful in the sense of an overall gain in performance.

All data shown below were gathered on a desktop system with some background activity due to several open browsers, clementine and pulse-audio as active audio components, an open mail client (kontact), an open LXC container, open Eclipse with Pydev and open ssh connections. Program tests were performed with the help of Jupyter notebooks. Typical background CPU consumption looks like this on Ksysguard:

Most of the consumption is due to audio. Small spikes on one CPU core due to the investigation of incoming mails were possible - but always below 20%.

One of the core ingredients to get an ANN running are matrix operations. More precisely: multiplications of 2-dim Numpy matrices (weight matrices) with input vectors. The dimensions of the weight matrices reflect the node-numbers of consecutive ANN-layers. The dimension of the input vector depends on the node number of the lower of two neighbor layers.

However, we perform such matrix operations NOT sequentially sample for sample of a collection of training data - we do it vectorized for so called mini-batches consisting of between 50 and 600000 individual samples of training data. Instead of operating with a matrix on just one feature vector of one training sample we use matrix multiplications whereby the second matrix often comprises many vectors of data samples.

I have described such multiplications already in a previous blog article; see Numpy matrix multiplication for layers of simple feed forward ANNs.

In the most simple case of an MLP with e.g.

- an input layer of 784 nodes (suitable for the MNIST dataset),
- one hidden layer with 100 nodes,
- another hidden layer with 50 nodes
- and an output layer of 10 nodes (fitting again the MNIST dataset)

and "mini"-batches of different sizes (between 20 and 20000). An input vector to the first hidden layer has a dimension of 100, so the weight matrix creating this input vector from the "output" of the MLP's input layer has a shape of 784x100. Multiplication and summation in this case is done over the dimension covering 784 features. When we work with mini-batches we want to do these operations in parallel for as many elements of a mini-batch as possible.

All in all we have to perform 3 matrix operations

(784x100) matrix on (784)-vector, (100x50) matrix on (100)-vector, (50x10) matrix on (50) vector

on our example ANN with 4 layers. However, we collect the data for N mini-batch samples in an array. This leads to Numpy matrix multiplications of the kind

(784x100) matrix on an (784, N)-array, (100x50) matrix on an (100, N)-array, (50x10) matrix on an (50, N)-array.

Thus, we deal with matrix multiplications of two 2-dim matrices. Linear algebra libraries should optimize such operations for different kinds of processors.

On my Linux system Python/Numpy use the openblas-library. This is confirmed by the output of command "np.__config__.show()":

openblas_info: libraries = ['openblas', 'openblas'] library_dirs = ['/usr/local/lib'] language = c define_macros = [('HAVE_CBLAS', None)] blas_opt_info: libraries = ['openblas', 'openblas'] library_dirs = ['/usr/local/lib'] language = c define_macros = [('HAVE_CBLAS', None)] openblas_lapack_info: libraries = ['openblas', 'openblas'] library_dirs = ['/usr/local/lib'] language = c define_macros = [('HAVE_CBLAS', None)] lapack_opt_info: libraries = ['openblas', 'openblas'] library_dirs = ['/usr/local/lib'] language = c define_macros = [('HAVE_CBLAS', None)]

and by

(ml1) myself@mytux:/projekte/GIT/ai/ml1/lib64/python3.6/site-packages/numpy/core> ldd _multiarray_umath.cpython-36m-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007ffe8bddf000) libopenblasp-r0-2ecf47d5.3.7.dev.so => /projekte/GIT/ai/ml1/lib/python3.6/site-packages/numpy/core/./../.libs/libopenblasp-r0-2ecf47d5.3.7.dev.so (0x00007fdd9d15f000) libm.so.6 => /lib64/libm.so.6 (0x00007fdd9ce27000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdd9cc09000) libc.so.6 => /lib64/libc.so.6 (0x00007fdd9c84f000) /lib64/ld-linux-x86-64.so.2 (0x00007fdd9f4e8000) libgfortran-ed201abd.so.3.0.0 => /projekte/GIT/ai/ml1/lib/python3.6/site-packages/numpy/core/./../.libs/libgfortran-ed201abd.so.3.0.0 (0x00007fdd9c555000)

In all tests discussed below I performed a series of calculations for different batch sizes

**N = 50, 100, 200, 250, 260, 500, 2000, 10000, 20000**

and repeated the full forward propagation 30 times (corresponding to 30 epochs in a full training series - but here *without* cost calculation and weight adjustment. I just did forward propagation.)

In a first experiment, I did not artificially limit the number of cores to be used. Measured **response times in seconds** are indicated in the following plot:

**Runtime for a free number of cores to use and different batch-sizes N**

We see that something dramatic happens between a batch size of 250 and 260. Below you see the plots for CPU core consumption for N=50, N=200, N=250, N=260 and N=2000.

The plots indicate that everything goes well up to **N=250**. Up to this point around 4 cores are used - leaving 4 cores relatively free. After **N=260** OpenBlas decides to use all 8 cores with a load of 100% - and performance suffers by more than a factor of 2.

This result support the idea to look for an optimum of the **number of cores "C"** to use.

For a MLP with neuron numbers (784, 300, 140, 10) I got the red curve for response time in the plot below. The second curve shows what performance is possible with just using 4 cores:

Note the significantly higher response times. We also see again that something strange happens at the change of the batch-size from 250 to 260.

The 100% CPU consumption even for a batch-size of only 50 is shown below:

Though different from the first test case also these plots indicate that - somewhat paradoxically - reducing the number of CPU cores available to OpenBlas could have a performance enhancing effect.

A bit of Internet research shows that one can limit the number of cores to use by OpenBlas e.g. via an **environment variable** for the shell, in which we start a Jupyter notebook. The relevant command to limit the **number of cores "C"** to 3 is :

export OPENBLAS_NUM_THREADS=3

Below you find plots for the response times required for the batch sizes N listed above and core numbers of

C=1, C=2, C=3, C=4, C=5, C=6, C=7, C=8 :

For **C=5** I did 2 different runs; the different results for C=5 show that the system reacts rather sensitively. It changes its behavior for larger core number drastically.

We also find an overall minimum of the response time:

**The overall optimum occurs for 400 < N < 500 for C=1, 2, 3, 4 - with the minimum region being broadest for C=3. The absolute minimum is reached on my CPU for C=4.**

We understand from the plots above that the number of cores to use become ** hyper-parameters** for the tuning of the performance of ANNs - at least as long as a standard multicore-CPU is used.

CPU consumption for N=50 and **C=2** looks like:

For comparison see the CPU consumption for N=20000 and **C=4**:

CPU consumption for N=20000 and **C=6**:

We see that between C=5 and C=6 CPU resources get heavily consumed; there are almost no reserves left in the Linux system for C ≥ 6.

For a full view on the situation I also looked at the response time variation with node numbers for a given number of CPU cores.

For **C=4** and node number cases

- 784, 300, 140, 10
- 784, 200, 100, 10
- 784, 100, 50, 10
- 784, 50, 20, 10

I got the following results:

There is some broad variation with the weight-matrix size; the bigger the weight-matrix the longer the calculation time. This is, of course, to be expected. Note that the variation with the batch-size number is relatively smooth - with an optimum around 400.

Now, look at the same plot for **C=6**:

Note that the response time is *significantly bigger* in all cases compared to the previous situation with C=4. In cases of a large matrix by around **36%** for N=2000. Also the variation with batch-size is more pronounced.

Still, even with 6 cores you do not get factors between 1.4 and 2.0 as compared to the case of C=8 (see above)!

As I do not know what the authors of OpenBlas are doing exactly, I refrain from technically understanding and interpreting the causes of the data shown above.

However, some consequences seem to be clear:

- It is a bad idea to provide all CPU cores to OpenBlas - which unfortunately is the default.
- The data above indicate that using only 4 out of 8 core threads is reasonable to get an optimum performance for vectorized matrix multiplications on my CPU.
- Not leaving at least 2 CPU cores free for other tasks is punished by significant performance losses.
- When leaving the decision of how many cores to use to OpenBlas a critical batch-size may exist for which the performance suddenly breaks down due to heavy multi-threading.

Whenever you deal with ANN or MLP simulations on a standard CPU (not GPU!) you should absolutely care about how many cores and related threads you want to offer to OpenBlas. As far as I understood from some Internet articles the number of cores to be used can be not only be controlled by Linux (shell) environment variables but also by os-commands in a Python program. You should perform tests to find optimum values for your CPU.

stackoverflow: numpy-suddenly-uses-all-cpus

stackoverflow: run-openblas-on-multicore

stackoverflow: multiprocessing-pool-makes-numpy-matrix-multiplication-slower

scicomp: why-isnt-my-matrix-vector-multiplication-scaling/1729

**Setting the number of threads via Python**

stackoverflow:

set-max-number-of-threads-at-runtime-on-numpy-openblas

codereview.stackexchange: better-way-to-set-number-of-threads-used-by-numpy

]]>

In my last article

A simple program for an ANN to cover the Mnist dataset – I

I presented already some simple initial code for the "__init__"-function of our Python class. In the present article we shall extend this function to provide initial weights on the nodes of our network.

The present status of our __init__function is:

def __init__(self, my_data_set = "mnist", n_hidden_layers = 1, ay_nodes_layers = [0, 100, 0], # array which should have as much elements as n_hidden + 2 n_nodes_layer_out = 10, # expected number of nodes in output layer my_activation_function = "sigmoid", my_out_function = "sigmoid", n_mini_batch = 1000, # number of data elements in a mini-batch vect_mode = 'cols', figs_x1=12.0, figs_x2=8.0, legend_loc='upper right', b_print_test_data = True ): ''' Initialization of MyANN Input: data_set: type of dataset; so far only the "mnist", "mnist_784" datsets are known We use this information to prepare the input data and learn about the feature dimension. This info is used in preparing the size of the input layer. n_hidden_layers = number of hidden layers => between input layer 0 and output layer n ay_nodes_layers = [0, 100, 0 ] : We set the number of nodes in input layer_0 and the output_layer to zero Will be set to real number afterwards by infos from the input dataset. All other numbers are used for the node numbers of the hidden layers. n_nodes_out_layer = expected number of nodes in the output layer (is checked); this number corresponds to the number of categories NC = number of labels to be distinguished my_activation_function : name of the activation function to use my_out_function : name of the "activation" function of the last layer which produces the output values n_mini_batch : Number of elements/samples in a mini-batch of training dtaa vect_mode: Are 1-dim data arrays (vctors) ordered by columns or rows ? figs_x1=12.0, figs_x2=8.0 : Standard sizing of plots , legend_loc='upper right': Position of legends in the plots b_print_test_data: Boolean variable to control the print out of some tests data ''' # Array (Python list) of known input data sets self._input_data_sets = ["mnist", "mnist_784", "mnist_keras"] self._my_data_set = my_data_set # X, y, X_train, y_train, X_test, y_test # will be set by analyze_input_data # X: Input array (2D) - at present status of MNIST image data, only. # y: result (=classification data) [digits represent categories in the case of Mnist] self._X = None self._X_train = None self._X_test = None self._y = None self._y_train = None self._y_test = None # relevant dimensions # from input data information; will be set in handle_input_data() self._dim_sets = 0 self._dim_features = 0 self._n_labels = 0 # number of unique labels - will be extracted from y-data # Img sizes self._dim_img = 0 # should be sqrt(dim_features) - we assume square like images self._img_h = 0 self._img_w = 0 # Layers # ------ # number of hidden layers self._n_hidden_layers = n_hidden_layers # Number of total layers self._n_total_layers = 2 + self._n_hidden_layers # Nodes for hidden layers self._ay_nodes_layers = np.array(ay_nodes_layers) # Number of nodes in output layer - will be checked against information from target arrays self._n_nodes_layer_out = n_nodes_layer_out # Weights # -------- # empty List for all weight-matrices for all layer-connections # Numbering : # w[0] contains the weight matrix which connects layer 0 (input layer ) to hidden layer 1 # w[1] contains the weight matrix which connects layer 1 (input layer ) to (hidden?) layer 2 self._ay_w = [] # Known Randomizer methods ( 0: np.random.randint, 1: np.random.uniform ) # ------------------ self.__ay_known_randomizers = [0, 1] # Types of activation functions and output functions # ------------------ self.__ay_activation_functions = ["sigmoid"] # later also relu self.__ay_output_functions = ["sigmoid"] # later also softmax # the following dictionaries will be used for indirect function calls self.__d_activation_funcs = { 'sigmoid': self._sigmoid, 'relu': self._relu } self.__d_output_funcs = { 'sigmoid': self._sigmoid, 'softmax': self._softmax } # The following variables will later be set by _check_and set_activation_and_out_functions() self._my_act_func = my_activation_function self._my_out_func = my_out_function self._act_func = None self._out_func = None # number of data samples in a mini-batch self._n_mini_batch = n_mini_batch # print some test data self._b_print_test_data = b_print_test_data # Plot handling # -------------- # Alternatives to resize plots # 1: just resize figure 2: resize plus create subplots() [figure + axes] self._plot_resize_alternative = 1 # Plot-sizing self._figs_x1 = figs_x1 self._figs_x2 = figs_x2 self._fig = None self._ax = None # alternative 2 does resizing and (!) subplots() self.initiate_and_resize_plot(self._plot_resize_alternative) # *********** # operations # *********** # check and handle input data self._handle_input_data() print("\nStopping program regularily") sys.exit()

The kind reader may have noticed that this is not exactly what was presented in the last article. I have introduced two additional parameters and corresponding class attributes:

"b_print_test_data" and "n_mini_batch".

The first of these parameters controls whether we print out some test data or not. The second parameter controls the number of test samples dealt with in parallel during propagation and optimization via the so called "mini-batch-approach" mentioned in the last article.

We said in the last article that we would provide the numbers of nodes of the ANN layers via a parameter list "ay_nodes_layers". We set the number of nodes for the input and the output layer, i.e. the first number and the last number in the list, to "0" in this array because these numbers are determined by properties of the input data set - here the MNIST dataset.

All other numbers in the array determine the amount of nodes of the * hidden layers* in consecutive order between the input and the output layer. So, the number at ay_nodes_layers[1] is the number of nodes in the first hidden layer, i.e. the layer which follows after the input layer.

In the last article we have understood already that the number of nodes in the input layer should be equal to the full number of "features" of our input data set - 784 in our case. The number of nodes of the output layer must instead be determined from the number of categories in our data set. This is equivalent to the number of distinct labels in the set of training data represented by an array "_y_train" (in case of MNIST: 10).

We provide three methods to check the node numbers defined by the user, set the node numbers for the input and output layers and print the numbers.

# Method which checks the number of nodes given for hidden layers def _check_layer_and_node_numbers(self): try: if (self._n_total_layers != (self._n_hidden_layers + 2)): raise ValueError except ValueError: print("The assumed total number of layers does not fit the number of hidden layers + 2") sys.exit() try: if (len(self._ay_nodes_layers) != (self._n_hidden_layers + 2)): raise ValueError except ValueError: print("The number of elements in the array for layer-nodes does not fit the number of hidden layers + 2") sys.exit(1) # Method which sets the number of nodes of the input and the layer def _set_nodes_for_input_output_layers(self): # Input layer: for the input layer we do NOT take into account a bias node self._ay_nodes_layers[0] = self._dim_features # Output layer: for the output layer we check the number of unique values in y_train try: if ( self._n_labels != (self._n_nodes_layer_out) ): raise ValueError except ValueError: print("The unique elements in target-array do not fit number of nodes in the output layer") sys.exit(1) self._ay_nodes_layers[self._n_total_layers - 1] = self._n_labels # Method which prints the number of nodes of all layers def _show_node_numbers(self): print("\nThe node numbers for the " + str(self._n_total_layers) + " layers are : ") print(self._ay_nodes_layers)

The code should be easy to understand. self._dim_features was set in the method "_common_handling_of mnist()" discussed in the last article. It was derived from the shape of the input data array _X_train. The number of unique labels was evaluated by the method "_get_num_labels()" - also discussed in the last article.

**Note:** The initial node numbers DO NOT include a bias node, yet.

If we extend the final commands in the "__init__"-function by :

# *********** # operations # *********** # check and handle input data self._handle_input_data() # check consistency of the node-number list with the number of hidden layers (n_hidden) self._check_layer_and_node_numbers() # set node numbers for the input layer and the output layer self._set_nodes_for_input_output_layers() self._show_node_numbers() print("\nStopping program regularily") sys.exit()

By testing our additional code in a Jupyter notebook we get a corresponding output for

ANN = myann.MyANN(my_data_set="mnist_keras", n_hidden_layers = 2, ay_nodes_layers = [0, 100, 50, 0], n_nodes_layer_out = 10, vect_mode = 'cols', figs_x1=12.0, figs_x2=8.0, legend_loc='upper right', b_print_test_data = False )

of:

The node numbers for the 4 defined layers are: [784 100 50 10]

Good!

Initial values for the ANN weights have to be given as matrices, i.e. 2-dim arrays. However, the randomizer functions provided by Numpy give you vectors as output. So, we need to reshape such vectors into the required form.

First we define a method to provide random floating point and integer numbers:

# --- # method to create an array of randomized values def _create_vector_with_random_values(self, r_low=None, r_high=None, r_size=None, randomizer=0 ): ''' Method to create a vector of length "r_size" with "random values" in [r_low, r_high] generated by method "randomizer" Input: ramdonizer : integer which sets randomizer method; presently only 0: np.random.uniform 1: np.randint [r_low, r_high]: range of the random numbers to be created r_size: Size of output array Output: A 1-dim numpy array of length rand-size - produced as a class member ''' # check parameters try: if (r_low==None or r_high == None or r_size == None ): raise ValueError except ValueError: print("One of the required parameters r_low, r_high, r_size has not been set") sys.exit(1) rmizer = int(randomizer) try: if (rmizer not in self.__ay_known_randomizers): raise ValueError except ValueError: print("randomizer not known") sys.exit(1) # 2 randomizers (so far) if (rmizer == 0): ay_r_out = np.random.randint(int(r_low), int(r_high), int(r_size)) if (rmizer == 1): ay_r_out = np.random.uniform(r_low, r_high, size=int(r_size)) return ay_r_out

Presently, we can only use two randomizer functions to be used - numpy.random.randint and numpy.random.uniform. The first one provides random integer values, the other one floating point values - both within a defined interval. The parameter "r_size" defines how many random numbers shall be created and put into an array. The code requires no further explanation.

Now, we define two methods to create the weight matrices for the connections

- between the input layer L0 to the first hidden layer,
- between further hidden layers and eventually between the last hidden layer and the output layer

As we allow for all possible connections between nodes the dimensions of the matrices are determined by the numbers of nodes in the connected neighboring layers. Each node of a layer L_n cam be connected to each node of layer L_(n+1). Our methods are:

# Method to create the weight matrix between L0/L1 # ------ def _create_WM_Input(self): ''' Method to create the input layer The dimension will be taken from the structure of the input data We need to fill self._w[0] with a matrix for conections of all nodes in L0 with all nodes in L1 We fill the matrix with random numbers between [-1, 1] ''' # the num_nodes of layer 0 should already include the bias node num_nodes_layer_0 = self._ay_nodes_layers[0] num_nodes_with_bias_layer_0 = num_nodes_layer_0 + 1 num_nodes_layer_1 = self._ay_nodes_layers[1] # fill the matrix with random values rand_low = -1.0 rand_high = 1.0 rand_size = num_nodes_layer_1 * (num_nodes_with_bias_layer_0) randomizer = 1 # method np.random.uniform w0 = self._create_vector_with_random_values(rand_low, rand_high, rand_size, randomizer) w0 = w0.reshape(num_nodes_layer_1, num_nodes_with_bias_layer_0) # put the weight matrix into array of matrices self._ay_w.append(w0.copy()) print("\nShape of weight matrix between layers 0 and 1 " + str(self._ay_w[0].shape)) # Method to create the weight-matrices for hidden layers def _create_WM_Hidden(self): ''' Method to create the weights of the hidden layers, i.e. between [L1, L2] and so on ... [L_n, L_out] We fill the matrix with random numbers between [-1, 1] ''' # The "+1" is required due to range properties ! rg_hidden_layers = range(1, self._n_hidden_layers + 1, 1) # for random operation rand_low = -1.0 rand_high = 1.0 for i in rg_hidden_layers: print ("Creating weight matrix for layer " + str(i) + " to layer " + str(i+1) ) num_nodes_layer = self._ay_nodes_layers[i] num_nodes_with_bias_layer = num_nodes_layer + 1 # the number of the next layer is taken without the bias node! num_nodes_layer_next = self._ay_nodes_layers[i+1] # assign random values rand_size = num_nodes_layer_next * num_nodes_with_bias_layer randomizer = 1 # np.random.uniform w_i_next = self._create_vector_with_random_values(rand_low, rand_high, rand_size, randomizer) w_i_next = w_i_next.reshape(num_nodes_layer_next, num_nodes_with_bias_layer) # put the weight matrix into our array of matrices self._ay_w.append(w_i_next.copy()) print("Shape of weight matrix between layers " + str(i) + " and " + str(i+1) + " = " + str(self._ay_w[i].shape))

Three things may need explanation:

- The first thing is the inclusion of a bias-node in all layers. A bias node only provides input to the next layer but does not receive input from a preceding layer. It gives an additional bias to the input a layer provides to a neighbor layer. (Actually, it provides an additional degree of freedom for the optimization process.) So, we include connections from a bias-node of a layer L_n to all nodes (without the bias-node) in layer L_(n+1).
- The second thing is the reshaping of the vector of random numbers into a matrix. Of course, the dimension of the vector of random numbers must fit the product of the 2 matrix dimensions. Note that the first dimension always corresponds to number of nodes in layer L_(n+1) and the second dimension to the number of nodes in layer L_n (including the bias-node)!

We need this special form to support the vectorized propagation properly later on. - The third thing is that we save our (Numpy) weight matrices as elements of a Python list.

In my opinion this makes the access to these matrices flexible and easy in the case of multiple hidden layers.

We must set the activation and the output function. This is handled by a method "_check_and_set_activation_and_out_functions()".

def _check_and_set_activation_and_out_functions(self): # check for known activation function try: if (self._my_act_func not in self.__d_activation_funcs ): raise ValueError except ValueError: print("The requested activation function " + self._my_act_func + " is not known!" ) sys.exit() # check for known output function try: if (self._my_out_func not in self.__d_output_funcs ): raise ValueError except ValueError: print("The requested output function " + self._my_out_func + " is not known!" ) sys.exit() # set the function to variables for indirect addressing self._act_func = self.__d_activation_funcs[self._my_act_func] self._out_func = self.__d_output_funcs[self._my_out_func] if self._b_print_test_data: z = 7.0 print("\nThe activation function of the standard neurons was defined as \"" + self._my_act_func + '"') print("The activation function gives for z=7.0: " + str(self._act_func(z))) print("\nThe output function of the neurons in the output layer was defined as \"" + self._my_out_func + "\"") print("The output function gives for z=7.0: " + str(self._out_func(z)))

It requires not much of an explanation. For the time being we just rely on the given definition of the "__init__"-interface, which sets both to "sigmoid()". The internal dictionaries __d_activation_funcs[] and __d_output_funcs[] provide the functions to internal variables (indirect addressing).

A test output for the enhanced code

# *********** # operations # *********** # check and handle input data self._handle_input_data() # check consistency of the node-number list with the number of hidden layers (n_hidden) self._check_layer_and_node_numbers() # set node numbers for the input layer and the output layer self._set_nodes_for_input_output_layers() self._show_node_numbers() # create the weight matrix between input and first hidden layer self._create_WM_Input() # create weight matrices between the hidden layers and between tha last hidden and the output layer self._create_WM_Hidden() # check and set activation functions self._check_and_set_activation_and_out_functions() print("\nStopping program regularily") sys.exit()

and a Jupyter cell

gives :

The shapes of the weight matrices correspond correctly to the numbers of nodes in the 4 layers defined. (Do not forget about the bias-nodes!).

We have reached a status where our ANN class can read in the MNIST dataset and set initial random values for the weights. This means that we can start to do some more interesting things. In the next article

A simple program for an ANN to cover the Mnist dataset – III – forward propagation

we shall program the "forward propagation". We shall perform the propagation for a mini-batch of many data samples in one step. We shall see that this is a very simple task, which only requires a few lines of code.

]]>

So, I thought, just let us set up a small Python3 and Numpy based program to create a simple ANN and train it for the MNIST dataset.

I take a shortcut and assume that readers are already acquainted with the following topics:

- what simple neural networks with a hidden layer look like,
- what a cost function and the gradient descent method is,
- what logistic regression is and what the cost function
**J**for it looks like, - why the back propagation of deviations from known values gives you the required partial derivatives for typical cost functions,
- what a mini-batch approach to optimization is.

These basics are well documented in literature; e.g. in the books of Geron and Rashka (see the references at the end of this article). However, I shall briefly comment on these topics whilst building the code.

We need a relatively well defined first objective for the usage of our ANN. We shall concentrate on classification tasks. As a first example we shall use the conventional MNIST data set. The MNIST data set consists of 28x28 px images of handwritten numbers. It is a standard data set used in many elementary courses on ML. The challenge for the ANN is that it should be able to recognize hand-written digits from a digitized gray-color image after some training.

Note that this task does NOT require the use of an ANN. "Stochastic Gradient Descent"-approaches to determine (linear) separation surfaces in combination with a One-versus-All strategy for multi-category-classification may be sufficient. See chapters 3 to 5 in the book of Geron for more information.

Regarding the build up of the ANN program, I basically follow an approach described by Raschka in his book. However, at multiple points I take the freedom to organize the code differently and comment in my own way ... I am only a beginner in Python; I hope my insights are helpful for others in the same situation.

In any case you should make yourself familiar with numpy arrays and their "shapes". I assume that you understand the multidimensional structure of Numpy arrays ....

To avoid confusion, I use the following wording:

**Category:** Each input data element is associated with a category to which it belongs. The classification algorithm (here: the ANN) may achieve a distinction between the association of input data with multiple categories. It should - after some training - detect (non-linear) separation interfaces for categories in a multidimensional feature space. In the case of MNIST we speak about ten categories corresponding to 10 digits, including zero.

**Label:** A category may be described by a label. Training data may provide a so called "target label array" **_y_train** for all input data. We must be prepared to transform target labels for input data into a usable form for an ANN, i.e. into a vectorized form which selects a specific category out of many. This process is called **"label encoding"**.

**Input data set:** A complete set of input data. Such a set consists of individual "**elements**" or "**records**". The MNIST input set of training data has 60000 records - which we provide via an array **_X_train**.

**Output data set:** A complete set of output data after propagation through the ANN for the input data set. The number of elements or records of the output data sets is equal to the number of records in the input data set. The output set will be represented by an array "**_ay_a_Out**".

**A data record of the input data set:** One distinct element of the input data set (and its array). Note that such an element itself may be a multidimensional array covering all features in a distinct form. Such arrays represents a so called "tensor".

**A data record of the output data set:** One distinct element of the output data set. Note that such an element itself may be an array covering all possible categories in a distinct form.

A neural network is composed of a series of sequential **layers** with **nodes**. All nodes of a specific layer can be connected with all nodes of neighboring layers. The simplified sketch below displays an ANN with just three layers - an input layer, a "hidden" middle layer and an output layer. Note that there can be (many) more hidden layers than just one.

**Input layer and its number of nodes**

To feed input data into the ANN we need an input layer with input nodes. How many? Well, this depends on the number of * features* your data set represents. In the MNIST case an 28x28 px image with simple gray values (integer number between 0 and 256) represents 28x28 = 768 different "

For other input data the number and structure of features may be different; the number of input nodes must then be adjusted accordingly. The number of nodes should, therefore, be a *parameter* or be derived from information on the type of input data. How complicated and structured features are mapped to a one dimensional input vector is a question one should think about carefully. (Most people today deal with a time set of data as just a special form of a feature - I regard this as questionable in some cases, but this is beyond this article series ...)

**Output layer and its number of nodes**

We shall use our ANN for * classification* tasks in the beginning. We, therefore, assume that the output of the ANN should distinguish between "strong>NC" different categories an input data set can belong to. In case of the MNIST dataset we can distinguish between 10 different digits. Thus an output layer must in this case at least comprise 10 different nodes. To be able to cover other data sets with a different number of categories the number of output nodes must be a parameter, too.

How we indicate the belonging to a category numerically - by a probability number between "0" and "1" or just a "1" at the right category and zeros otherwise - can be a matter of discussion. It is also a question of the cost function we wish to use. We will come back to this point later.

**The numbers of "hidden layers" and their nodes**

We want these numbers to be parameters, too. For simple datasets we do not need big networks, but we want to able to play around a bit with 1 up to 3 layers. (For an ANN to recognize hand written digits an input layer "layer 1" and only one hidden layer "layer 2" before the output layer"layer 3" - are required.)

**Activation and output functions**

The nodes in hidden layers use a so called "activation function" to transform the *aggregated* input from different feeding nodes of the previous layer into one distinct value in a given interval - e.g. between 0 and 1. Again, we should be prepared to have a parameter to choose between different activation functions.

We should be aware of the fact that the nodes of the output layers need special consideration as the "activation function" there produces the final output - which must allow for a distinction of categories. This may lead to a special form - e.g. a kind of probability function. So, output functions should also be regarded as variable.

I develop my code as a Python *module* in an PyDEV/Eclipse which uses a virtual Python environment. I described the setup of such an environment in detail in XXXX. In the directory structure there I place a module "myann.py" at the location "...../ml_1/mynotebooks/mycode/myann.py". We need to import some libraries first

''' Module to create a simple layered neural network for the MNIST data set Created on 23.08.2019 @author: ramoe ''' import numpy as np import math import sys import time import tensorflow from sklearn.datasets import fetch_mldata from sklearn.datasets import fetch_openml from keras.datasets import mnist as kmnist from scipy.special import expit from matplotlib import pyplot as plt #from matplotlib.colors import ListedColormap #import matplotlib.patches as mpat #from keras.activations import relu

Why do I import "tensorflow" and "keras"?

Well, only for the purpose to create the input data of MNIST quickly. Sklearn's "fetchml_data" is doomed to end. "fetch_openml" does not use caching in some older versions and is also otherwise terribly slow. But, "keras", which in turn needs tensorflow as a backend, provides its own tool to provide the MNIST data.

We need "scipy" to get an optimized version of the "sigmoid"-function which is one important version of an activation function. "numpy" and "math" are required for fast array- and math-operations. "time" is required to measure the run time of program segments and "mathplotlib" will help to visualize some information during training.

We encapsulate most of the functionality in a class and its methods. Python provides the "__init__"-function, which we can use as a kind of "constructor" - although it technically is not the same as a constructor in other languages. Anyway, we can use it as an interface to provide parameters and initialize variables in a class instance.

We shall build up our "__init__"-function during the next articles step by step in form of well separated segments; in the beginning we only look a attributes and methods required to set up (MNIST) input data, to create the basic network layers and to create arrays with the (MNIST) input data.

**Parameters**

class MyANN: def __init__(self, my_data_set = "mnist", n_hidden_layers = 1, ay_nodes_layers = [0, 100, 0], # array which should have as much elements as n_hidden + 2 n_nodes_layer_out = 10, # number of nodes in output layer my_activation_function = "sigmoid", my_out_function = "sigmoid", vect_mode = 'cols', figs_x1=12.0, figs_x2=8.0, legend_loc='upper right' ): ''' Initialization of MyANN Input: data_set: type of dataset; so far only the "mnist", "mnist_784" and the "mnist_keras" datsets are known. We use this information to prepare the input data and learn about the feature dimension. This info is used in preparing the size of the input layer. n_hidden_layers = number of hidden layers => between input layer 0 and output layer n ay_nodes_layers = [0, 100, 0 ] : We set the number of nodes in input layer_0 and the output_layer to zero Will be set to real number afterwards by infos from the input dataset. All other numbers are used for the node numbers of the hidden layers. n_nodes_layer_out = expected number of nodes in the output layer (is checked); this number corresponds to the number of categories to be distinguished my_activation_function : name of the activation function to use my_out_function : name of the "activation" function of the last layer whcih produces the output values vect_mode: Are 1-dim data arrays (vectors) ordered by columns or rows ? figs_x1=12.0, figs_x2=8.0 : Standard sizing of plots , legend_loc='upper right': Position of legends in the plots '''

You see that I defined multiple parameters, which are explained in the doc string. We use a name to choose the dataset to train our ANN on. We assume that special methods to fetch the required data are implemented in our class. This requires that the class knows exactly which data sets it is capable to handle. We provide an list with this information below. The other parameters should be clear from their explanation.

We first initialize a bunch of class attributes which we shall use to define the network of layers, nodes, weights, to keep our input data and functions.

# Array (Python list) of known input data sets self.__input_data_sets = ["mnist", "mnist_784", "mnist_keras"] self._my_data_set = my_data_set # X, y, X_train, y_train, X_test, y_test # will be set by analyze_input_data # X: Input array (2D) - at present status of MNIST image data, only. # y: result (=classification data) [digits represent categories in the case of Mnist] self._X = None self._X_train = None self._X_test = None self._y = None self._y_train = None self._y_test = None # relevant dimensions # from input data information; will be set in handle_input_data() self._dim_sets = 0 self._dim_features = 0 self._n_labels = 0 # number of unique labels - will be extracted from y-data # Img sizes self._dim_img = 0 # should be sqrt(dim_features) - we assume square like images self._img_h = 0 self._img_w = 0 # Layers # ------ # number of hidden layers self._n_hidden_layers = n_hidden_layers # Number of total layers self._n_total_layers = 2 + self._n_hidden_layers # Nodes for hidden layers self._ay_nodes_layers = np.array(ay_nodes_layers) # Number of nodes in output layer - will be checked against information from target arrays self._n_nodes_layer_out = n_nodes_layer_out # Weights # -------- # empty List for all weight-matrices for all layer-connections # Numbering : # w[0] contains the weight matrix which connects layer 0 (input layer ) to hidden layer 1 # w[1] contains the weight matrix which connects layer 1 (input layer ) to (hidden?) layer 2 self._ay_w = [] # Known Randomizer methods ( 0: np.random.randint, 1: np.random.uniform ) # ------------------ self.__ay_known_randomizers = [0, 1] # Types of activation functions and output functions # ------------------ self.__ay_activation_functions = ["sigmoid"] # later also relu self.__ay_output_functions = ["sigmoid"] # later also softmax # the following dictionaries will be used for indirect function calls self.__d_activation_funcs = { 'sigmoid': self._sigmoid, 'relu': self._relu } self.__d_output_funcs = { 'sigmoid': self._sigmoid, 'softmax': self._softmax } # The following variables will later be set by _check_and set_activation_and_out_functions() self._my_act_func = my_activation_function self._my_out_func = my_out_function self._act_func = None self._out_func = None # Plot handling # -------------- # Alternatives to resize plots # 1: just resize figure 2: resize plus create subplots() [figure + axes] self._plot_resize_alternative = 1 # Plot-sizing self._figs_x1 = figs_x1 self._figs_x2 = figs_x2 self._fig = None self._ax = None # alternative 2 does resizing and (!) subplots() self.initiate_and_resize_plot(self._plot_resize_alternative) # *********** # operations # *********** # check and handle input data self._handle_input_data() print("\nStopping program regularily") sys.exit()

To make things not more complicated as necessary in the beginning I omit the usage of properties and full encapsulation of private attributes. For convenience reasons I also use only one underscore for some attributes and functions/methods to allow for external usage. This is helpful in a testing phase. However, many items can in the end be switched to really private properties or methods.

**List of known input datasets**

The list of known input data sets is kept in the variable "self.__input_data_sets". The variables

self._X, self._X_train, self._X_test, self._y, self._y_train, self._y_test

will be used to keep all data of the chosen dataset, the training data, the test data for checking of the reliability of the algorithm after training, and the corresponding target data (y_...) for classification. The target data in the MNIST case contain the *digit* the respective image (X_..) represents.

All of the named attributes will become numpy arrays. A method called "**_handle_input_data(self)**" will load the (MNIST) input data and fill the numpy arrays.

The input arrays "X_..." will via their *dimensions* provide the information on the number of data sets (_dim_sets) and the number of features (_dim_features). The target data arrays "Y_..." provide the number of "classes" (MNIST: 10 digits) to distinguish between by the ANN algorithm. We keep this number in the attribute "_n_labels".

It is also useful to keep the pixel dimensions of input image data. At least for MNIST we assume quadratic images (_img_h = img_w = _dim_img).

**Layers and weights**

The number of total layers ("_n_total_layers") is by 2 bigger than the number of hidden layers (_n_hidden_layers).

We take the number of nodes in the layers from the respective list provided as an input parameter "ay_nodes_layers" to our class. We transform the list into a numpy array "_ay_nodes_layers". The expected number of nodes in the output layer is used for consistency checks and saved in "_n_nodes_layer_out".

The "weights" of a network must be given in form of matrices - as they describe connections between nodes of different adjacent layers. As the number of layers is not fixed but can be set by the user, I use a list "**_ay_w**" to collect such matrices in the order of layer_0 (input) to layer_n (output).

Weights, i.e. the matrix elements must initially be defined by random numbers. To provide such numbers we have to use randomizer functions. Depending on the kind (floating point numbers, integer numbers) we use at least two randomizers (randint, uniform).

Usable activation and output functions are named in Python dictionaries which point to respective methods. This allows for "**indirect addressing**" of these functions later on. You may recognize this by the direct refernece of the dictionary elements to defined class methods (no strings are used!).

For the time being we work with the "sigmoid" and the "relu" functions for activation and the "sigmoid" and "softmax" functions for output creation. The attributes "self._act_func" and "self._out_func" are used later on to invoke the functions requested by the respective parameters of the classes interface.

The final part of the code segment given above is used for plot sizing with the help of "matplotlib"; a method "initiate_and_resize_plot()" takes care of this. It can use 2 alternative ways of doing so.

Now let us turn to some methods. We first need to read in and prepare the input data. We use a method "**_handle_input_data()**" to work on this problem.

For the time being we have only three different ways to load the MNIST dataset from different origins. We need not yet call different methods but deal with each MNIST source within the method:

# Method to handle different types of input data sets def _handle_input_data(self): ''' Method to deal with the input data: - check if we have a known data set ("mnist" so far) - reshape as required - analyze dimensions and extract the feature dimension(s) ''' # check for known dataset try: if (self._my_data_set not in self._input_data_sets ): raise ValueError except ValueError: print("The requested input data" + self._my_data_set + " is not known!" ) sys.exit() # handle the mnist original dataset if ( self._my_data_set == "mnist"): mnist = fetch_mldata('MNIST original') self._X, self._y = mnist["data"], mnist["target"] print("Input data for dataset " + self._my_data_set + " : \n" + "Original shape of X = " + str(self._X.shape) + "\n" + "Original shape of y = " + str(self._y.shape)) self._X_train, self._X_test, self._y_train, self._y_test = self._X[:60000], self._X[60000:], self._y[:60000], self._y[60000:] # handle the mnist_784 dataset if ( self._my_data_set == "mnist_784"): mnist2 = fetch_openml('mnist_784', version=1, cache=True, data_home='~/scikit_learn_data') self._X, self._y = mnist2["data"], mnist2["target"] print ("data fetched") # the target categories are given as strings not integers self._y = np.array([int(i) for i in self._y]) print ("data modified") print("Input data for dataset " + self._my_data_set + " : \n" + "Original shape of X = " + str(self._X.shape) + "\n" + "Original shape of y = " + str(self._y.shape)) self._X_train, self._X_test, self._y_train, self._y_test = self._X[:60000], self._X[60000:], self._y[:60000], self._y[60000:] # handle the mnist_keras dataset if ( self._my_data_set == "mnist_keras"): (self._X_train, self._y_train), (self._X_test, self._y_test) = kmnist.load_data() len_train = self._X_train.shape[0] #print(len_train) print("Input data for dataset " + self._my_data_set + " : \n" + "Original shape of X_train = " + str(self._X_train.shape) + "\n" + "Original Shape of y_train = " + str(self._y_train.shape)) len_test = self._X_test.shape[0] #print(len_test) print("Original shape of X_test = " + str(self._X_test.shape) + "\n" + "Original Shape of y_test = " + str(self._y_test.shape)) self._X_train = self._X_train.reshape(len_train, 28*28) self._X_test = self._X_test.reshape(len_test, 28*28) # Common Mnist handling if ( self._my_data_set == "mnist" or self._my_data_set == "mnist_784" or self._my_data_set == "mnist_keras" ): self._common_handling_of_mnist() # Other input data sets can not yet be handled

We first check, whether the input parameter fits a known dataset - and raise an error if otherwise. The data come in different forms for the three sources of MNIST. For each set we want to extract arrays

self._X_train, self._X_test, self._y_train, self._y_test.

We have to do this a bit differently for the 3 cases. Note that the "mnist_784" set from "fetch_openml" gives the target

category values in form of *strings* and not integers. We correct this after loading.

The fastest method is the MNIST dataset based on "keras"; the keras function "kmnist.load_data()" provides already a 60000:10000 ratio for training and test data. However, we get the images in a (60000, 28, 28) array shape; we therefore **reshape** the "_X_train"-array to (60000, 784) and "_X_test"-array to (10000, 784).

The further handling of the MNIST data requires some common analysis.

# Method for common input data handling of Mnist data sets def _common_handling_of_mnist(self): print("\nFinal input data for dataset " + self._my_data_set + " : \n" + "Shape of X_train = " + str(self._X_train.shape) + "\n" + "Shape of y_train = " + str(self._y_train.shape) + "\n" + "Shape of X_test = " + str(self._X_test.shape) + "\n" + "Shape of y_test = " + str(self._y_test.shape) ) # mixing the training indices shuffled_index = np.random.permutation(60000) self._X_train, self._y_train = self._X_train[shuffled_index], self._y_train[shuffled_index] # set dimensions self._dim_sets = self._y_train.shape[0] self._dim_features = self._X_train.shape[1] self._dim_img = math.sqrt(self._dim_features) # we assume square images self._img_h = int(self._dim_img) self._img_w = int(self._dim_img) # Print dimensions print("\nWe have " + str(self._dim_sets) + " data sets for training") print("Feature dimension is " + str(self._dim_features) + " (= " + str(self._img_w)+ "x" + str(self._img_h) + ")") # we need to encode the digit labels of mnist self._get_num_labels() self._encode_all_mnist_labels()

As you see we retrieve some of our class attributes which we shall use during training and do some printing. This is trivial. Not so trivial is the handling of the output data.

What shape do we expect for the "_X_train" and "_y_train"? Each element of the input data set is an array with values for all features. Thus the "_X_train.shape" should be (60000, 784). For _y_train we expect a simple integer describing the digit to which the MNIST input image corresponds. Thus we expect a one dimensional array with _y_train.shape = (60000).

However, the output data of our ANN for one input element come as an array of values for different categories - and not as a simple number. This shows that we need to *encode* the "_y_train"-data, i.e. the target labels, into a usable array form. We use two methods to achieve this:

# Method to encode mnist labels def _get_num_labels(self): self._n_labels = len(np.unique(self._y_train)) print("The number of labels is " + str(self._n_labels)) # Method to encode all mnist labels def _encode_all_mnist_labels(self, b_print=True): ''' We shall use vectorized input and output - i.e. we process a whole batch of input data sets in parallel (see article in the Linux blog) The output array will then have the form OUT(i_out_node, idx) where i_out_node enumerates the node of the last layer (i.e. the category) idx enumerates the data set within a batch, After training, if y_train[idx] = 6, we would expect an output value of OUT[6,idx] = 1.0 and OUT[i_node, idx]=0.0 otherwise for a categorization decision in the ideal case. Realistically, we will get a distribution of numbers over the nodes with values between 0.0 and 1.0 - with hopefully the maximum value at the right node OUT[6,idx]. The following method creates an arrays OneHot[i_out_node, idx] with OneHot[i_node_out, idx] = 1.0, if i_node_out = int(y[idx]) OneHot(i_node_out, idx] = 0.0, if i_node_out != int(y[idx]) This will allow for a vectorized comparison of calculated values and knwon values during training ''' self._ay_onehot = np.zeros((self._n_labels, self._y_train.shape[0])) # ay_oneval is just for convenience and printing purposes self._ay_oneval = np.zeros((self._n_labels, self._y_train.shape[0], 2)) if b_print: print("\nShape of y_train = " + str(self._y_train.shape)) print("Shape of ay_onehot = " + str(self._ay_onehot.shape)) # the next block is just for illustration purposes and a better understanding if b_print: values = enumerate(self._y_train[0:12]) print("\nValues of the enumerate structure for the first 12 elements : = ") for iv in values: print(iv) # here we prepare the array for vectorized comparison print("\nLabels for the first 12 datasets:") for idx, val in enumerate(self._y_train): self._ay_onehot[val, idx ] = 1.0 self._ay_oneval[val, idx, 0] = 1.0 self._ay_oneval[val, idx, 1] = val if b_print: print("\nShape of ay_onehot = " + str(self._ay_onehot.shape)) print(self._ay_onehot[:, 0:12]) #print("Shape of ay_oneval = " + str(self._ay_oneval.shape)) #print(self._ay_oneval[:, 0:12, :])

The first method only determines the number of labels (= number of categories).

We see from the code of the second method that we encode the target labels in the form of two arrays. The relevant one for our optimization algorithm will be "_ay_onehot". This array is 2-dimensional. Why?

A big advantage of the optimization method we shall use later on during training is that we will perform weight adjustment for a **whole bunch** of training data *in one step*. Meaning:

We propagate a whole bunch of test data in parallel throughout the grid, get an array with result data (output array) for which we then calculate a value of the total cost function and an array containing the difference of the output array to an array of correct values (error) for all test data of the bunch. The "error" (i.e. the difference) will be back-propagated to determine the derivative of the cost function for corrections of the node weights.

Such a bunch is called a "**batch**" and if it is significantly smaller than the whole set of training data - a "**mini-batch**". Working with "mini-batches" during optimization is a compromise between using the full data set for gradient determination during each optimization step ("*batch approach*") or using just one input data element of the training set for gradient descent correction ("*stochastic approach*"). See chapter 4 of the book of Geron and chapter 2 in the book of Raschka for some information on this topic.

The advantage of mini-batches is that we can use vectorized linear algebra operations over all elements of the batch. Linear Algebra libraries are optimized to perform such operations on modern CPUs and GPUs.

You really should keep in mind for understanding the code for the propagation and optimization algorithms discussed in forthcoming articles that the cost function is determined **over all elements of a batch** and the derivative determination is a matrix like operation involving all input elements of each defined batch! This corresponds to the point that the separation interface in the feature space must be adjusted with respect to all given data points in the end. Mini-batches only help in so far as we look at selected samples of data to achieve bigger steps of our gradient guided descent into an optimum in the beginning - with the disadvantage of making some jumpy stochastic turns instead of a smoother approach.

**What is the shape of the output array? **

A single input element of the bunch is an array of 784 feature values. The corresponding output array is an array with values for 10 categories (here digits). But, what about a whole bunch of test data, i.e. a "batch"?

As I have explained already in my last article

Numpy matrix multiplication for layers of simple feed forward ANNs

that the output array for a bunch of test data will have the form "_ay_a_Out[i_out_node, idx]" with:

*i_out_node*enumerating the node of the last layer, i.e. the various possible category*idx*enumerating the data set within a batch of training data

We shall construct the output function such that it provides values within the interval [0,1] for each node of the output layer. We define a perfectly working grid after training as one that would produce a "1" for the correct category node (i.e. the expected digit) and "0" otherwise.

For error-back-propagation we need to compare the real results with the correct category values for the test data. To be able to do this we must build up a 2-dim array of the same shape as "_ay_a_Out" with correct output values for all test data of the batch. E.g.: If we expect the digit 7 for an input array of index idx within the set of training data, we need a 2-dim output array with the element [[0,0,0,0,0,0,0,1,0,0], idx].

By using Numpy's zero()-function and Pythons "enumerate()"-function we achieve exactly this for all data elements of the training data set. See the method "_encode_all_mnist_labels()". Thus, _ay_onehot.shape is expected to be (10, 60000). From this 2-dim array we can later cut out bunches of consecutive test data for mini-batches.

We print out the values for the first 12 elements of the input data set.

The array "_ay_oneval" is provided for convenience and print purposes: it provides the expected *digit* value in addition.

Let us test the reading of the input data and the label encoding with a Jupyter notebook. In previous articles I have described already how to do so.

I build a Jupyter notebook called "myANN" (in my present working directory "/projekte/GIT/ai/ml1/mynotebooks").

I start it with

myself@mytux:/projekte/GIT/ai/ml1> source bin/activate (ml1) myself@mytux:/projekte/GIT/ai/ml1> jupyter notebook [I 15:07:30.953 NotebookApp] Writing notebook server cookie secret to /run/user/21001/jupyter/notebook_cookie_secret [I 15:07:38.754 NotebookApp] jupyter_tensorboard extension loaded. [I 15:07:38.754 NotebookApp] Serving notebooks from local directory: /projekte/GIT/ai/ml1 [I 15:07:38.754 NotebookApp] The Jupyter Notebook is running at: [I 15:07:38.754 NotebookApp] http://localhost:8888/?token=06c2626c8724f65d1e3c4a50457da0d6db414f88a40c7baf [I 15:07:38.755 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 15:07:38.771 NotebookApp]

and add two cells. The first one is for imports of libraries.

By the last line I import my present class code. With the second cell I start my class; the "__init__"-function is automatically executed:

Note that the display of "_ay_onehot" shows the categories in vertical direction (rows) and the index for the input data element in horizontal direction (columns)! You see that correspondence of the labels in the enumerate structure correspond to the "1"s in the "_ay_onehot"-array.

Reading the MNIST dataset into Numpy arrays via Keras is simple. We have prepared an array "_ay_onehot"- which we shall use during optimization to calculate a difference between calculated output of the ANN and the expected category for an element of our training data set.

In the next article

A simple program for an ANN to cover the Mnist dataset – II

we shall define initial weights for our ANN.

**Referenced Books**

"Python machine Learning", Seb. Raschka, 2016, Packt Publishing, Birmingham, UK

"Machine Learning mit Sckit-Learn & TensorFlow", A. Geron, 2018, O'REILLY, dpunkt.verlag GmbH, Heidelberg, Deutschland

**Links regarding cost (or loss) functions and logistic regression**

https://towardsdatascience.com/introduction-to-logistic-regression-66248243c148

https://cmci.colorado.edu/classes/INFO-4604/files/slides-5_logistic.pdf

Wikipedia article on Loss functions for classification

https://towardsdatascience.com/optimization-loss-function-under-the-hood-part-ii-d20a239cde11

https://stackoverflow.com/questions/32986123/why-the-cost-function-of-logistic-regression-has-a-logarithmic-expression

https://medium.com/technology-nineleaps/logistic-regression-gradient-descent-optimization-part-1-ed320325a67e

https://blog.algorithmia.com/introduction-to-loss-functions/

uni leipzig on logistic regression

The reason for this failure was something I missed already at the time when I upgraded from Leap 42.3 to Leap 15.0:

SuSE uses the "**update-alternatives**" mechanism since Leap 15.0. You easily get information from SuSE web-sites how to use this mechanism. See e.g.

https://en.opensuse.org/SDB:Change_Display_Manager

However, I wanted to get some background-information about update-alternatives, e.g. about which files or directories are used to keep related information. So, here are some links which may help others, too:

https://linux.die.net/man/8/update-alternatives

http://manpages.ubuntu.com/manpages/trusty/de/man8/update-alternatives.8.html

https://debiananwenderhandbuch.de/update-alternatives.html

https://en.opensuse.org/openSUSE: Packaging Multiple Version guidelines

https://wiki.ubuntuusers.de/Alternativen-System/

In short: update-alternatives provides the possibility to easily define and switch between different programs for the same (or a very similar) purpose on a Linux system. It works via symlinks, which are defined in the directory "/etc/alternatives".

As expected you can use "update-alternatives" on the command line. However, on an Opensuse-Leap-system yast and yast2 can be enhanced by a package "yast2-alternatives". After installation you find a respective entry below the main point "miscellaneous" of the yast2 menu. This allows for quick changes between already defined alternatives - e.g. for the desktop session, the displaymanager, java, ftp and others.

]]>