# A simple Python program for an ANN to cover the MNIST dataset – XI – confusion matrix

Welcome back to my readers who followed me through the (painful?) process of writing a Python class to simulate a “Multilayer Perceptron” [MLP]. The pain in my case resulted from the fact that I am still a beginner in Machine Learning [ML] and Python. Nevertheless, I hope that we have meanwhile acquired some basic understanding of how a MLP works and “learns”. During the course of the last articles we had a close look at such nice things as “forward propagation”, “gradient descent”, “mini-batches” and “error backward propagation”. For the latter I gave you a mathematical description to grasp the background of the matrix operations involved.

Where do we stand after 10 articles and a PDF on the math?

We have a working code

• with some parameters to control layers and node numbers, learning and momentum rates and regularization,
• with many dummy parts for other output and activation functions than the sigmoid function we used so far,
• with prepared code fragments for applying MSE instead of “Log Loss” as a cost function,
• and with dummy parts for handling different input datasets than the MNIST example.

The code is not yet optimized; it includes e.g. many statements for tests which we should eliminate or comment out. A completely open conceptual aspect is the optimization of the adaption of the learning rate; it is very primitive so far. We also need an export/import functionality to be able to perform training with a series of limited epoch numbers per run.
We also should save the weights and accuracy data after a fixed epoch interval to be able to analyze a bit more after training. Another idea – though probably costly – is to even perform intermediate runs on the test data set an get some information on the development of the averaged error on the test data set.

Despite all these deficits, which we need to cover in some more articles, we are already able to perform an insightful task – namely to find out with which numbers and corresponding images of the MNIST data set our MLP has problems with. This leads us to the topics of a confusion matrix and other measures for the accuracy of our algorithm.

However, before we look at these topics, we first create some useful code, which we can save inside cells of the Jupyter notebook we maintain for testing our class “MyANN”.

# Some functions to evaluate the prediction capability of our ANN after training

For further analysis we shall apply the following functions later on:

```# ------ predict results for all test data
# *************************
def predict_all_test_data():
size_set = ANN._X_test.shape[0]

li_Z_in_layer_test  = [None] * ANN._n_total_layers
li_Z_in_layer_test[0] = ANN._X_test

# Transpose input data matrix
ay_Z_in_0T       = li_Z_in_layer_test[0].T
li_Z_in_layer_test[0] = ay_Z_in_0T
li_A_out_layer_test  = [None] * ANN._n_total_layers

# prediction by forward propagation of the whole test set
ANN._fw_propagation(li_Z_in = li_Z_in_layer_test, li_A_out = li_A_out_layer_test, b_print = False)
ay_predictions_test = np.argmax(li_A_out_layer_test[ANN._n_total_layers-1], axis=0)

# accuracy
ay_errors_test = ANN._y_test - ay_predictions_test
acc = (np.sum(ay_errors_test == 0)) / size_set
print ("total acc for test data = ", acc)

def predict_all_train_data():
size_set = ANN._X_train.shape[0]

li_Z_in_layer_test  = [None] * ANN._n_total_layers
li_Z_in_layer_test[0] = ANN._X_train
# Transpose
ay_Z_in_0T       = li_Z_in_layer_test[0].T
li_Z_in_layer_test[0] = ay_Z_in_0T
li_A_out_layer_test  = [None] * ANN._n_total_layers

ANN._fw_propagation(li_Z_in = li_Z_in_layer_test, li_A_out = li_A_out_layer_test, b_print = False)
Result = np.argmax(li_A_out_layer_test[ANN._n_total_layers-1], axis=0)
Error = ANN._y_train - Result
acc = (np.sum(Error == 0)) / size_set
print ("total acc for train data = ", acc)

# Plot confusion matrix
# orginally from Runqi Yang;
# see https://gist.github.com/hitvoice/36cf44689065ca9b927431546381a3f7
def cm_analysis(y_true, y_pred, filename, labels, ymap=None, figsize=(10,10)):
"""
Generate matrix plot of confusion matrix with pretty annotations.
The plot image is saved to disk.
args:
y_true:    true label of the data, with shape (nsamples,)
y_pred:    prediction of the data, with shape (nsamples,)
filename:  filename of figure file to save
labels:    string array, name the order of class labels in the confusion matrix.
use `clf.classes_` if using scikit-learn models.
with shape (nclass,).
ymap:      dict: any -> string, length == nclass.
if not None, map the labels & ys to more understandable strings.
Caution: original y_true, y_pred and labels must align.
figsize:   the size of the figure plotted.
"""
if ymap is not None:
y_pred = [ymap[yi] for yi in y_pred]
y_true = [ymap[yi] for yi in y_true]
labels = [ymap[yi] for yi in labels]
cm = confusion_matrix(y_true, y_pred, labels=labels)
cm_sum = np.sum(cm, axis=1, keepdims=True)
cm_perc = cm / cm_sum.astype(float)
* 100
annot = np.empty_like(cm).astype(str)
nrows, ncols = cm.shape
for i in range(nrows):
for j in range(ncols):
c = cm[i, j]
p = cm_perc[i, j]
if i == j:
s = cm_sum[i]
annot[i, j] = '%.1f%%\n%d/%d' % (p, c, s)
elif c == 0:
annot[i, j] = ''
else:
annot[i, j] = '%.1f%%\n%d' % (p, c)
cm = pd.DataFrame(cm, index=labels, columns=labels)
cm.index.name = 'Actual'
cm.columns.name = 'Predicted'
fig, ax = plt.subplots(figsize=figsize)
ax=sns.heatmap(cm, annot=annot, fmt='')
#plt.savefig(filename)

#
# Plotting
# **********
def plot_ANN_results():
num_epochs  = ANN._n_epochs
num_batches = ANN._n_batches
num_tot = num_epochs * num_batches

cshape = ANN._ay_costs.shape
print("n_epochs = ", num_epochs, " n_batches = ", num_batches, "  cshape = ", cshape )
tshape = ANN._ay_theta.shape
print("n_epochs = ", num_epochs, " n_batches = ", num_batches, "  tshape = ", tshape )

#sizing
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 12
fig_size[1] = 5

# Two figures
# -----------
fig1 = plt.figure(1)
fig2 = plt.figure(2)

# first figure with two plot-areas with axes
# --------------------------------------------

ax1_1.plot(range(len(ANN._ay_costs)), ANN._ay_costs)
ax1_1.set_xlim (0, num_tot+5)
ax1_1.set_ylim (0, 1500)
ax1_1.set_xlabel("epochs * batches (" + str(num_epochs) + " * " + str(num_batches) + " )")
ax1_1.set_ylabel("costs")

ax1_2.plot(range(len(ANN._ay_theta)), ANN._ay_theta)
ax1_2.set_xlim (0, num_tot+5)
ax1_2.set_ylim (0, 0.15)
ax1_2.set_xlabel("epochs * batches (" + str(num_epochs) + " * " + str(num_batches) + " )")
ax1_2.set_ylabel("averaged error")

```

The first function “predict_all_test_data()” allows us to create an array with the predicted values for all test data. This is based on a forward propagation of the full set of test data; so we handle some relatively big matrices here. The second function delivers prediction values for all training data; the operations of propagation algorithm involve even bigger matrices here. You will nevertheless experience that the calculations are performed very quickly. Prediction is much faster than training!

The third function “cm_analysis()” is not from me, but taken from Github Gist; see below. The fourth function “plot_ANN_results()” creates plots of the evolution of the cost function and the averaged error after training. We come back to these functions below.

To be able to use these functions we need to perform some more imports first. The full list of statements which we should place in the first Jupyter cell of our test notebook now reads:

```import numpy as np
import numpy.random as npr
import math
import sys
import pandas as pd
from sklearn.datasets import fetch_openml
from sklearn.metrics import confusion_matrix
from scipy.special import expit
import seaborn as sns
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
import matplotlib.patches as mpat
import time
import imp
from mycode import myann
```

Note the new lines for the import of the “pandas” and “seaborn” libraries. Please inform yourself about the purpose of each library on the Internet.

# Limited Accuracy

In the last article we performed some tests which showed a thorough robustness of our MLP regarding the MNIST datatset. There was some slight overfitting, but
playing around with hyper-parameters showed no extraordinary jump in “accuracy“, which we defined to be the percentage of correctly predicted records in the test dataset.

In general we can say that an accuracy level of 95% is what we could achieve within the range of parameters we played around with. Regression regularization (Lambda2 > 0) had some positive impact. A structural change to a MLP with just one layer did NOT give us a real breakthrough regarding CPU-time consumption, but when going down to 50 or 30 nodes in the intermediate layer we saw at least some reduction by up to 25%. But then our accuracy started to become worse.

Whilst we did our tests we measured the ANN’s “accuracy” by comparing the number of records for which our ANN did a correct prediction with the total number of records in the test data set. This is a global measure of accuracy; it averages over all 10 digits, i.e. all 10 classification categories. However, if we want to look a bit deeper into the prediction errors our MLP obviously produces it is, however, useful to introduce some more quantities and other measures of accuracy, which can be applied on the level of each output category.

# Measures of accuracy, related quantities and classification errors for a specific category

The following quantities and basic concepts are often used in the context of ML algorithms for classification tasks. Predictions of our ANN will not be error free and thus we get an accuracy less than 100%. There are different reasons for this – and they couple different output categories. In the case of MNIST the output categories correspond to the digits 0 to 9. Let us take a specific output category, namely the digit “5”. Then there are two basic types of errors:

• The network may have predicted a “3” for a MNIST image record, which actually represents a “5” (according to the “y_train”-value for this record). This error case is called a “False Negative“.
• The network may have predicted a “5” for a MNIST image record, which actually represents a “3” according to its “y_train”-value. This error case is called a “False Positive“.

Both cases mark some difference between an actual and predicted number value for a MNIST test record. Technically, “actual” refers to the number value given by the related record in our array “ANN._y_test”. “Predicted” refers to the related record in an array “ay_prediction_test”, which our function “predict_all_test_data()” returns (see the code above).

Regarding our example digit “5” we obviously can distinguish between the following quantities:

• AN : The total number of all records in the test data set which actually correspond to our digit “5”.
• TP: The number of “True Positives”, i.e. the number of those cases correctly detected as “5”s.
• FP: The number of “False Positives”, i.e. the number of those cases where our ANN falsely predicts a “5”.
• FN: The number of “False Negatives”, i.e. the number of those cases where our ANN falsely predicts another digit than “5”, but where it actually should predict a “5”.

Then we can calculate the following ratios which all somehow measure “accuracy” for a specific output category:

• Precision:
TP / (TP + FP)
• Recall:
TP / ( TP + FN))
• Accuracy:
TP / AN
• F1:
TP / ( TP + 0.5*(FN + TP) )

A careful reader will (rightly) guess that the quantity “recall” corresponds to what we would naively define as “accuracy” – namely the ratio TP/AN.
From its definition it is clear that the quantity “F1” gives us a weighted average between the measures “precision” and “recall”.

How can we get these numbers for all 10 categories from our MLP after training ?

# Confusion matrix

When we want to analyze our basic error types per category we need to look at the discrepancy between predicted and actual data. This suggests a presentation in form of a matrix with all for all possible category values both in x- and y-direction. The cells of such a matrix – e.g. a cell for an actual “5” and a predicted “3” – could e.g. be filled with the corresponding FN-number.

We will later on develop our own code to solve the task of creating and displaying such a matrix. But there is a nice guy called Runqi Yang who shared some code for precisely this purpose on GitHub Gist; see https://gist.github.com/hitvoice/36c…
We can use his suggested code as it is in our context. We have already presented it above in form of the function “cm_analysis()“, which uses the pandas and seaborn libraries.

After a training run with the following parameters

```try:
ANN = myann.MyANN(my_data_set="mnist_keras", n_hidden_layers = 2,
ay_nodes_layers = [0, 70, 30, 0],
n_nodes_layer_out = 10,
my_loss_function = "LogLoss",
n_size_mini_batch = 500,
n_epochs = 1800,
n_max_batches = 2000,  # small values only for test runs
lambda2_reg = 0.2,
lambda1_reg = 0.0,
vect_mode = 'cols',
learn_rate = 0.0001,
decrease_const = 0.000001,
mom_rate   = 0.00005,
shuffle_batches = True,
print_period = 50,
figs_x1=12.0, figs_x2=8.0,
legend_loc='upper right',
b_print_test_data = True
)
except SystemExit:
print("stopped")
```

we get

and

and eventually

When I studied the last plot for a while I found it really instructive. Each of its cell outside the diagonal obviously contains the number of “False Negative” records for these two specific category values – but with respect to actual value.

What more do we learn from the matrix? Well, the numbers in the cells on the diagonal, in a row and in a
column are related to our quantities TP, FN and FP:

• Cells on the diagonal: For the diagonal we should find many correct “True Positive” values compared to the actual correct MNIST digits. (At least if all numbers are reasonably distributed across the MNIST dataset). We see that this indeed is the case. The ration of “True Positives” and the “Actual Positives” is given as a percentage and with the related numbers inside the respective cells on the diagonal.
• Cells of a row: The values in the cells of a row (without the cell on the diagonal) of the displayed matrix give us the numbers/ratios for “False Negatives” – with respect to the actual value. If you sum up the individual FN-numbers you get the total number of “False negatives”, which of course is the difference between the total number AN and the number TP for the actual category.
• Cells of a column: The column cells contain the numbers/ratios for “False Positives” – with respect to the predicted value. If you sum up the individual FN-numbers you get the total number of “False Positives” with respect to the predicted column value.

So, be a bit careful: A FN value with respect to an actual row value is a FP value with respect to the predicted column value – if the cell is one outside the diagonal!

All ratios are calculated with respect to the total actual numbers of data records for a specific category, i.e. a digit.

Looking closely we detect that our code obviously has some problems with distinguishing pictures of “5”s with pictures of “3”s, “6”s and “8”s. The same is true for “8”s and “3”s or “2s”. Also the distinction between “9”s, “3”s and “4”s seems to be difficult sometimes.

# Does the confusion matrix change due to random initial weight values and mini-batch-shuffling?

We have seen already that statistical variations have no big impact on the eventual accuracy when training converges to points in the parameter-space close to the point for the minimum of the overall cost-function. Statistical effects between to training runs stem in our case from statistically chosen initial values of the weights and the changes to our mini-batch composition between epochs. But as long as our training converges (and ends up in a global minimum) we should not see any big impact on the confusion matrix. And indeed a second run leads to:

The values are pretty close to those of the first run.

# Precision, Recall values per digit category and our own confusion matrix

Ok, we now can look at the nice confusion matrix plot and sum up all the values in a row of the confusion matrix to get the total FN-number for the related actual digit value. Or sum up the entries in a column to get the total FP-number. But we want to calculate these values from the ANN’s prediction results without looking at a plot and summation handwork. In addition we want to get the data of the confusion matrix in our own Numpy matrix array independently of foreign code. The following box displays the code for two functions, which are well suited for this task:

```# A class to print in color and bold
class color:
PURPLE = '\033[95m'
CYAN = '\033[96m'
DARKCYAN = '\033[36m'
BLUE = '\033[94m'
GREEN = '\033[92m'
YELLOW = '\033[93m'
RED = '\033[91m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'
END = '\033[
0m'

def acc_values(ay_pred_test, ay_y_test):
ay_x = ay_pred_test
ay_y = ay_y_test
# -----
#- dictionary for all false positives for all 10 digits
fp = {}
fpnum = {}
irg = range(10)
for i in irg:
key = str(i)
xfpi = np.where(ay_x==i)[0]
fpi = np.zeros((10000, 3), np.int64)

n = 0
for j in xfpi:
if ay_y[j] != i:
row = np.array([j, ay_x[j], ay_y[j]])
fpi[n] = row
n+=1

fpi_real   = fpi[0:n]
fp[key]    = fpi_real
fpnum[key] = fp[key].shape[0]

#- dictionary for all false negatives for all 10 digits
fn = {}
fnnum = {}
irg = range(10)
for i in irg:
key = str(i)
yfni = np.where(ay_y==i)[0]
fni = np.zeros((10000, 3), np.int64)

n = 0
for j in yfni:
if ay_x[j] != i:
row = np.array([j, ay_x[j], ay_y[j]])
fni[n] = row
n+=1

fni_real = fni[0:n]
fn[key] = fni_real
fnnum[key] = fn[key].shape[0]

#- dictionary for all true positives for all 10 digits
tp = {}
tpnum = {}
actnum = {}
irg = range(10)
for i in irg:
key = str(i)
ytpi = np.where(ay_y==i)[0]
actnum[key] = ytpi.shape[0]
tpi = np.zeros((10000, 3), np.int64)

n = 0
for j in ytpi:
if ay_x[j] == i:
row = np.array([j, ay_x[j], ay_y[j]])
tpi[n] = row
n+=1

tpi_real = tpi[0:n]
tp[key] = tpi_real
tpnum[key] = tp[key].shape[0]

#- We create an array for the precision values of all 10 digits
ay_prec_rec_f1 = np.zeros((10, 9), np.int64)
print(color.BOLD + "Precision, Recall, F1, Accuracy, TP, FP, FN, AN" + color.END +"\n")
print(color.BOLD + "i  ", "prec  ", "recall  ", "acc    ", "F1       ", "TP    ",
"FP    ", "FN    ", "AN" + color.END)
for i in irg:
key = str(i)
tpn = tpnum[key]
fpn = fpnum[key]
fnn = fnnum[key]
an  = actnum[key]
precision = tpn / (tpn + fpn)
prec = format(precision, '7.3f')
recall = tpn / (tpn + fnn)
rec = format(recall, '7.3f')
accuracy = tpn / an
acc = format(accuracy, '7.3f')
f1 = tpn / ( tpn + 0.5 * (fnn+fpn) )
F1 = format(f1, '7.3f')
TP = format(tpn, '6.0f')
FP = format(fpn, '6.0f')
FN = format(fnn, '6.0f')
AN = format(an,  '6.0f')

row = np.array([i, precision, recall, accuracy, f1, tpn, fpn, fnn, an])
ay_prec_rec_f1[i] = row
print (i, prec, rec, acc, F1, TP, FP, FN, AN)

return tp, tpnum, fp, fpnum, fn, fnnum, ay_prec_rec_f1

def create_cf(ay_fn, ay_tpnum):
''' fn: array with false negatives row = np.array([j, x[j], y[j]])
'''
cf = np.zeros((10, 10), np.int64)
rgi = range(10)
rgj = range(10)
for i in rgi:
key = str(i)
fn_i = ay_fn[key][ay_fn[key][:,2] == i]
for j in rgj:
if j!= i:
fn_ij = fn_i[fn_i[:,1] == j]
#print(i, j, fn_ij)
num_fn_ij = fn_ij.shape[0]
cf[i,j] = num_fn_ij
if j==i:
cf[i,j] = ay_tpnum[key]

cols=["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]
df = pd.DataFrame(cf, columns=cols, index=cols)
# print( "\n", df, "\n")
# df.style

return cf, df

```

The first function takes a array with prediction values (later on provided externally
by our “ay_predictions_test”) and compares its values with those of an y_test array which contains the actual values (later provided externally by our “ANN._y_test”). Then it uses array-slicing to create new arrays with information on all error records, related indices and the confused category values. Eventually, the function determines the numbers for AN, TP, FP and FN (per digit category) and prints the gathered information. It also returns arrays with information on records which are “True Positives”, “False Positives”, “False Negatives” and the various numbers.

The second function uses array-slicing of the array which contains all information on the “False Negatives” to reproduce the confusion matrix. It involves Pandas to produce a styled output for the matrix.

Now you can run the above code and the following one in Jupyter cells – of course, only after you have completed a training and a prediction run:

For my last run I got the following data:

We again see that especially “5”s and “9”s have a problem with FNs. When you compare the values of the last printed matrix with those in the plot of the confusion matrix above, you will see that our code produces the right FN/FP/TP-values. We have succeeded in producing our own confusion matrix – and we have all values directly available in our own Numpy arrays.

# Some images of “4”-digits with errors

We can use the arrays which we created with functions above to get a look at the images. We use the function “plot_digits()” of Aurelien Geron at handson-ml2 chapter 03 on classification to plot several images in a series of rows and columns. The code is pretty easy to understand; at its center we find the matplotlib-function “imshow()”, which we have already used in other ML articles.

We again perform some array-slicing of the arrays our function “acc_values()” (see above) produces to identify the indices of images in the “X_test”-dataset we want to look at. We collect the first 50 examples of “true positive” images of the “4”-digit, then we take the “false positives” of the 4-digit and eventually the “fales negative” cases. We then plot the images in this order:

```def plot_digits(instances, images_per_row=10, **options):
size = 28
images_per_row = min(len(instances), images_per_row)
images = [instance.reshape(size,size) for instance in instances]
n_rows = (len(instances) - 1) // images_per_row + 1
row_images = []
n_empty = n_rows * images_per_row - len(instances)
images.append(np.zeros((size, size * n_empty)))
for row in range(n_rows):
rimages = images[row * images_per_row : (row + 1) * images_per_row]
row_images.append(np.concatenate(rimages, axis=1))
image = np.concatenate(row_images, axis=0)
plt.imshow(image, cmap = mpl.cm.binary, **options)
plt.axis("off")

ay_tp, ay_tpnum, ay_fp, ay_fpnum, ay_fn, ay_
fnnum, ay_prec_rec_f1 = \
acc_values(ay_pred_test = ay_predictions_test, ay_y_test = ANN._y_test)

idx_act = str(4)

# fetching the true positives
num_tp = ay_tpnum[idx_act]
idx_tp = ay_tp[idx_act][:,[0]]
idx_tp = idx_tp[:,0]
X_test_tp = ANN._X_test[idx_tp]

# fetching the false positives
num_fp = ay_fpnum[idx_act]
idx_fp = ay_fp[idx_act][:,[0]]
idx_fp = idx_fp[:,0]
X_test_fp = ANN._X_test[idx_fp]

# fetching the false negatives
num_fn = ay_fnnum[idx_act]
idx_fn = ay_fn[idx_act][:,[0]]
idx_fn = idx_fn[:,0]
X_test_fn = ANN._X_test[idx_fn]

# plotting
# +++++++++++
plt.figure(figsize=(12,12))

# plotting the true positives
# --------------------------
plt.subplot(321)
plot_digits(X_test_tp[0:25], images_per_row=5 )
plt.subplot(322)
plot_digits(X_test_tp[25:50], images_per_row=5 )

# plotting the false positives
# --------------------------
plt.subplot(323)
plot_digits(X_test_fp[0:25], images_per_row=5 )
plt.subplot(324)
plot_digits(X_test_fp[25:], images_per_row=5 )

# plotting the false negatives
# ------------------------------
plt.subplot(325)
plot_digits(X_test_fn[0:25], images_per_row=5 )
plt.subplot(326)
plot_digits(X_test_fn[25:], images_per_row=5 )

```

The first row of the plot shows the (first) 50 “True Positives” for the “4”-digit images in the MNIST test data set. The second row shows the “False Positives”, the third row the “False Negatives”.

Very often you can guess why our MLP makes a mistake. However, in some cases we just have to acknowledge that the human brain is a much better pattern recognition machine than a stupid MLP 🙂 .

# Conclusion

With the help of a “confusion matrix” it is easy to find out for which MNIST digit-images our algorithm has major problems. A confusion matrix gives us the necessary numbers of those digits (and their images) for which the MLP wrongly predicts “False Positives” or “False Negatives”.

We have also seen that there are three quantities – precision, recall, F1 – which are useful to describe the accuracy of a classification algorithm per classification category.

We have written some code to collect all necessary information about “confused” images into our own Numpy arrays after training. Slicing of Numpy arrays proved to be useful, and matplotlib helped us to visualize examples of the wrongly classified MNIST digit-images.

In the next article
A simple program for an ANN to cover the Mnist dataset – XII – accuracy evolution, learning rate, normalization
we shall extract some more information on the evolution of accuracy during training. We shall also make use of a “clustering” technique to reduce the number of input nodes.

# A simple Python program for an ANN to cover the MNIST dataset – IX – First Tests

This brief article continues my series on a Python program for simple MLPs.

With the code for “Error Backward Propagation” we have come so far that we can perform first tests. As planned from the beginning we take the MNIST dataset as a test example. In a first approach we do not rebuild the mini-batches with each epoch. Neither do we vary the MLP setup.

What we are interested in is the question whether our artificial neural network converges with respect to final weight values during training. I.e. we want to see whether the training algorithm finds a reasonable global minimum on the cost hyperplane over the parameter space of all weights.

# Test results

We use the following parameters

```    ANN = myann.MyANN(my_data_set="mnist_keras", n_hidden_layers = 2,
ay_nodes_layers = [0, 70, 30, 0],
n_nodes_layer_out = 10,

#my_loss_function = "MSE",
my_loss_function = "LogLoss",
n_size_mini_batch = 500,

n_epochs = 1500,
n_max_batches = 2000,

lambda2_reg = 0.1,
lambda1_reg = 0.0,

vect_mode = 'cols',

learn_rate = 0.0001,
decrease_const = 0.000001,
mom_rate   = 0.00005,

figs_x1=12.0, figs_x2=8.0,
legend_loc='upper right',
b_print_test_data = True
)
```

We use two hidden layers with 70 and 30 nodes. Note that we pick a rather small learning rate, which gets diminished even further. The number of epochs (1500) is quite high; with 4 CPU threads on an i7-6700K processor training takes around 40 minutes if started from a Jupyter notebook. (Our program is not yet optimized.)

I supplemented my code with some statements to print out data for the total costs and the averaged error for mini-batches. We get the
following series of data for the last mini-batch within every 50th epoch.

```Starting epoch 1
total costs of mini_batch =  1757.1499806500506
avg total error of mini_batch =  0.17150838451718683
---------
Starting epoch 51
total costs of mini_batch =  532.5817607913532
avg total error of mini_batch =  0.034658195573307196
---------
Starting epoch 101
total costs of mini_batch =  436.67115522484687
avg total error of mini_batch =  0.023496458964699055
---------
Starting epoch 151
total costs of mini_batch =  402.381331415108
avg total error of mini_batch =  0.020836159866695597
---------
Starting epoch 201
total costs of mini_batch =  342.6296325512483
avg total error of mini_batch =  0.016565121882126693
---------
Starting epoch 251
total costs of mini_batch =  319.5995117831668
avg total error of mini_batch =  0.01533372596379799
---------
Starting epoch 301
total costs of mini_batch =  288.2201307002896
avg total error of mini_batch =  0.013799141451102469
---------
Starting epoch 351
total costs of mini_batch =  272.40526022720826
avg total error of mini_batch =  0.013499221607285198
---------
Starting epoch 401
total costs of mini_batch =  251.02417188663628
avg total error of mini_batch =  0.012696943309314687
---------
Starting epoch 451
total costs of mini_batch =  231.92274565746214
avg total error of mini_batch =  0.011152542115360705
---------
Starting epoch 501
total costs of mini_batch =  216.34658280101385
avg total error of mini_batch =  0.010692239864121407
---------
Starting epoch 551
total costs of mini_batch =  215.21791509166042
avg total error of mini_batch =  0.010999255316821901
---------
Starting epoch 601
total costs of mini_batch =  207.79645393570436
avg total error of mini_batch =  0.011123079894527222
---------
Starting epoch 651
total costs of mini_batch =  188.33965068903723
avg total error of mini_batch =  0.009868734062493835
---------
Starting epoch 701
total costs of mini_batch =  173.07625091642274
avg total error of mini_batch =  0.008942065167336382
---------
Starting epoch 751
total costs of mini_batch =  174.98264336120369
avg total error of mini_batch =  0.009714870291761567
---------
Starting epoch 801
total costs of mini_batch =  161.10229359519792
avg total error of mini_batch =  0.008844419847237179
---------
Starting epoch 851
total costs of mini_batch =  155.4186141788981
avg total error of mini_batch =  0.008244783820578621
---------
Starting epoch 901
total costs of mini_batch =  158.88876607392308
avg total error of mini_batch =  0.008970678691005138
---------
Starting epoch 951
total costs of mini_batch =  148.61870772570722
avg total error of mini_batch =  0.008124438423034456
---------
Starting epoch 1001
total costs of mini_batch =  152.16976618516264
avg total error of mini_batch =  0.009151413825781066
---------
Starting epoch 1051
total costs of mini_batch =  142.24802525081637
avg total error of mini_batch =  0.008297161160449798
---------
Starting epoch 1101
total costs of mini_batch =  137.3828515603569
avg total error of mini_batch =  0.007659755348989629
---------
Starting epoch 1151
total costs of mini_batch =  129.8472897084494
avg total error of mini_batch =  0.007254892176613871
---------
Starting epoch 1201
total costs of mini_batch =  139.30002497623792
avg total error of mini_batch =  0.007881199505625214
---------
Starting epoch 1251
total costs of mini_batch =  138.0323454321882
avg total error of mini_batch =  0.00807373439996105
---------
Starting epoch 1301
total costs of mini_batch =  117.95701570484076
avg total error of mini_batch =  0.006378071703153664
---------
Starting epoch 1351
total costs of mini_batch =  125.
71869046937177
avg total error of mini_batch =  0.0072716189968114265
---------
Starting epoch 1401
total costs of mini_batch =  117.3485602627176
avg total error of mini_batch =  0.006291182169676069
---------
Starting epoch 1451
total costs of mini_batch =  118.09317470010767
avg total error of mini_batch =  0.0066519021636054195
---------
Starting epoch 1491
total costs of mini_batch =  112.69566736699439
avg total error of mini_batch =  0.006151660466611035

------
Total training Time_CPU:  2430.0504785089997
```

We can display the results also graphically; a code fragemnt to do this, may look like follows in a Jupyter cell:

```# Plotting
# **********
num_epochs  = ANN._ay_costs.shape[0]
num_batches = ANN._ay_costs.shape[1]
num_tot = num_epochs * num_batches

ANN._ay_costs = ANN._ay_costs.reshape(num_tot)
ANN._ay_theta = ANN._ay_theta .reshape(num_tot)

#sizing
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 12
fig_size[1] = 5

# Two figures
# -----------
fig1 = plt.figure(1)
fig2 = plt.figure(2)

# first figure with two plot-areas with axes
# --------------------------------------------

ax1_1.plot(range(len(ANN._ay_costs)), ANN._ay_costs)
ax1_1.set_xlim (0, num_tot+5)
ax1_1.set_ylim (0, 2000)
ax1_1.set_xlabel("epochs * batches (" + str(num_epochs) + " * " + str(num_batches) + " )")
ax1_1.set_ylabel("costs")

ax1_2.plot(range(len(ANN._ay_theta)), ANN._ay_theta)
ax1_2.set_xlim (0, num_tot+5)
ax1_2.set_ylim (0, 0.2)
ax1_2.set_xlabel("epochs * batches (" + str(num_epochs) + " * " + str(num_batches) + " )")
ax1_2.set_ylabel("averaged error")

```

We get:

This looks quite promising!
The vertical spread of both curves is due to the fact that we plotted cost and error data for each mini-batch. As we know the cost hyperplanes of the batches differ from each other and the hyperplane of the total costs for all training data. So do the cost and error values.

# Secondary test: Rate of correctly and wrongly predicted values of the training and the test data sets

With the following code in a Jupyter cell we can check the relative percentage of correctly predicted MNIST numbers for the training data set and the test data set:

```# ------ all training data
# *************************
size_set = ANN._X_train.shape[0]

li_Z_in_layer_test  = [None] * ANN._n_total_layers
li_Z_in_layer_test[0] = ANN._X_train
# Transpose
ay_Z_in_0T       = li_Z_in_layer_test[0].T
li_Z_in_layer_test[0] = ay_Z_in_0T
li_A_out_layer_test  = [None] * ANN._n_total_layers

ANN._fw_propagation(li_Z_in = li_Z_in_layer_test, li_A_out = li_A_out_layer_test, b_print = True)
Result = np.argmax(li_A_out_layer_test[3], axis=0)
Error = ANN._y_train - Result
acc_train = (np.sum(Error == 0)) / size_set
print ("total accuracy for training data = ", acc_train)

# ------ all test data
# *************************
size_set = ANN._X_test.shape[0]

li_Z_in_layer_test  = [None] * ANN._n_total_layers
li_Z_in_layer_test[0] = ANN._X_test
# Transpose
ay_Z_in_0T       = li_Z_in_layer_test[0].T
li_Z_in_layer_test[0] = ay_Z_in_0T
li_A_out_layer_test  = [None] * ANN._n_total_layers

ANN._fw_propagation(li_Z_in = li_Z_in_layer_test, li_A_out = li_A_out_layer_test, b_print = True)
Result = np.argmax(li_A_out_layer_
test[3], axis=0)
Error = ANN._y_test - Result
acc_test = (np.sum(Error == 0)) / size_set
print ("total accuracy for test data = ", acc_test)
```

“acc” stands for “accuracy”.

We get

total accuracy for training data = 0.9919
total accuracy for test data        = 0.9645

So, there is some overfitting – but not much.

# Conclusion

Our training algorithm and the error backward propagation seem to work.

The real question is, whether we produced the accuracy values really efficiently: In our example case we needed to fix around 786*70 + 70*30 + 30*10 = 57420 weight values. This is close to the total amount of training data (60000). A smaller network with just one hidden layer would require much fewer values – and the training would be much faster regarding CPU time.

So, in the next article
A simple program for an ANN to cover the Mnist dataset – X – mini-batch-shuffling and some more tests
we shall extend our tests to different setups of the MLP.

# The moons dataset and decision surface graphics in a Jupyter environment – V – a class for plots and some experiments

We proceed with our exercises on the moons dataset. This series of articles is intended for readers which – as me – are relatively new both to Python and Machine Learning. By working with examples we try to extend our knowledge about the tools “Juypter notebooks” and “Eclipse/PyDev” for setting up experiments which require plots for an interpretation.

We have so far used a Jupyter notebook to perform some initial experiments for creating and displaying a decision surface between the moons dataset clusters with an algorithm called “LinearSVC”. If you followed everything I described in the last articles

you may now have gathered around 20 different cells with code. Part of the cells’ code was used to learn some basics about contour and scatter plots. This code is now irrelevant for further experiments. Time to consolidate our plotting knowledge.

In the last article I promised to put plot-related code into a Python class. The class itself can become a part of a Python module – which we in turn can import into the code of Jupyter notebook. By doing this we can reduce the number of cells in a notebook drastically. The importing of external classes is thus helpful for concentrating on “real” data analysis experiments with different learning and predicting algorithms and/or a variation of their parameters.

I assume that you have some basic knowledge on how classes are build in Python. If not please see an introductory book on Python 3.

# A class for plotting simple decision surfaces in a 2-dimensional space

In the articles

I had shown how to set up Eclipse PyDev to be used in the context of a Python virtual environment. In our special environment “ml1” used by our Jupyter notebook “moons1.ipynb” we have the following directory structure:

“ml1” has a sub-directory “mynotebooks” which contains notebook files as our “moons1.ipynb”. To provide a place for other general code there we open up a directory “mycode“. In it we create a file “myplots.py” for a module
myplots“, which shall comprise our own Python classes for plotting.

We distribute the code discussed in the last 2 articles of this series into methods of a class “MyDecisionPlot“; we put the following code into our file “myplots.py” with the Pydev editor.

```'''
Created on 15.07.2019
Module to gather classes for plotting
@author: rmo
'''
import numpy as np
import sys
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
import matplotlib.patches as mpat
#from matplotlib import ticker, cm
#from mpl_toolkits import mplot3d

class MyDecisionPlot():
'''
This class allows for
1) decision surfaces in 2D (x1,x2) planes
2) plotting scatter data of datapoints
'''

def __init__(self, X, y, predictor = None, ax_x_delta=1.0, ax_y_delta=1.0,
mesh_res=0.01, alpha=0.4, bcontour=1, bscatter=1,
figs_x1=12.0, figs_x2=8.0,
x1_lbl='x1', x2_lbl='x2',
legend_loc='upper right'
):
'''
Constructor of MyDecisionPlot
Input:
X: Input array (2D) for learning- and predictor-algorithm as VSM
y: result data for learning- and predictor-algorithm
ax_x_delta, ax_y_delta : delta for extension of both axis beyond the given X, y-data
mesh_res: resolution of the mesh spanned in the (x1,x2)-plane (x_max-x_min) * mesh_res
alpha:  transparency  of contours
bcontour: 0: Do not plot contour areas 1: plot contour areas
bscatter: 0: Do not plot scatter points of the input data sample 1: Plot scatter plot of the input data sample
figs_x1: plot size in x1 direction
figs_x2: plot size in x2 direction
x1_lbl, x2_lbl : axes lables
legend_loc : position of a legend
Ouptut:
Internal: self._mesh_points (mesh points created)
External: Plots - shoukd cone up automatically in Jupyter notebooks
'''

# initiate some internal variables
self._x1_min = 0.0
self._x1_max = 1.0
self._x2_min = 0.0
self._x2_max = 1.0

# Alternatives to resize plots
# 1: just resize figure  2: resize plus create subplots() [figure + axes]
self._plot_resize_alternative = 2

# X (x1,x2)-Input array
self.__X = X
self.__y = y
self._Z  = None

# predictor = algorithm to create y-values for new (x1,x2)-data points
self._predictor = predictor

# meshdata
self._resolution = mesh_res   # resolution of the mesh
self.__ax_x_delta = ax_x_delta
self.__ax_y_delta = ax_y_delta
self._alpha = alpha
self._bscatter = bscatter
self._bcontour = bcontour

self._xm1 = None
self._xm2 = None
self._mesh_points = None

# set marker array and define colormap
self._markers = ('s', 'x', 'o', '^', 'v')
self._colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
self._cmap = ListedColormap(self._colors[:len(np.unique(y))])

self._x1_lbl = x1_lbl
self._x2_lbl = x2_lbl
self._legend_loc = legend_loc

# Plot-sizing
self._figs_x1 = figs_x1
self._figs_x2 = figs_x2
self._fig = None
self._ax  = None
# alternative 2 does resizing and (!) subplots()
self.initiate_and_resize_plot(self._plot_resize_alternative)

r
# create mesh in x1, x2 - direction with mesh_res resolution
# meshpoint-array will be creted with right dimension for plotting data
self.create_mesh()
# Array meshpoints should exist now

if(self._bcontour == 1):
try:
if self._predictor == None:
raise ValueError
except ValueError:
print ("You must provide an algorithm = 'predictor' as parameter")
#sys.exit(0)
sys.exit()

self.make_contourplot()
else:
if (self._bscatter == 1):
self.make_scatter_plot()

# method to create a dense mesh in the (x1,x2)-plane
def create_mesh(self):
'''
Method to create a dense mesh in an (x1,x2) plane
Input: x1, x2-data are constructed from array self.__X
Output: A suitable array of meshpoints is written to self._mesh_points()
'''
try:
self._x1_min = self.__X[:, 0].min()
self._x1_max = self.__X[:, 0].max()
except ValueError: # as e:
print ("cannot determine x1_min = X[:,0].min() or x1_max = X[:,0].max()")
try:
self._x2_min = self.__X[:, 1].min()
self._x2_max = self.__X[:, 1].max()
except ValueError: # as e:
print ("cannot determine x2_min =X[:,1].min()) or x2_max = X[:,1].max()")

self._x1_min, self._x1_max = self._x1_min - self.__ax_x_delta, self._x1_max + self.__ax_x_delta
self._x2_min, self._x2_max = self._x2_min - self.__ax_x_delta, self._x2_max + self.__ax_x_delta

#create mesh data (x1, x2)
self._xm1, self._xm2 = np.meshgrid( np.arange(self._x1_min, self._x1_max, self._resolution),
np.arange(self._x2_min, self._x2_max, self._resolution))

#print (self._xm1)
# for hasattr the variable cannot be provate !
#print ("xm1 is there: ", hasattr(self,'_xm1' ) )

# ordering and transposing of the mesh-matrix
# for understanding the structure and transpose requirements see linux-blog.anracom.con
self._mesh_points = np.array([self._xm1.ravel(), self._xm2.ravel()]).T

try:
if( hasattr(self, '_mesh_points') == False ):
raise ValueError
except ValueError:
print("The required array mesh_points has not been created!")
exit

# -------------
# Some helper functions to change valus on the fly if necessary

def set_mesh_res(self, new_mesh_res):
self._resolution = new_mesh_res

def change_predictor(self, new_predictor):
self._predictor = new_predictor

def change_alpha(self, new_alpha):
self._alpha = new_alpha

def change_figs(self, new_figs_x1, new_figs_x2):
self._figs_x1 = new_figs_x1
self._figs_x2 = new_figs_x2

# -------------
# method to get subplots and resize the figure
# -------------
def initiate_and_resize_plot(self, alternative=2 ):

# Alternative 1 to resize plots - works afte rimports to Jupyter notebooks, too
if alternative == 1:
self._fig_size = plt.rcParams["figure.figsize"]
self._fig_size[0] = self._figs_x1
self._fig_size[1] = self._figs_x2
plt.rcParams["figure.figsize"] = self._fig_size

# Not working for sizing if plain subplots() is used
#plt.figure(figsize=(self._figs_x1 , self._figs_x2))
#self._fig, self._ax = plt.subplots()
put the figsize-parameter into the subplots() function

# Alternative 2 for resizing plots and using subplots()
# we use this alternative as we may need the axes later for 3D plots
if alternative == 2:
self._fig, self._ax = plt.subplots(figsize=(self._figs_x1 , self._figs_x2))

# ***********************************************

# -------------
# method to create contour plots
# -------------
def make_contourplot(self):
'''
Method to create a contourplot based on a dense mesh of points in an (x1,x2) plane
and a predictor algorithm which allows for value calculations
'''

try:
if( not hasattr(self, '_mesh_points') ):
raise ValueError
except ValueError:
print("The required array mesh_points has not been created!")
exit

# Predict values for all meshpoints
try:
self._Z = self._predictor.predict(self._mesh_points)
except AttributeError:
print("method predictor.predict() does not exist")

#reshape
self._Z = self._Z.reshape(self._xm1.shape)
#print (self._Z)

# make the plot
plt.contourf(self._xm1, self._xm2, self._Z, alpha=self._alpha, cmap=self._cmap)

# create a scatter-plot of data sample as well
if (self._bscatter == 1):
self.make_scatter_plot()

self.make_plot_legend()

# -------------
# method to create a scatter plot of the data sample
# -------------
def make_scatter_plot(self):
alpha2 = self._alpha + 0.4
if (alpha2 > 1.0 ):
alpha2 = 1.0
for idx, yv in enumerate(np.unique(self.__y)):
plt.scatter(x=self.__X[self.__y==yv, 0], y=self.__X[self.__y==yv, 1],
alpha=alpha2, c=[self._cmap(idx)], marker=self._markers[idx], label=yv)

if self._bscatter == 0:
self._bscatter = 1

self.make_plot_legend()

# -------------
# method to add a legend
# -------------
def make_plot_legend(self):
plt.xlim(self._x1_min, self._x1_max)
plt.ylim(self._x2_min, self._x2_max)
plt.xlabel(self._x1_lbl)
plt.ylabel(self._x2_lbl)

# we have two cases
#     a) for a scatter plot we have array values where the legend is taken from automatically
#     b) For apure contourplot we need to prepare a legend with "patches" (kind og labels) used by pyplot.legend()
if (self._bscatter == 1):
plt.legend(loc=self._legend_loc)
else:
red_patch  = mpat.Patch(color='red',  label='0', alpha=0.4)
blue_patch = mpat.Patch(color='blue', label='1', alpha=0.4)
plt.legend(handles=[red_patch, blue_patch], loc=self._legend_loc)

```

This certainly is no masterpiece of superior code design; so you may change it. However, the code is good enough for our present purposes.

Note that we have to import basic Python modules into the namespace of this module. This is conventionally done at the top of the file.

Note also the 2 alternatives offered for resizing a plot! Both work also for “inline” plotting in a Jupyter environment; see the text below.

# Using the module “myplots” in a Jupyter notebook

In a terminal we move to our example directory “/projekte/GIT/ai/ml1” and start our virtual Python environment:

```myself@mytux: /projekte/GIT/ai/ml1> source bin/activate
(ml1) myself@mytux:/projekte/GIT/ai/ml1>
jupyter notebook
[I 11:46:15.942 NotebookApp] Serving notebooks from local directory: /projekte/GIT/ai/ml1
[I 11:46:15.942 NotebookApp] The Jupyter Notebook is running at:
....
....
```

We then open our old notebook “moons1” and save it under the name “moons2”:

We delete all old cells. Then we change the first cell of our new notebook to the following contents:

```import imp
%matplotlib inline
from mycode import myplots

from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import PolynomialFeatures
from sklearn.svm import LinearSVC
from sklearn.svm import SVC
```

You see that we imported the “myplots” module from the “package” directory “mycode”. The Jupyter notebook will find the path to “mycode” as long as we have opened the notebook on a higher directory level. Which we did … see above.

Note the statement with a so called “magic command” for our notebook:

%matplotlib inline

There are many other “magic commands” and parameters which you can study at
Ipython magic commands

The command “%matplotlib inline” informs the notebook to show plots created by any imported modules “inline”, i.e. in the visual context of the affected cell. This specific magic directive should be issued before matplotlib/pyplot is imported into any of the external python modules which we in turn import into the cell code.

A call of plt.show() in our class’s method “make_contourplot()” is no longer necessary afterwards.

If we, however, want to resize the plots in comparison to Jupyter standard values we have to control this by parameters of our plot class. Such parameters are offered already in the interface of the class; but they can be changed by a method “change_figs(new_figs_x1, new_figs_x2)), too.

In a second cell of our new notebook we now prepare the moon data set

```X, y = make_moons(noise=0.1, random_state=5)
```

Further cells will be used for quick individual experiments with the moons dataset. E.g.:

```imp.reload(myplots)
X, y = make_moons(noise=0.1, random_state=5)

# contour plot of the moons data - with scatter plot / training with LinearSVC
polynomial_degree = 3
max_iterations = 6000
polynomial_svm_clf = Pipeline([
("poly_features", PolynomialFeatures(degree=polynomial_degree)),
("scaler", StandardScaler()),
("svm_clf", LinearSVC(C=18, loss="hinge", max_iter=max_iterations))
])
#training
polynomial_svm_clf.fit(X, y)

#plotting
MyPlt = myplots.MyDecisionPlot(X, y, polynomial_svm_clf, bcontour = 1, bscatter=1 )

```

The last type of cell just handles the setup and training of our specific algorithm “LinearSVC” and delegates plotting to our class.

# Testing the new notebook

A test of the 3 cells in their order gives

All well! This is exactly what we hoped to achieve.

# Three experiments with a varying polynomial degree

As we now have a simple and compact cell template for experiments we add three
further cells where we vary the degree of the polynomials for LinearSVC. Below we show the results for degree 6, 7 and for comparison also for a degree of 2.

On a modern computer it should take almost no time to produce the results. (We shall learn how to measure CPU-time in the next article).

We understand that we at least need a polynomial of degree 3 to separate the data reasonably. However, polynomials with an even degree (>= 4) separate the 2 data regions very differently compared to polynomials with an uneven degree (>=3) in outer areas of the (x1,x2)-plane (where no training data were placed):

Depending on the polynomial degree our Linear SVC algorithm obviously extrapolates in different ways to regions without such data. And we have no clue which of the polynomials is the better choice …

This poses a warning for the use of AI in general:

We should be extremely careful to trust predictions of any AI algorithm in parameter regions for which the algorithm must extrapolate – as long as we have no real data points available there to discriminate between multiple solutions that all work equally well in regions with given training data.

# Would general modules be imported twice in a Jupyter cell – via the import of an external module, which itself includes import statements, and a direct import statement in a Jupyter cell?

The question posed in the headline is an interesting one for me as a Python beginner. Coming from other programming languages I get a bit nervous when I see the possibility for import statements referring to a specific module both in another already imported module and by a direct import statement afterwards in a Jupyter cell. E.g. we import numpy indirectly via our “myplots” module, but we could and sometimes must import it in addition directly in our Jupyter cell.

Note that we must make the general modules as numpy, matplotlib, etc. available in the namespace environment of our private module “myplots”. Otherwise the functions cannot be used there. The Jupyter cell, however, corresponds to an independent namespace environment. So, an import may indeed be required there, too, if we plan to handle numpy arrays via code in such a cell.

Reading a bit about the Python import logic on the Internet reveals that a double import or overwriting should not take place; instead an already imported piece of code only gets multiple references in the various specific namespaces of different code environments.

We can test this with the following code in a Jupyter cell:

Note that numpy is also imported by our “myplots”. However, the length of the list produced by sys.modules.keys(), which enumerates all possible module reference points (including imports) does not change.

What if we in the
course of or experiments need to change the code of our imported module? Then we need to reload the module in a Jupyter cell before we run it again. In our case (Python 3!) this can be done by the command

As the code of our first cell reveals, the general package “imp” must have been imported before we can use its reload-function.

# Conclusion

We saw that it is easy to use our own modules with Python class code, which we created in an Eclipse/PyDev environment, in a Jupyter notebook. We just use Python’s standard import mechanism in Jupyter cells to get access to our classes. We applied this to a module with a simple class for preparing decision surface plots based on contour and scatter plot routines of matplotlib. We imported the module in a new Jupyter notebook.

Plots created by our imported class-methods were displayed correctly within the cell environment as soon as we used the magic directive “%matplotlib inline” within our notebook.

In addition we used our new notebook with its compact cell structure for some quick experiments: We set different values for the polynomial degree parameter of our LinearSVC algorithm. We saw that the results of algorithms should be interpreted with caution if the predictions refer to data points in regions of the representation or feature space which were not at all covered by the data sets for training.

The prediction results (= extrapolations) of even one and the same algorithm with different parameters may deviate strongly in such regions – and we may not have any reliable indications to decide which of the predictions and their basic parameter setting are better.

In the next article of this series

The moons dataset and decision surface graphics in a Jupyter environment – VI – Kernel-based SVC algorithms

we shall have a look at what kind of decision surface some other SVM algorithms than LinearSVC create for the moons dataset. In addition shall briefly discuss kernel based algorithms.

|