Nvidia GPU-support of Tensorflow/Keras on Opensuse Leap 15

When you start working with Google's Tensorflow on multi-layer and "deep learning" artificial neural networks the performance of the required mathematical operations may sooner or later become important. One approach to better performance is the use of a GPU (or multiple GPUs) instead of a CPU. Personally, I am not yet in a situation where GPU support is really required. My experimental CNNs are too small, yet. But starting with Keras and Tensorflow is a good point to cover the use of a GPU on my Opensuse Leap 15 systems anyway. Actually, it is also helpful for some tasks in security related environments, too. One example is testing the quality of passphrases for encryption. With JtR you may gain a factor of 10 in performance. It is interesting, how much faster an old 960 GTX card will be for a simple Tensorflow test application than my i7 CPU.

I have used Nvidia GPUs almost all my Linux life. To get GPU support for Nvidia graphics cards you need to install CUDA in its present version. This is 10.1 in August 2019. You get download and install information for CUDA at
https://developer.nvidia.com/cuda-zone => https://developer.nvidia.com/cuda-downloads
For an RPM for the x86-64 architecture and Opensuse Leap see:
https://developer.nvidia.com/cuda-downloads?....

Installation of "CUDA" and "cudcnn"

You may install the downloaded RPM (in my "case cuda-repo-opensuse15-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm") via YaST. After this first step you in a second step install the meta-packet named "cuda", which is available in YaST at this point. Or just install all other packets with "cuda" in the name (with the exception of the source code and dev-packets) via YaST.

A directory "/usr/local/cuda" will be built; its entries are soft links to files in a directory "/usr/local/cuda-10.1".

Note the "include" and the "lib64" sub-directories! After the installation, also links should exist in the central "/usr/lib64"-directory pointing to the files in "/usr/local/cuda/lib64".

Note from the file-endings that the particular present version [Aug. 2019) of the files may be something like "10.1.168".

Another important point is that you need to install "cudnn" (cudnn-10.1-linux-x64-v7.6.2.24.tgz) - a Nvidia specific library for certain Deep Learning program elements, which shall be executed on Nvidia GPU chips. You get these files via "https://developer.nvidi.com/cudnn". Unfortunately, you must become member of the Nvidia developer community to get access to these special files. After you downloaded the tgz-file and expanded it, you find some directories "include" and "lib64" with relevant files. You just copy these files (as user root) into the directories "/usr/local/cuda/include" and "/usr/local/cuda/lib64", respectively. Check the owner/group and rights of the copied files afterwards and change them to root/root and standard rights - just as given for the other files in teh target directories.

The final step is the follwoing:
Create links by dragging the contents of "/usr/local/cuda/include" to "/usr/include" and chose the option "Link here". Do the same for the files of "/usr/local/cuda/lib64" with "/usr/lib64" as the target directory. If you look at the link-directories of the files now in "usr/include" and "usr/lib64" you see exactly which files were given by the CUDA and cudcnn installation.

Additional libraries
In case you want to use Keras it is recommended to install the "openblas" libraries including the development packages on the Linux OS level. On an Opensuse system just search for packages with "openblas" and install them all. The same is true for the h5py-libraries. In your virtual python environment execute:
< p style="margin-left:50px;"pip3 install --upgrade h5py

Problems with errors regarding missing CUDA libraries after installation

Two stupid things may happen after this straight-forward installation :

  • The link structure between "/usr/lib64" and the files in "/usr/local/cuda/include" and "/usr/local/cuda/lib64" may be incomplete.
  • Although there are links from files as "libcufftw.so.10" to something like "libcufftw.so.10.1.168" some libraries and TensorFlow components may expect additional links as "libcufftw.so.10.0" to "libcufftw.so.10.1.168"

Both points lead to error messages when I tried to use GPU related test statements on a PyDEV console or Jupyter cell. Watch out for error messages which tell you about errors when opening specific libraries! In the case of Jupyter you may find such messages on the console or terminal window from which you started your test.

A quick remedy is to use a file-manager as "dolphin" as user root, mark all files in "/usr/local/cuda/include" and "usr/local/cuda/lib64" and place them as (soft) links into "/usr/include" and "/usr/lib64", respectively. Then create additional links there for the required libraries "libXXX.so.10.0" to "libXXX.so.10.1.168", where "XXX" stands for some variable part of the file name.

A simple test with Keras and the mnist dataset

I assume that you have installed the packages for tensorflow, tensorflow-gpu (!) and keras with pip3 in your Python virtualenv. Note that the package "tensorflow-gpu" MUST be installed after "tensorflow" to make the use of the GPU possible.

Then a test with a simple CNN for the "mnist" datatset can deliver information on performance differences :

Cell 1 of a Jupyter notebook:

import time 
import tensorflow as tf
from keras import backend as K
from tensorflow.python.client import device_lib
from keras.datasets import mnist
from keras import models
from keras import layers
from keras.utils import to_categorical

# function to provide CPU/GPU information 
# ---------------------------------------
def get_CPU_GPU_details():
    print("GPU ? ", tf.test.is_gpu_available())
    tf.test.gpu_device_name()
    print(device_lib.list_local_devices())

# information on available CPUs/GPUs
# --------------------------------------
if tf.test.is_gpu_available(
    cuda_only=False,
    min_cuda_compute_capability=None):
    print ("GPU is available")
get_CPU_GPU_details()

# Setting a parameter GPU or CPU usage 
#--------------------------------------
#gpu = False 
gpu = True
if gpu: 
    GPU = True;  CPU = False; num_GPU = 1; num_CPU = 1
else: 
    GPU = False; CPU = True;  num_CPU = 1; num_GPU = 0
num_cores = 6

# control of GPU or CPU usage in the TF environment
# -------------------------------------------------
# See the literature links at the article's end for more information  

config = tf.ConfigProto(intra_op_parallelism_threads=num_cores,
                        inter_op_parallelism_threads=num_cores, 
                        allow_soft_placement=True,
                        device_count = {'CPU' : num_CPU,
                                        'GPU' : num_GPU}, 
                        log_device_placement=True

                       )
config.gpu_options.per_process_gpu_memory_fraction=0.4
config.gpu_options.force_gpu_compatible = True
session = tf.Session(config=config)
K.set_session(session)

#--------------------------
# Loading the mnist datatset via Keras 
#--------------------------
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()
network = models.Sequential()
network.add(layers.Dense(512, activation='relu', input_shape=(28*28,)))
network.add(layers.Dense(10, activation='softmax'))
network.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
train_images = train_images.reshape((60000, 28*28))
train_images = train_images.astype('float32') / 255
test_images = test_images.reshape((10000, 28*28))
test_images = test_images.astype('float32') / 255
train_labels = to_categorical(train_labels)
test_labels = to_categorical(test_labels)

Output of the code in cell 1:

GPU is available
GPU ?  True
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 17801622756881051727
, name: "/device:XLA_GPU:0"
device_type: "XLA_GPU"
memory_limit: 17179869184
locality {
}
incarnation: 6360207884770493054
physical_device_desc: "device: XLA_GPU device"
, name: "/device:XLA_CPU:0"
device_type: "XLA_CPU"
memory_limit: 17179869184
locality {
}
incarnation: 7849438889532114617
physical_device_desc: "device: XLA_CPU device"
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 2115403776
locality {
  bus_id: 1
  links {
  }
}
incarnation: 4388589797576737689
physical_device_desc: "device: 0, name: GeForce GTX 960, pci bus id: 0000:01:00.0, compute capability: 5.2"
]

Note the control settings for GPU usage via the parameter gpu and the variable "config". If you do NOT want to use the GPU execute

config = tf.ConfigProto(device_count = {'GPU': 0, 'CPU' : 1})

Information on other control parameters which can be used together with "tf.ConfigProto" is provided here:
https://stackoverflow.com/questions/40690598/can-keras-with-tensorflow-backend-be-forced-to-use-cpu-or-gpu-at-will

Cell 2 of a Jupyter notebook for performance measurement during training:

start_c = time.perf_counter()
with tf.device("/GPU:0"):
    network.fit(train_images, train_labels, epochs=5, batch_size=30000)
end_c = time.perf_counter()
if CPU: 
    print('Time_CPU: ', end_c - start_c)  
else:  
    print('Time_GPU: ', end_c - start_c)  

Output of the code in cell 2 :

Epoch 1/5
60000/60000 [==============================] - 0s 3us/step - loss: 0.5817 - acc: 0.8450
Epoch 2/5
60000/60000 [==============================] - 0s 3us/step - loss: 0.5213 - acc: 0.8646
Epoch 3/5
60000/60000 [==============================] - 0s 3us/step - loss: 0.4676 - acc: 0.8832
Epoch 4/5
60000/60000 [==============================] - 0s 3us/step - loss: 0.4467 - acc: 0.8837
Epoch 5/5
60000/60000 [==============================] - 0s 3us/step - loss: 0.4488 - acc: 0.8726
Time_GPU:  0.7899935730001744

Now change the following lines in cell 1

 
...
gpu = False 
#gpu = True 
...

Executing the code in cell 1 and cell 2 then gives:

Epoch 1/5
60000/60000 [==============================] - 0s 6us/step - loss: 0.4323 - acc: 0.8802
Epoch 2/5
60000/60000 [==============================] - 0s 7us/step - loss: 0.3932 - acc: 0.8972
Epoch 3/5
60000/60000 [==============================] - 0s 6us/step - loss: 0.3794 - acc: 0.8996
Epoch 4/5
60000/60000 [==============================] - 0s 6us/step - loss: 0.3837 - acc: 0.8941
Epoch 5/5
60000/60000 [==============================] - 0s 6us/step - loss: 0.3830 - acc: 0.8908
Time_CPU:  1.9326397939985327

Thus the GPU is faster by a factor of 2.375 !
At least for the chosen batch size of 30000! You should play a bit around with the batch size to understand its impact.
2.375 is not a big factor - but I have a relatively old GPU (GTX 960) and a relatively fast CPU i7-6700K mit 4GHz Taktung: So I take what I get 🙂 . A GTX 1080Ti would give you an additional factor of around 4.

Watching GPU usage during Python code execution

A CLI command which gives you updated information on GPU usage and memory consumption on the GPU is

nvidia-smi -lms 250

It gives you something like

Mon Aug 19 22:13:18 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960     On   | 00000000:01:00.0  On |                  N/A |
| 20%   44C    P0    33W / 160W |   3163MiB /  4034MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      4124      G   /usr/bin/X                                   610MiB |
|    0      4939      G   kwin_x11                                      54MiB |
|    0      4957      G   /usr/bin/krunner                               1MiB |
|    0      4959      G   /usr/bin/plasmashell                         195MiB |
|    0      5326      G   /usr/bin/akonadi_archivemail_agent             2MiB |
|    0      5332      G   /usr/bin/akonadi_imap_resource                 2MiB |
|    0      5338      G   /usr/bin/akonadi_imap_resource                 2MiB |
|    0      5359      G   /usr/bin/akonadi_mailfilter_agent              2MiB |
|    0      5363      G   /usr/bin/akonadi_sendlater_agent               2MiB |
|    0      5952      C   /usr/lib64/libreoffice/program/soffice.bin    38MiB |
|    0      8240      G   /usr/lib64/firefox/firefox                     1MiB |
|    0     13012      C   /projekte/GIT/ai/ml1/bin/python3            2176MiB |
|    0     14233      G   ...uest-channel-token=14555524607822397280    62MiB |
+-----------------------------------------------------------------------------+

During code execution some of the displayed numbers - e.g for GPU-Util, GPU memory Usage - will start to vary.

Links

https://medium.com/@liyin2015/tensorflow-cpus-and-gpus-configuration-9c223436d4ef
https://www.tensorflow.org/beta/guide/using_gpu
https://stackoverflow.com/questions/40690598/can-keras-with-tensorflow-backend-be-forced-to-use-cpu-or-gpu-at-will
https://stackoverflow.com/questions/42706761/closing-session-in-tensorflow-doesnt-reset-graph
http://www.science.smith.edu/dftwiki/index.php/Setting up Tensorflow 1.X on Ubuntu 16.04 w/ GPU support
https://hackerfall.com/story/which-gpus-to-get-for-deep-learning
https://towardsdatascience.com/measuring-actual-gpu-usage-for-deep-learning-training-e2bf3654bcfd

 

The moons dataset and decision surface graphics in a Jupyter environment – V – a class for plots and some experiments

We proceed with our exercises on the moons dataset. This series of articles is intended for readers which - as me - are relatively new both to Python and Machine Learning. By working with examples we try to extend our knowledge about the tools "Juypter notebooks" and "Eclipse/PyDev" for setting up experiments which require plots for an interpretation.

We have so far used a Jupyter notebook to perform some initial experiments for creating and displaying a decision surface between the moons dataset clusters with an algorithm called "LinearSVC". If you followed everything I described in the last articles

The moons dataset and decision surface graphics in a Jupyter environment – I
The moons dataset and decision surface graphics in a Jupyter environment – II – contourplots
The moons dataset and decision surface graphics in a Jupyter environment – III – Scatter-plots and LinearSVC
The moons dataset and decision surface graphics in a Jupyter environment – IV – plotting the decision surface

you may now have gathered around 20 different cells with code. Part of the cells' code was used to learn some basics about contour and scatter plots. This code is now irrelevant for further experiments. Time to consolidate our plotting knowledge.

In the last article I promised to put plot-related code into a Python class. The class itself can become a part of a Python module - which we in turn can import into the code of Jupyter notebook. By doing this we can reduce the number of cells in a notebook drastically. The importing of external classes is thus helpful for concentrating on "real" data analysis experiments with different learning and predicting algorithms and/or a variation of their parameters.

I assume that you have some basic knowledge on how classes are build in Python. If not please see an introductory book on Python 3.

A class for plotting simple decision surfaces in a 2-dimensional space

In the articles

Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – I
Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – II
Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – III

I had shown how to set up Eclipse PyDev to be used in the context of a Python virtual environment. In our special environment "ml1" used by our Jupyter notebook "moons1.ipynb" we have the following directory structure:

"ml1" has a sub-directory "mynotebooks" which contains notebook files as our "moons1.ipynb". To provide a place for other general code there we open up a directory "mycode". In it we create a file "myplots.py" for a module "myplots", which shall comprise our own Python classes for plotting.

We distribute the code discussed in the last 2 articles of this series into methods of a class "MyDecisionPlot"; we put the following code into our file "myplots.py" with the Pydev editor.

'''
Created on 15.07.2019
Module to gather classes for plotting
@author: rmo
'''
import numpy as np
import sys
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
import matplotlib.patches as mpat 
#from matplotlib import ticker, cm
#from mpl_toolkits import mplot3d

class MyDecisionPlot():
    '''
    This class allows for 
    1) decision surfaces in 2D (x1,x2) planes 
    2) plotting scatter data of datapoints 
    '''


    def __init__(self, X, y, predictor = None, ax_x_delta=1.0, ax_y_delta=1.0, 
                 mesh_res=0.01, alpha=0.4, bcontour=1, bscatter=1, 
                 figs_x1=12.0, figs_x2=8.0, 
                 x1_lbl='x1', x2_lbl='x2',   
                 legend_loc='upper right'
                 ):
        '''
        Constructor of MyDecisionPlot
        Input: 
            X: Input array (2D) for learning- and predictor-algorithm as VSM 
            y: result data for learning- and predictor-algorithm
            ax_x_delta, ax_y_delta : delta for extension of both axis beyond the given X, y-data
            mesh_res: resolution of the mesh spanned in the (x1,x2)-plane (x_max-x_min) * mesh_res
            alpha:  transparency  of contours 
            bcontour: 0: Do not plot contour areas 1: plot contour areas 
            bscatter: 0: Do not plot scatter points of the input data sample 1: Plot scatter plot of the input data sample 
            figs_x1: plot size in x1 direction 
            figs_x2: plot size in x2 direction 
            x1_lbl, x2_lbl : axes lables 
            legend_loc : position of a legend
        Ouptut:
            Internal: self._mesh_points (mesh points created) 
            External: Plots - shoukd cone up automatically in Jupyter notebooks 
        '''
        
                
        # initiate some internal variables 
        self._x1_min = 0.0
        self._x1_max = 1.0
        self._x2_min = 0.0
        self._x2_max = 1.0
        
        # Alternatives to resize plots 
        # 1: just resize figure  2: resize plus create subplots() [figure + axes] 
        self._plot_resize_alternative = 2 
                
        # X (x1,x2)-Input array 
        self.__X = X
        self.__y = y 
        self._Z  = None
        
        # predictor = algorithm to create y-values for new (x1,x2)-data points 
        self._predictor = predictor 
        
        # meshdata
        self._resolution = mesh_res   # resolution of the mesh 
        self.__ax_x_delta = ax_x_delta 
        self.__ax_y_delta = ax_y_delta
        self._alpha = alpha
        self._bscatter = bscatter
        self._bcontour = bcontour 
        
        self._xm1 = None
        self._xm2 = None
        self._mesh_points = None 
        
        # set marker array and define colormap 
        self._markers = ('s', 'x', 'o', '^', 'v')
        self._colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
        self._cmap = ListedColormap(self._colors[:len(np.unique(y))])
        
        self._x1_lbl = x1_lbl
        self._x2_lbl = x2_lbl
        self._legend_loc = legend_loc
        
        # Plot-sizing
        self._figs_x1 = figs_x1
        self._figs_x2 = figs_x2
        self._fig = None
        self._ax  = None 
                # alternative 2 does resizing and (!) subplots() 
        self.initiate_and_resize_plot(self._plot_resize_alternative) 
               
        
        # create mesh in x1, x2 - direction with mesh_res resolution 
        # meshpoint-array will be creted with right dimension for plotting data 
        self.create_mesh()
        # Array meshpoints should exist now 
        
        if(self._bcontour == 1):
            try:
                if self._predictor == None:
                    raise ValueError
            except ValueError:
                print ("You must provide an algorithm = 'predictor' as parameter")
                #sys.exit(0)   
                sys.exit()   
            
            self.make_contourplot()
        else:
            if (self._bscatter == 1):
                self.make_scatter_plot() 


    # method to create a dense mesh in the (x1,x2)-plane 
    def create_mesh(self):
        '''
        Method to create a dense mesh in an (x1,x2) plane 
        Input: x1, x2-data are constructed from array self.__X
        Output: A suitable array of meshpoints is written to self._mesh_points()
        '''
        try:
            self._x1_min = self.__X[:, 0].min()
            self._x1_max = self.__X[:, 0].max()
        except ValueError: # as e: 
            print ("cannot determine x1_min = X[:,0].min() or x1_max = X[:,0].max()")
        try:
            self._x2_min = self.__X[:, 1].min()
            self._x2_max = self.__X[:, 1].max()
        except ValueError: # as e: 
            print ("cannot determine x2_min =X[:,1].min()) or x2_max = X[:,1].max()")
            
        self._x1_min, self._x1_max = self._x1_min - self.__ax_x_delta, self._x1_max + self.__ax_x_delta
        self._x2_min, self._x2_max = self._x2_min - self.__ax_x_delta, self._x2_max + self.__ax_x_delta
        
        #create mesh data (x1, x2) 
        self._xm1, self._xm2 = np.meshgrid( np.arange(self._x1_min, self._x1_max, self._resolution), 
                                            np.arange(self._x2_min, self._x2_max, self._resolution))
        
        #print (self._xm1)
        # for hasattr the variable cannot be provate ! 
        #print ("xm1 is there: ", hasattr(self,'_xm1' ) )
                          
        # ordering and transposing of the mesh-matrix   
        # for understanding the structure and transpose requirements see linux-blog.anracom.con          
        self._mesh_points = np.array([self._xm1.ravel(), self._xm2.ravel()]).T
        
        try:
            if( hasattr(self, '_mesh_points') == False ):
                raise ValueError
        except ValueError:
            print("The required array mesh_points has not been created!")
            exit

    # -------------
    # Some helper functions to change valus on the fly if necessary 
              
    def set_mesh_res(self, new_mesh_res): 
        self._resolution = new_mesh_res
        
    def change_predictor(self, new_predictor):
        self._predictor = new_predictor
        
    def change_alpha(self, new_alpha):    
        self._alpha = new_alpha
    
    def change_figs(self, new_figs_x1, new_figs_x2):    
        self._figs_x1 = new_figs_x1
        self._figs_x2 = new_figs_x2
    
    
    # -------------
    # method to get subplots and resize the figure       
    # -------------
    def initiate_and_resize_plot(self, alternative=2 ):
        
        # Alternative 1 to resize plots - works afte rimports to Jupyter notebooks, too
        if alternative == 1:
            self._fig_size = plt.rcParams["figure.figsize"]
            self._fig_size[0] = self._figs_x1
            self._fig_size[1] = self._figs_x2
            plt.rcParams["figure.figsize"] = self._fig_size
        
        # Not working for sizing if plain subplots() is used 
        #plt.figure(figsize=(self._figs_x1 , self._figs_x2))
        #self._fig, self._ax = plt.subplots()
        # instead you have to put the figsize-parameter into the subplots() function 
        
        # Alternative 2 for resizing plots and using subplots()
        # we use this alternative as we may need the axes later for 3D plots 
        if alternative == 2:
            self._fig, self._ax = plt.subplots(figsize=(self._figs_x1 , self._figs_x2))
    
    
    # ***********************************************
    
    # -------------
    # method to create contour plots      
    # -------------
    def make_contourplot(self):
        '''
        Method to create a contourplot based on a dense mesh of points in an (x1,x2) plane 
        and a predictor algorithm which allows for value calculations
        '''
        
        try:
            if( not hasattr(self, '_mesh_points') ):
                raise ValueError
        except ValueError:
            print("The required array mesh_points has not been created!")
            exit
        
        # Predict values for all meshpoints 
        try:
            self._Z = self._predictor.predict(self._mesh_points)
        except AttributeError: 
            print("method predictor.predict() does not exist") 
        
        #reshape     
        self._Z = self._Z.reshape(self._xm1.shape)
        #print (self._Z)
        
        # make the plot
        plt.contourf(self._xm1, self._xm2, self._Z, alpha=self._alpha, cmap=self._cmap)     
        
        # create a scatter-plot of data sample as well 
        if (self._bscatter == 1):
            self.make_scatter_plot()
            
        self.make_plot_legend()
        
           
                 
    # -------------
    # method to create a scatter plot of the data sample 
    # -------------
    def make_scatter_plot(self):
        alpha2 = self._alpha + 0.4 
        if (alpha2 > 1.0 ):
            alpha2 = 1.0
        for idx, yv in enumerate(np.unique(self.__y)): 
            plt.scatter(x=self.__X[self.__y==yv, 0], y=self.__X[self.__y==yv, 1], 
                        alpha=alpha2, c=[self._cmap(idx)], marker=self._markers[idx], label=yv)
        
        if self._bscatter == 0:
            self._bscatter = 1
        
        self.make_plot_legend()
          
        
    # -------------
    # method to add a legend  
    # -------------
    def make_plot_legend(self):
        plt.xlim(self._x1_min, self._x1_max)
        plt.ylim(self._x2_min, self._x2_max)
        plt.xlabel(self._x1_lbl)
        plt.ylabel(self._x2_lbl)
        
        # we have two cases 
        #     a) for a scatter plot we have array values where the legend is taken from automatically
        #     b) For apure contourplot we need to prepare a legend with "patches" (kind og labels) used by pyplot.legend() 
        if (self._bscatter == 1):
            plt.legend(loc=self._legend_loc)
        else:
            red_patch  = mpat.Patch(color='red',  label='0', alpha=0.4)
            blue_patch = mpat.Patch(color='blue', label='1', alpha=0.4)
            plt.legend(handles=[red_patch, blue_patch], loc=self._legend_loc)

    

 
This certainly is no masterpiece of superior code design; so you may change it. However, the code is good enough for our present purposes.

Note that we have to import basic Python modules into the namespace of this module. This is conventionally done at the top of the file.

Note also the 2 alternatives offered for resizing a plot! Both work also for "inline" plotting in a Jupyter environment; see the text below.

Using the module "myplots" in a Jupyter notebook

In a terminal we move to our example directory "/projekte/GIT/ai/ml1" and start our virtual Python environment:

myself@mytux: /projekte/GIT/ai/ml1> source bin/activate    
(ml1) myself@mytux:/projekte/GIT/ai/ml1> jupyter notebook
[I 11:46:14.687 NotebookApp] Writing notebook server cookie secret to /run/user/1999/jupyter/notebook_cookie_secret
[I 11:46:15.942 NotebookApp] Serving notebooks from local directory: /projekte/GIT/ai/ml1
[I 11:46:15.942 NotebookApp] The Jupyter Notebook is running at:
....
....

We then open our old notebook "moons1" and save it under the name "moons2":

We delete all old cells. Then we change the first cell of our new notebook to the following contents:

import imp
%matplotlib inline
from mycode import myplots

from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import PolynomialFeatures
from sklearn.svm import LinearSVC
from sklearn.svm import SVC

You see that we imported the "myplots" module from the "package" directory "mycode". The Jupyter notebook will find the path to "mycode" as long as we have opened the notebook on a higher directory level. Which we did ... see above.

Note the statement with a so called "magic command" for our notebook:

%matplotlib inline

There are many other "magic commands" and parameters which you can study at
Ipython magic commands

The command "%matplotlib inline" informs the notebook to show plots created by any imported modules "inline", i.e. in the visual context of the affected cell. This specific magic directive should be issued before matplotlib/pyplot is imported into any of the external python modules which we in turn import into the cell code.

A call of plt.show() in our class's method "make_contourplot()" is no longer necessary afterwards.

If we, however, want to resize the plots in comparison to Jupyter standard values we have to control this by parameters of our plot class. Such parameters are offered already in the interface of the class; but they can be changed by a method "change_figs(new_figs_x1, new_figs_x2)), too.

In a second cell of our new notebook we now prepare the moon data set

X, y = make_moons(noise=0.1, random_state=5)

Further cells will be used for quick individual experiments with the moons dataset. E.g.:

imp.reload(myplots)
X, y = make_moons(noise=0.1, random_state=5)

# contour plot of the moons data - with scatter plot / training with LinearSVC 
polynomial_degree = 3
max_iterations = 6000
polynomial_svm_clf = Pipeline([
  ("poly_features", PolynomialFeatures(degree=polynomial_degree)),
  ("scaler", StandardScaler()),
  ("svm_clf", LinearSVC(C=18, loss="hinge", max_iter=max_iterations))
])
#training
polynomial_svm_clf.fit(X, y)

#plotting 
MyPlt = myplots.MyDecisionPlot(X, y, polynomial_svm_clf, bcontour = 1, bscatter=1 )

The last type of cell just handles the setup and training of our specific algorithm "LinearSVC" and delegates plotting to our class.

Testing the new notebook

A test of the 3 cells in their order gives

All well! This is exactly what we hoped to achieve.

Three experiments with a varying polynomial degree

As we now have a simple and compact cell template for experiments we add three further cells where we vary the degree of the polynomials for LinearSVC. Below we show the results for degree 6, 7 and for comparison also for a degree of 2.

On a modern computer it should take almost no time to produce the results. (We shall learn how to measure CPU-time in the next article).

We understand that we at least need a polynomial of degree 3 to separate the data reasonably. However, polynomials with an even degree (>= 4) separate the 2 data regions very differently compared to polynomials with an uneven degree (>=3) in outer areas of the (x1,x2)-plane (where no training data were placed):

Depending on the polynomial degree our Linear SVC algorithm obviously extrapolates in different ways to regions without such data. And we have no clue which of the polynomials is the better choice ...

This poses a warning for the use of AI in general:

We should be extremely careful to trust predictions of any AI algorithm in parameter regions for which the algorithm must extrapolate - as long as we have no real data points available there to discriminate between multiple solutions that all work equally well in regions with given training data.

Would general modules be imported twice in a Jupyter cell - via the import of an external module, which itself includes import statements, and a direct import statement in a Jupyter cell?

The question posed in the headline is an interesting one for me as a Python beginner. Coming from other programming languages I get a bit nervous when I see the possibility for import statements referring to a specific module both in another already imported module and by a direct import statement afterwards in a Jupyter cell. E.g. we import numpy indirectly via our "myplots" module, but we could and sometimes must import it in addition directly in our Jupyter cell.

Note that we must make the general modules as numpy, matplotlib, etc. available in the namespace environment of our private module "myplots". Otherwise the functions cannot be used there. The Jupyter cell, however, corresponds to an independent namespace environment. So, an import may indeed be required there, too, if we plan to handle numpy arrays via code in such a cell.

Reading a bit about the Python import logic on the Internet reveals that a double import or overwriting should not take place; instead an already imported piece of code only gets multiple references in the various specific namespaces of different code environments.

We can test this with the following code in a Jupyter cell:

Note that numpy is also imported by our "myplots". However, the length of the list produced by sys.modules.keys(), which enumerates all possible module reference points (including imports) does not change.

Reloading modules in Jupyter cells

What if we in the course of or experiments need to change the code of our imported module? Then we need to reload the module in a Jupyter cell before we run it again. In our case (Python 3!) this can be done by the command

imp.reload(myplots)

As the code of our first cell reveals, the general package "imp" must have been imported before we can use its reload-function.

Conclusion

We saw that it is easy to use our own modules with Python class code, which we created in an Eclipse/PyDev environment, in a Jupyter notebook. We just use Python's standard import mechanism in Jupyter cells to get access to our classes. We applied this to a module with a simple class for preparing decision surface plots based on contour and scatter plot routines of matplotlib. We imported the module in a new Jupyter notebook.

Plots created by our imported class-methods were displayed correctly within the cell environment as soon as we used the magic directive "%matplotlib inline" within our notebook.

In addition we used our new notebook with its compact cell structure for some quick experiments: We set different values for the polynomial degree parameter of our LinearSVC algorithm. We saw that the results of algorithms should be interpreted with caution if the predictions refer to data points in regions of the representation or feature space which were not at all covered by the data sets for training.

The prediction results (= extrapolations) of even one and the same algorithm with different parameters may deviate strongly in such regions - and we may not have any reliable indications to decide which of the predictions and their basic parameter setting are better.

In the next article of this series

The moons dataset and decision surface graphics in a Jupyter environment – VI – Kernel-based SVC algorithms

we shall have a look at what kind of decision surface some other SVM algorithms than LinearSVC create for the moons dataset. In addition shall briefly discuss kernel based algorithms.

 

The moons dataset and decision surface graphics in a Jupyter environment – IV – plotting the decision surface

In this article series

The moons dataset and decision surface graphics in a Jupyter environment – I
The moons dataset and decision surface graphics in a Jupyter environment – II – contourplots
The moons dataset and decision surface graphics in a Jupyter environment – III – Scatter-plots and LinearSVC

we used the moons data set to build up some basic knowledge about using a Jupyter notebook for experiments, Pipelines and SVM-algorithms of SciKit-Sklearn and plotting functionality of matplotlib.

Our ultimate goal is to write some code for plotting a decision surface between the moon shaped data clusters. The ability to visualize data sets and related decision surfaces is a key to quickly testing the quality of different SVM-approaches. Otherwise, you would have to run some kind of analysis code to get an impression of what is going on and possible deficits of the determined separation surface.

In most cases, a visual impression of the separation surface for complexly shaped data sets will give you much clearer information. With just one look you get answers to the following questions:

  • How well does the decision surface really separate the data points of the clusters? Are there data points which are placed on the wrong side of the decision surface?
  • How reasonable does the decision surface look like? How does it extend into regions of the representation space not covered by the data points of the training set?
  • Which parameters of our SVM-approach influences what regarding the shape of the surface?

In the second article of this series we saw how we would create contour-plots. The motivation behind this was that a decision surface is something as the border between different areas of data points in an (x1,x2)-plane for which we get different distinct Z(x1,x2)-values. I.e., a contour line separating contour areas is an example of a decision surface in a 2-dimensional plane.

During the third article we learned in addition how we could visualize the various distinct data points of a training set via a scatter-plot.

We then applied some analysis tools to analyze the moons data - namely the "LinearSVC" algorithm together with "PolynomialFeatures" to cover non-linearity by polynomial extensions of the input data.

We did this in form of a Sklearn Pipeline for a step-wise transformation of our data set plus the definition of a predictor algorithm. Our LinearSVC-algorithm was trained with 3000 iterations (for a polynomial degree of 3) - and we could predict values for new data points.

In this article we shall combine all previous insights to produce a visual impression of the decision interface determined by LinearSVC. We shall put part of our code into a wrapper function. This will help us to efficiently visualize the results of further classification experiments.

Predicting Z-values for a contour plot in the (x1,x2) representation space of the moons dataset

To allow for the necessary interpolations done during contour plotting we need to cover the visible (x1,x2)-area relatively densely and systematically by data points. We then evaluate Z-values for all these points - in our case distinct values, namely 0 and 1. To achieve this we build a mesh of data points both in x1- and x2-direction. We saw already in the second article how numpy's meshgrid() function can help us with this objective:

resolution = 0.02
x1_min, x1_max = X[:, 0].min()  - 1, X[:, 0].max() + 1
x2_min, x2_max = X[:, 1].min()  - 1, X[:, 1].max() + 1
xm1, xm2 = np.meshgrid( np.arange(x1_min, x1_max, resolution), 
                        np.arange(x2_min, x2_max, resolution))

We extend our area quite a bit beyond the defined limits of (x1,x2) coordinates in our data set. Note that xm1 and xm2 are 2-dim arrays (!) of the same shape covering the surface with repeated values in either coordinate! We shall need this shape later on in our predicted Z-array.

To get a better understanding of the structure of the meshgrid data we start our Jupyter notebook (see the last article), and, of course, first run the cell with the import statements

import numpy as np
import matplotlib
from matplotlib import pyplot as plt
from matplotlib import ticker, cm
from mpl_toolkits import mplot3d

from matplotlib.colors import ListedColormap
from sklearn.datasets import make_moons

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import PolynomialFeatures
from sklearn.svm import LinearSVC
from sklearn.svm import SVC

Then we run the cell that creates the moons data set to get the X-array of [x1,x2] coordinates plus the target values y:

X, y = make_moons(noise=0.1, random_state=5)
#X, y = make_moons(noise=0.18, random_state=5)
print('X.shape: ' + str(X.shape))
print('y.shape: ' + str(y.shape))
print("\nX-array: ")
print(X)
print("\ny-array: ")
print(y)

Now we can apply the "meshgrid()"-function in a new cell:

You see the 2-dim structure of the xm1- and xm2-arrays.

Rearranging the mesh data for predictions
How do we predict data values? In the last article we did this in the following form

z_test = polynomial_svm_clf.predict([[x1_test_1, x2_test_1], 
                                     [x1_test_2, x2_test_2], 
                                     [x1_test_3, x2_test_3],
                                     [x1_test_3, x2_test_3]
                                    ])      

"polynomial_svm_clf" was the trained predictor we got by our pipeline with the LinearSVC algorithm and a subsequent training.

The "predict()"-function requires its input values as a 1-dim array, where each element provides a (x1, x2)-pair of coordinate values. But how do we get such pairs from our strange 2-dimensional xm1- and xm2-arrays?

We need a bit of array- or matrix-wizardry here:

Numpy gives us the function "ravel()" which transforms a 2d-array into a 1-d array AND numpy also gives us the possibility to transpose a matrix (invert the axes) via "array().T". (Have a look at the numpy-documentation e.g. at https://docs.scipy.org/doc/).

We can use these options in the following way - see the test example:

The involved logic should be clear by now. So, the next step should be something like

Z = polynomial_svm_clf.predict( np.array([xm1.ravel(), xm2.ravel()] ).T)

However, in the second article we already learned that we need Z in the same shape as the 2-dim mesh coordinate-arrays to create a contour-plot with contourf(). We, therefore, need to reshape the Z-array; this is easy - numpy contains a method reshape() for numpy-array-objects : Z = Z.reshape(xm1.shape). It is sufficient to use xm1 - it has the same shape as xm2.

Applying contourf()

To distinguish contour areas we need a color map for our y-target-values. Later on we will also need different optical markers for the data points. So, for the contour-plot we add some statements like

markers = ('s', 'x', 'o', '^', 'v')
colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
# fetch unique values of y into array and associate with colors  
cmap = ListedColormap(colors[:len(np.unique(y))])

Z = Z.reshape(xm1.shape)

# see article 2 for the use of contourf()
plt.contourf(xm1, xm2, Z, alpha=0.4, cmap=cmap)  

Let us put all this together; as the statements to create a plot obviously are many, we first define a function "plot_decision_surface()" in a notebook cell and run the cell contents:

Now, let us test - with a new cell that repeats some of our code of the last article for training:

Yeah - we eventually got our decision surface!

But this result still is not really satisfactory - we need the data set points in addition to see how good the 2 clusters are separated. But with the insights of the last article this is now a piece of cake; we extend our function and run the definition cell

def plot_decision_surface(X, y, predictor, ax_delta=1.0, mesh_res = 0.01, alpha=0.4, bscatter=1,  
                          figs_x1=12.0, figs_x2=8.0, x1_lbl='x1', x2_lbl='x2', 
                          legend_loc='upper right'):

    # some arrays and colormap
    markers = ('s', 'x', 'o', '^', 'v')
    colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
    cmap = ListedColormap(colors[:len(np.unique(y))])

    # plot size  
    fig_size = plt.rcParams["figure.figsize"]
    fig_size[0] = figs_x1 
    fig_size[1] = figs_x2
    plt.rcParams["figure.figsize"] = fig_size

    # mesh points 
    resolution = mesh_res
    x1_min, x1_max = X[:, 0].min()  - 1, X[:, 0].max() + 1
    x2_min, x2_max = X[:, 1].min()  - 1, X[:, 1].max() + 1
    xm1, xm2 = np.meshgrid( np.arange(x1_min, x1_max, resolution), 
                            np.arange(x2_min, x2_max, resolution))
    mesh_points = np.array([xm1.ravel(), xm2.ravel()]).T

    # predicted vals 
    Z = predictor.predict(mesh_points)
    Z = Z.reshape(xm1.shape)

    # plot contur areas 
    plt.contourf(xm1, xm2, Z, alpha=alpha, cmap=cmap)

    # add a scatter plot of the data points 
    if (bscatter == 1): 
        alpha2 = alpha + 0.4 
        if (alpha2 > 1.0 ):
            alpha2 = 1.0
        for idx, yv in enumerate(np.unique(y)): 
            plt.scatter(x=X[y==yv, 0], y=X[y==yv, 1], 
                        alpha=alpha2, c=[cmap(idx)], marker=markers[idx], label=yv)
            
    plt.xlim(x1_min, x1_max)
    plt.ylim(x2_min, x2_max)
    plt.xlabel(x1_label)
    plt.ylabel(x2_label)
    if (bscatter == 1):
        plt.legend(loc=legend_loc)

Now we get:

So far, so good ! We see that our specific model of the moons data separates the (x1,x2)-plane into two areas - which has two wiggles near our data points, but otherwise asymptotically approaches almost a diagonal.

Hmmm, one could bet that this is model specific. Therefore, let us do a quick test for a polynomial_degree=4 and max_iterations=6000. We get

Surprise, surprise ... We have already 2 different models fitting our data.

Which one do you believe to be "better" for extrapolations into the big (x1,x2)-plane? Even in the vicinity of the leftmost and rightmost points in x1-direction we would get different predictions of our models for some points. We see that our knowledge is insufficient - i.e. the test data do not provide enough information to really distinguish between different models.

Conclusion

After some organization of our data we had success with our approach of using a contour plot to visualize a decision surface in the 2-dimensional space (x1,x2) of input data X for our moon clusters. A simple wrapper function for surface plotting equips us now for further fast experiments with other algorithms.

To become better organized, we should save this plot-function for decision surfaces as well as a simpler function for pure scatter plots in a Python class and import the functionality later on.

We shall create such a class within Eclipse PyDev as a first step in the next article:

The moons dataset and decision surface graphics in a Jupyter environment - V - a class for plots and some experiments

Afterward we shall look at other SVM algorithms - as the "polynomial kernel" and the "Gaussian kernel". We shall also have a look at the impact of some of the parameters of the algorithms. Stay tuned ...