Variational Autoencoder with Tensorflow – XIV – Change of variational model parameters at inference time

In my last post of this series I compared a Variational Autoencoder [VAE] with only a tiny amount of KL-loss to a standard Autoencoder [AE]. See the links at

Variational Autoencoder with Tensorflow – XIII – Does a VAE with tiny KL-loss behave like an AE? And if so, why?

for more information and preparational posts.

Both the Keras models for the VAE and the AE were trained on the CelebA dataset of images of human heads. We found a tight similarity regarding the clustering of predicted data points for the training and test data in the latent space. In addition the VAE with tiny KL loss failed to reconstruct reasonable human face images from arbitrarily chosen points in the latent space. Just as a standard AE does. In forthcoming posts we will continue to study the relation between VAEs and AEs.

But in this post I want to briefly point out an interesting technical problem which may arise when you start to tests predictions for certain data samples after a training phase. Your Encoder or Decoder models may include parameters which you want to experiment with when predicting results for interesting input data. This raises the question whether we can vary such parameters at inference time. Actually, this was not quite as easy as it seemed to be when I started with respective experiments. To perform them I had to learn two aspects of Keras models I had not been aware of before.

How to switch of the z-point variation at inference time?

In my particular case the starting point was the following consideration:

At inference time there is no real need for using the logvar-based variation around mu-values predicted by the Encoder.

The variation in z-point values in VAEs is done by adding a statistical value to mu-values. The added term is based on a log_var value multiplied by a statistically fluctuating factor “epsilon”, which comes from a normal Gaussian distribution around zero. mu, therefore, is the center of a distribution a specific input tensor is mapped to in latent space for consecutive predictions. The mu- and log_var-values depend on weights of two dense layers of the Encoder and thus indirectly on the optimization during training.

But while the variation is essential during training one may NOT regard it necessary for predictions. During inference we may in some experiments have good reasons to only refer to the central mu value when predicting a z-point in latent space. For test and analysis purposes it could be interesting to omit the log_var contribution.

The question then is: How we can switch off the log_var component for the Encoder’s predictions, i.e. predictions of our Keras based encoder model?

One idea is to include variables of a Python class hosting the Keras models for the Encoder, Decoder and the composed VAE in the function for the calculation of z-point vectors.

The mu-, logvar and sampling layers of the VAE’s Encoder model encapsulated in a Python class

During this post series we have encapsulated the code for Encoder, Decoder and resulting VAE models in a Python class. Remember that the Encoder produced its output, namely z-points in the latent space via two “dense” Keras layers and a sampling layer (based on a Keras Lambda-layer). The dense layers followed a series of convolutional layers and gave us mu and log_var values. The Lambda-layer produced the eventual z-point-vector including the variation. In my case the code fragments for the layers look similar to the following ones:

# .... Layer model of the Encoder part 
...
...     # Definition of an input layer and multiple Conv2D layers 
...
        # "Variational" part - 2 Dense layers for a statistical distribution of z-points  
        self.mu      = Dense(self.z_dim, name='mu')(x)
        self.log_var = Dense(self.z_dim, name='log_var')(x)
        # Layer to provide a z_point in the Latent Space for each sample of the batch 
        self._encoder_output = Lambda(z_point_sampling, name='encoder_output')([self.mu, self.log_var])
...
        # The Encoder model 
        self.encoder = Model(inputs=self._encoder_input, outputs=[self._encoder_output, self.mu, self.log_var], name="encoder")
...

The “self” refers to a class “MyVariationalAutoencoder” comprising the Encoder’s, Decoder’s and the VAE’s model and layer structures. See for details and explained code fragments of the class e.g. the posts around Variational Autoencoder with Tensorflow 2.8 – X – VAE application to CelebA images.

The sampling is in my case done by a function “z_point_sampling”:

        def z_point_sampling(args):
            '''
            A point in the latent space is calculated statistically 
            around an optimized mu for each sample 
            '''
            mu, log_var = args # Note: These are 1D tensors !
            epsilon = B.random_normal(shape=B.shape(mu), mean=0., stddev=1.)
            return mu + B.exp(log_var / 2.) * epsilon * self.enc_v_eps_factor

You see that this function uses a class member variable “self.enc_v_eps_factor”.

Switch the variation with log_var on and off for predictions?

Our objective is to switch the log_var contribution on or off for input certain images or batches of such images fed into the Encoder. For this purpose we could in principle use the variable “self.enc_v_eps_factor” as a kind of boolean switch with values of either 0.0 or 1.0. To set the variable I had defined two class methods:

    def set_enc_to_predict(self):
        self.enc_v_eps_factor = 0.0 
    
    def set_enc_to_train(self):  
        self.enc_v_eps_factor = 1.0 

The basic idea was that the sampling function would pick the value of enc_v_eps_factor given at the runtime of a prediction, i.e. at inference time. This assumption was, however, wrong. At least in a certain sense.

Is a class variable change impacting a layer output noted during consecutive predictions of a Keras model?

Let us assume that we have instantiated our class and assigned it to a Python variable MyVae. Let us further assume the the comprised Keras models are referenced by variables

  • MyVae.encoder (for the Encoder part),
  • MyVae.decoder (for the Decoder part)
  • and MyVae.model (for the full VAE-model).

We do not care about further details of the VAE (consisting of the Encoder, the Decoder and GradientTape based cost control). But we should not forget that all the models’ layers and their weights determine cost functions derivatives and are therefore targets of the optimization performed during training. All factors determining gradients and value calculation with given weights are encoded with the compilation of a Keras model – for training purposes [using model.fit() with a Keras model], but as well for predictions! Without a compiled Keras model we cannot use model.predict().

This means: As soon as you have a compiled Keras model almost and load the weight-values saved after a a sufficient number of training epochs everything is settled for inference and predictions. Including the present value of self.enc_v_eps_factor. At compile time?

Well, thinking about it a bit more in detail from a developer perspective tells us:
The compilation would in principle not prevent the use of a changed variable at the run-time of a prediction. But on the other hand side we also have the feeling that Keras must do something to make training (which also requires predictions in the sense of a forward pass) and later raw predictions for batches at inference time pretty fast. Intermediate functionality changes would hamper performance – if only for the reason that you have to watch out for such changes.

So, it is natural to assume that Keras would keep any factors in the Lambda-layer taken from a class variable constant after compilation and during training or inference, i.e. predictions.

If this assumption were true then chain of actions AFTER a training of a VAE model (!) like

Define a Keras based VAE-model with a sampling layer and a factor enc_v_eps_factor = 1   =>   compile it (including the sub-models for the Encoder and Decoder)   =>   load weight parameters which had been saved after training   =>   switch the value of the class variable enc_v_eps_factor to 0.0   =>   Load an image or image batch for prediction

would probably NOT work as expected. To be honest: This is “wisdom” derived after experiments which did not give me the naively expected results.

Indeed, some first simple experiments showed: The value of enc_v_eps_factor (e.g. enc_v_eps_factor = 1) which was given at compile-time was used during all following prediction calculations, e.g. for a particular image in tensorial form. So, a command sequence like

MyVae.set_enc_to_train()
MyVae.encoder.compile() 
...
# Load weights from a set path 
MyVae.model.load_weights(path_weights)
...

z_point, mu, log_var = MyVae.encoder.predict(img1)
print(z_point, mu)
MyVae.set_enc_to_predict(self)
z_point, mu, log_var = MyVae.encoder.predict(img1)

did not give different results. Note, that I did not change the value of enc_v_eps_factor between compile time and the first call for a prediction.

Let us look at the example in more detail.

A concrete example

After a full training of my VAE on CelebA for 24 epochs I checked for a maximum log_var-value. Non-negligible values may indeed occur for certain images and special z-vector components for some images despite the tiny KL-loss. And indeed such a value occurred for a very special (singular) image with two diagonal black border stripes on the left and right of the photographed person’s head. I do not show the image due to digital rights concerns. But let us look at the predicted z, mu and log_var values (for a specific z-point vector component) of this image. Before I compiled the VAE model I had set enc_v_eps_factor = 1.0 :

# Set the learning rate and COMPILE the model 
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
learning_rate = 0.0005

# The following is only required for compatibility reasons
b_old_optimizer = True     

# Set enc_v_eps_factor to 1.0
MyVae.set_enc_to_train()

# Separate Encoder compilation
# - does not harm the compilation of the full VAE-model, but is useful to avoid later trace warnings after changes 
MyVae.encoder.compile()

# Compilation of the full VAE model (Encoder, Decoder, cost functions used by GradientTape) 
MyVae.compile_myVAE(learning_rate=learning_rate, b_old_optimizer = b_old_optimizer )
...

# Load weights from a set path 
MyVae.model.load_weights(path_weights)
...
# start predictions
...

with

    def compile_myVAE(self, learning_rate, b_old_optimizer = False):
        # Version 1.1 of 221212
        # Forced to special handling of the optimizer for data resulting from training before TF 11.2 , due to warnings:  
        #      ValueError: You are trying to restore a checkpoint from a legacy Keras optimizer into a v2.11+ Optimizer, 
        #      which can cause errors. Please update the optimizer referenced in your code to be an instance 
        #      of `tf.keras.optimizers.legacy.Optimizer`, e.g.: `tf.keras.optimizers.legacy.Adam`.

        # Optimizer handling
        # ~~~~~~~~~ 
        if b_old_optimizer: 
            optimizer = tf.keras.optimizers.legacy.Adam(learning_rate=learning_rate)
        else:    
            optimizer = Adam(learning_rate=learning_rate)
        ....

        # Solution type with train_step() and GradientTape()  
        if self.solution_type == 3:
            self.model.compile(optimizer=optimizer)

Details of the compilation are not important – though you may be interested in the fact that training data saved after a training based on Python module version < 11.2 of TF 2 requires a legacy version of the optimizer when later using a module version ≥ 11.2 (corresponding to TF V2.11 and above).

However, the really important point is that the compilation is done given a certain value of enc_v_eps_factor = 1.

Then we load the image in form a prepared training batch with just one element set and provide it to the predict() function of the Keras model for the Encoder. We perform two predictions and ahead of the second one we change the value of enc_v_eps_factor to 0.0:

# Choose an image 
j = 123223
img = x_train[j] # this has already a tensor compatible format 
img_list = []
img_list.append(img)
tf_img = tf.convert_to_tensor(img_list)

# Encoder prediction 
z_points, mu, logvar  = MyVae.encoder.predict(tf_img)

print(z_points[0][230])
print(mu[0][230])
print(logvar[0][230])

# !!!! Set enc_v_eps_factor to 0.0 !!!!
MyVae.set_enc_to_predict()

# New Encoder prediction 
z_points, mu, logvar  = MyVae.encoder.predict(tf_img)

print()
print()

print(z_points[0][230])
print(mu[0][230])
print(logvar[0][230])

The result is:

...
2.637196
-0.141142
3.3761873
...
-0.2085279
-0.141142
3.3761873

The result depends on statistical variations for the factor epsilon (Gaussian statistics; see the sampling function above).

But the central point is not the deviation for the two different prediction calls. The real point is that we have used MyVae.set_enc_to_predict() ahead of the second prediction and, yet, the values for mu and log_var for the special z_point-component (230; out of 256 components) were NOT identical. I.e. the variable value enc_v_eps_factor = 1.0, which we set before the compilation, was used during both of our prediction calculations!

Can we just recompile between different calls to model.predict() ?

The experiment described above seems to indicate that the value for the class variable enc_v_eps_factor given at compile time is used during all consecutive predictions. We could, of course, enforce a zero variation for all predictions by using MyVae.set_enc_to_predict() ahead of the compilation of the Encoder model. But this would not give us no flexibility to switch the log_var contribution off ahead of predictions for some special images and then turn it on again for other images.

But the is simple – if we need not do this switching permanently: We just recompile the Encoder model!

Compilation does not take much time for Encoder models with only a few (convolutional and dense) layers. Let us test this by modifying the code above:

# Choose an image 
j = 123223
img = x_train[j] # this has already a tensor compatible format 
img_list = []
img_list.append(img)
tf_img = tf.convert_to_tensor(img_list)

# Set enc_v_eps_factor to 0.0
MyVae.set_enc_to_predict()
# !!!!
MyVae.encoder.compile() 

# Encoder prediction 
z_points, mu, logvar  = MyVae.encoder.predict(tf_img)
# Decoder prediction - just for fun 
reco_list = MyVae.decoder.predict(z_points) # just for fun 
print(z_points[0][230])
print(mu[0][230])
print(logvar[0][230])

print() 
print()

# Set enc_v_eps_factor back to 1.0
MyVae.set_enc_to_train()
MyVae.encoder.compile()

# New Encoder prediction 
z_points, mu, logvar  = MyVae.encoder.predict(tf_img)

print(z_points[0][230])
print(mu[0][230])
print(logvar[0][230])

We get

Shape of img_list =  (1, 96, 96, 3)
eps_fact =  0.0
1/1 [==============================] - 0s 280ms/step
1/1 [==============================] - 0s 18ms/step
Shape of reco_list =  (1, 96, 96, 3)

-0.141142
-0.141142
3.3761873


eps_fact =  1.0
1/1 [==============================] - 0s 288ms/step

-0.63676023
-0.141142
3.3761873

This is exactly what we want!

The function for the prediction step of a Keras model is cached at inference time …

The example above gave us the impression that it could be the compilation of a model which “settles” all of the functionality used during predictions, i.e. at inference time. Actually, this is no quite true.

The documentation on a Keras model helped me to get a better understanding. Near the section on the method “predict()” we find some other interesting functions. A look at the remarks on “predict_step()“, reveals (quotation)

The logic for one inference step.

This method can be overridden to support custom inference logic. This method is called by Model.make_predict_function.

This method should contain the mathematical logic for one step of inference. This typically includes the forward pass.

This leads us to the function “make_predict_function()” for Keras models. And there we find the following interesting remarks – I quote:

This method can be overridden to support custom inference logic. This method is called by Model.predict and Model.predict_on_batch.

Typically, this method directly controls tf.function and tf.distribute.Strategy settings, and delegates the actual evaluation logic to Model.predict_step.

This function is cached the first time Model.predict or Model.predict_on_batch is called. The cache is cleared whenever Model.compile is called. You can skip the cache and generate again the function with force=True.

Ah! The function predict_step() normally covers the forward pass through the network and “make_predict_function()” caches the resulting (function) object at the first invocation of model.predict(). And the respective cache is not cleared automatically.

So, what really may have hindered my changes of the sampling functionality at inference time is a cache filled at the first call to encoder.predict()!

Let us test this!

Changing the sampling parameters after compilation, but before the first call of encoder.predict()

If our suspicion is right we should be able to set up the model from scratch again, compile it, use MyVae.set_enc_to_predict() and afterward call MyVae.encoder.predict() – and get the same values for mu and z_point.

So we do something like

# Build encoder according to layer parameters 
MyVae._build_enc()
# Build decoder according to layer parameters 
MyVae._build_dec()
# Build the VAE-model 
MyVae._build_VAE()
...

# Set variable to 1.0
MyVae.set_enc_to_train()

# Compile 
learning_rate = 0.0005
b_old_optimizer = True     
MyVae.compile_myVAE(learning_rate=learning_rate, b_old_optimizer = b_old_optimizer )
MyVae.encoder.compile()   # used to prevent retracing delays - when later changing encoder variables 

...
# Load weights from a set path 
MyVae.model.load_weights(path_weights)
...

# preparation if the selected img. 
...
...

MyVae.set_enc_to_predict()
print("eps_fact = ", MyVae.enc_v_eps_factor)
# Note: NO recompilation is done !

# First prediction 
z_points, mu, logvar  = MyVae.encoder.predict(tf_img)
print()
print(z_points[0][230])
print(mu[0][230])
print(logvar[0][230])
..

Note that the change of enc_v_eps_factor ahead of the first call of predict(). And, indeed:

Shape of img_list =  (1, 96, 96, 3)
eps_fact =  0.0
...
-0.141142
-0.141142
3.3761873

Use make_predict_function(force=True) to clear and refill the cache for predict_step() and its forward pass functionality

The other option the documentation indicates is to use the function make_predict_function(force=True).
This leads to yet another experiment:

# img preparation 
....

# Set enc_v_eps_factor to 1.0
MyVae.set_enc_to_train()
MyVae.encoder.compile() 
print("eps_fact = ", MyVae.enc_v_eps_factor)

# Encoder prediction 
z_points, mu, logvar  = MyVae.encoder.predict(tf_img)
print(z_points[0][230])
print(mu[0][230])
print(logvar[0][230])

print() 
print()

# Set enc_v_eps_factor to 0.0
MyVae.set_enc_to_predict()
# !!!!
MyVae.encoder.make_predict_function(
    force=True
)
print("eps_fact = ", MyVae.enc_v_eps_factor)

# Encoder prediction 
z_points, mu, logvar  = MyVae.encoder.predict(tf_img)
print(z_points[0][230])
print(mu[0][230])
print(logvar[0][230])

We get

...
eps_fact =  1.0
1/1 [==============================] - 0s 287ms/step

-5.5451365
-0.141142
3.3761873


eps_fact =  0.0
1/1 [==============================] - 0s 271ms/step

-0.141142
-0.141142
3.3761873

Yes, exactly as expected. This again shows us that it is the cache which counts after the first call of model.predict() – and not the compilation of the Keras model (for the Encoder) !

Other approaches?

The general question of changing parameters at inference time also triggers the question whether we may be able to deliver parameters to the function model.predict() and transfer them further to customized variants of predict_step(). I found a similar question at stack overflow
Passing parameters to model.predict in tf.keras.Model

However, the example there was rather special – and I did not apply the lines of thought explained there to my own case. But the information given in the answer may still be useful for other readers.

Conclusion

In this post we have seen that we can change parameters influencing the forward pass of a Keras model at inference time. We saw, however, that we have to clear and fill a cache to make the changes effective. This can be achieved by

  • either applying a recompilation of the model
  • or enforcing a clearance and refilling of the cache for the model’s function predict_step().

In the special case of a VAE this allows for deactivating and re-activating the logvar-dependent statistical variation of the z-points a specific image is mapped to by the Encoder model during predictions. This gives us the option to focus on the central mu-dependent position of certain images in the latent space during experiments at inference time.

In the next post of this series we shall have a closer look at the filamental structure of the latent space of a VAE with tiny KL loss in comparison to the z-space structure of a VAE with sufficiently high KL loss.

 

Ceterum censeo: The worst fascist, war criminal and killer living today is the Putler. He must be isolated at all levels, be denazified and sooner than later be imprisoned. Somebody who orders the systematic destruction of civilian infrastructure must be fought and defeated because he is a permanent danger to basic principles of humanity – not only in Europe. Long live a free and democratic Ukraine!

 

Nvidia GPU-support of Tensorflow/Keras on Opensuse Leap 15

When you start working with Google’s Tensorflow on multi-layer and “deep learning” artificial neural networks the performance of the required mathematical operations may sooner or later become important. One approach to better performance is the use of a GPU (or multiple GPUs) instead of a CPU. Personally, I am not yet in a situation where GPU support is really required. My experimental CNNs are too small, yet. But starting with Keras and Tensorflow is a good point to cover the use of a GPU on my Opensuse Leap 15 systems anyway. Actually, it is also helpful for some tasks in security related environments, too. One example is testing the quality of passphrases for encryption. With JtR you may gain a factor of 10 in performance. It is interesting, how much faster an old 960 GTX card will be for a simple Tensorflow test application than my i7 CPU.

I have used Nvidia GPUs almost all my Linux life. To get GPU support for Nvidia graphics cards you need to install CUDA in its present version. This is 10.1 in August 2019. You get download and install information for CUDA at
https://developer.nvidia.com/cuda-zone => https://developer.nvidia.com/cuda-downloads
For an RPM for the x86-64 architecture and Opensuse Leap see:
https://developer.nvidia.com/cuda-downloads?….

Installation of “CUDA” and “cudcnn”

You may install the downloaded RPM (in my “case cuda-repo-opensuse15-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm”) via YaST. After this first step you in a second step install the meta-packet named “cuda”, which is available in YaST at this point. Or just install all other packets with “cuda” in the name (with the exception of the source code and dev-packets) via YaST.

A directory “/usr/local/cuda” will be built; its entries are soft links to files in a directory “/usr/local/cuda-10.1“.

Note the “include” and the “lib64” sub-directories! After the installation, also links should exist in the central “/usr/lib64“-directory pointing to the files in “/usr/local/cuda/lib64“.

Note from the file-endings that the particular present version [Aug. 2019) of the files may be something like “10.1.168“.

Another important point is that you need to install “cudnn” (cudnn-10.1-linux-x64-v7.6.2.24.tgz) – a Nvidia specific library for certain Deep Learning program elements, which shall be executed on Nvidia GPU chips. You get these files via “https://developer.nvidi.com/cudnn“. Unfortunately, you must become member of the Nvidia developer community to get access to these special files. After you downloaded the tgz-file and expanded it, you find some directories “include” and “lib64” with relevant files. You just copy these files (as user root) into the directories “/usr/local/cuda/include” and “/usr/local/cuda/lib64”, respectively. Check the owner/group and rights of the copied files afterwards and change them to root/root and standard rights – just as given for the other files in teh target directories.

The final step is the follwoing:
Create links by dragging the contents of “/usr/local/cuda/include” to “/usr/include” and chose the option “Link here”. Do the same for the files of “/usr/local/cuda/lib64” with “/usr/lib64” as the target directory. If you look at the link-directories of the files now in “usr/include” and “usr/lib64” you see exactly which files were given by the CUDA and cudcnn installation.

nAdditional libraries
In case you want to use Keras it is recommended to install the “openblas” libraries including the development packages on the Linux OS level. On an Opensuse system just search for packages with “openblas” and install them all. The same is true for the h5py-libraries. In your virtual python environment execute:
< p style="margin-left:50px;"pip3 install --upgrade h5py

Problems with errors regarding missing CUDA libraries after installation

Two stupid things may happen after this straight-forward installation :

  • The link structure between “/usr/lib64” and the files in “/usr/local/cuda/include” and “/usr/local/cuda/lib64” may be incomplete.
  • Although there are links from files as “libcufftw.so.10” to something like “libcufftw.so.10.1.168” some libraries and TensorFlow components may expect additional links as “libcufftw.so.10.0” to “libcufftw.so.10.1.168”

Both points lead to error messages when I tried to use GPU related test statements on a PyDEV console or Jupyter cell. Watch out for error messages which tell you about errors when opening specific libraries! In the case of Jupyter you may find such messages on the console or terminal window from which you started your test.

A quick remedy is to use a file-manager as “dolphin” as user root, mark all files in “/usr/local/cuda/include” and “usr/local/cuda/lib64” and place them as (soft) links into “/usr/include” and “/usr/lib64”, respectively. Then create additional links there for the required libraries “libXXX.so.10.0” to “libXXX.so.10.1.168“, where “XXX” stands for some variable part of the file name.

A simple test with Keras and the mnist dataset

I assume that you have installed the packages for tensorflow, tensorflow-gpu (!) and keras with pip3 in your Python virtualenv. Note that the package “tensorflow-gpu” MUST be installed after “tensorflow” to make the use of the GPU possible.

Then a test with a simple CNN for the “mnist” datatset can deliver information on performance differences :

Cell 1 of a Jupyter notebook:

import time 
import tensorflow as tf
from keras import backend as K
from tensorflow.python.client import device_lib
from keras.datasets import mnist
from keras import models
from keras import layers
from keras.utils import to_categorical

# function to provide CPU/GPU information 
# ---------------------------------------
def get_CPU_GPU_details():
    print("GPU ? ", tf.test.is_gpu_available())
    tf.test.gpu_device_name()
    print(device_lib.list_local_devices())

# information on available CPUs/GPUs
# --------------------------------------
if tf.test.is_gpu_available(
    cuda_only=False,
    min_cuda_compute_capability=None):
    print ("GPU is available")
get_CPU_GPU_details()

# Setting a parameter GPU or CPU usage 
#--------------------------------------
#gpu = False 
gpu = True
if gpu: 
    GPU = True;  CPU = False; num_GPU = 1; num_CPU = 1
else: 
    GPU = False; CPU = True;  num_CPU = 1; num_GPU = 0
num_cores = 6

# control of GPU or CPU usage in the TF environment
# -------------------------------------------------
# See the literature links at the article's end for more information  

config = tf.ConfigProto(intra_op_parallelism_threads=num_cores,
                        inter_op_parallelism_threads=num_cores, 
                        allow_soft_placement=True,
                        device_count = {'CPU' : num_CPU,
                                        'GPU' : num_GPU}, 
                        log_device_placement=True

                       )
config.gpu_options.per_process_gpu_memory_
fraction=0.4
config.gpu_options.force_gpu_compatible = True
session = tf.Session(config=config)
K.set_session(session)

#--------------------------
# Loading the mnist datatset via Keras 
#--------------------------
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()
network = models.Sequential()
network.add(layers.Dense(512, activation='relu', input_shape=(28*28,)))
network.add(layers.Dense(10, activation='softmax'))
network.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
train_images = train_images.reshape((60000, 28*28))
train_images = train_images.astype('float32') / 255
test_images = test_images.reshape((10000, 28*28))
test_images = test_images.astype('float32') / 255
train_labels = to_categorical(train_labels)
test_labels = to_categorical(test_labels)

Output of the code in cell 1:

GPU is available
GPU ?  True
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 17801622756881051727
, name: "/device:XLA_GPU:0"
device_type: "XLA_GPU"
memory_limit: 17179869184
locality {
}
incarnation: 6360207884770493054
physical_device_desc: "device: XLA_GPU device"
, name: "/device:XLA_CPU:0"
device_type: "XLA_CPU"
memory_limit: 17179869184
locality {
}
incarnation: 7849438889532114617
physical_device_desc: "device: XLA_CPU device"
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 2115403776
locality {
  bus_id: 1
  links {
  }
}
incarnation: 4388589797576737689
physical_device_desc: "device: 0, name: GeForce GTX 960, pci bus id: 0000:01:00.0, compute capability: 5.2"
]

Note the control settings for GPU usage via the parameter gpu and the variable “config”. If you do NOT want to use the GPU execute

config = tf.ConfigProto(device_count = {‘GPU’: 0, ‘CPU’ : 1})

Information on other control parameters which can be used together with “tf.ConfigProto” is provided here:
https://stackoverflow.com/questions/40690598/can-keras-with-tensorflow-backend-be-forced-to-use-cpu-or-gpu-at-will

Cell 2 of a Jupyter notebook for performance measurement during training:

start_c = time.perf_counter()
with tf.device("/GPU:0"):
    network.fit(train_images, train_labels, epochs=5, batch_size=30000)
end_c = time.perf_counter()
if CPU: 
    print('Time_CPU: ', end_c - start_c)  
else:  
    print('Time_GPU: ', end_c - start_c)  

Output of the code in cell 2 :

Epoch 1/5
60000/60000 [==============================] - 0s 3us/step - loss: 0.5817 - acc: 0.8450
Epoch 2/5
60000/60000 [==============================] - 0s 3us/step - loss: 0.5213 - acc: 0.8646
Epoch 3/5
60000/60000 [==============================] - 0s 3us/step - loss: 0.4676 - acc: 0.8832
Epoch 4/5
60000/60000 [==============================] - 0s 3us/step - loss: 0.4467 - acc: 0.8837
Epoch 5/5
60000/60000 [==============================] - 0s 3us/step - loss: 0.4488 - acc: 0.8726
Time_GPU:  0.7899935730001744

Now change the following lines in cell 1

 
...
gpu = False 
#gpu = True 
...

Executing the code in cell 1 and cell 2 then gives:

Epoch 1/5
60000/60000 [==============================] - 0s 6us/step - loss: 0.4323 - acc: 0.8802
Epoch 2/5
60000/60000 [==============================] - 0s 7us/step - loss: 0.3932 - acc: 0.8972
Epoch 3/5
60000/60000 [==============================] - 0s 6us/step - loss: 0.3794 - acc: 0.8996
Epoch 4/5
60000/60000 [==============================] - 0s 6us/step - loss: 0.3837 - acc: 0.8941
nEpoch 5/5
60000/60000 [==============================] - 0s 6us/step - loss: 0.3830 - acc: 0.8908
Time_CPU:  1.9326397939985327

Thus the GPU is faster by a factor of 2.375 !
At least for the chosen batch size of 30000! You should play a bit around with the batch size to understand its impact.
2.375 is not a big factor – but I have a relatively old GPU (GTX 960) and a relatively fast CPU i7-6700K mit 4GHz Taktung: So I take what I get 🙂 . A GTX 1080Ti would give you an additional factor of around 4.

Watching GPU usage during Python code execution

A CLI command which gives you updated information on GPU usage and memory consumption on the GPU is

nvidia-smi -lms 250

It gives you something like

Mon Aug 19 22:13:18 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960     On   | 00000000:01:00.0  On |                  N/A |
| 20%   44C    P0    33W / 160W |   3163MiB /  4034MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      4124      G   /usr/bin/X                                   610MiB |
|    0      4939      G   kwin_x11                                      54MiB |
|    0      4957      G   /usr/bin/krunner                               1MiB |
|    0      4959      G   /usr/bin/plasmashell                         195MiB |
|    0      5326      G   /usr/bin/akonadi_archivemail_agent             2MiB |
|    0      5332      G   /usr/bin/akonadi_imap_resource                 2MiB |
|    0      5338      G   /usr/bin/akonadi_imap_resource                 2MiB |
|    0      5359      G   /usr/bin/akonadi_mailfilter_agent              2MiB |
|    0      5363      G   /usr/bin/akonadi_sendlater_agent               2MiB |
|    0      5952      C   /usr/lib64/libreoffice/program/soffice.bin    38MiB |
|    0      8240      G   /usr/lib64/firefox/firefox                     1MiB |
|    0     13012      C   /projekte/GIT/ai/ml1/bin/python3            2176MiB |
|    0     14233      G   ...uest-channel-token=14555524607822397280    62MiB |
+-----------------------------------------------------------------------------+

During code execution some of the displayed numbers – e.g for GPU-Util, GPU memory Usage – will start to vary.

Links

https://medium.com/@liyin2015/tensorflow-cpus-and-gpus-configuration-9c223436d4ef
https://www.tensorflow.org/beta/guide/using_gpu
https://stackoverflow.com/questions/40690598/can-keras-with-tensorflow-backend-be-forced-to-use-cpu-or-gpu-at-will
https://stackoverflow.com/questions/42706761/closing-session-in-tensorflow-
doesnt-reset-graph

http://www.science.smith.edu/dftwiki/index.php/Setting up Tensorflow 1.X on Ubuntu 16.04 w/ GPU support
https://hackerfall.com/story/which-gpus-to-get-for-deep-learning
https://towardsdatascience.com/measuring-actual-gpu-usage-for-deep-learning-training-e2bf3654bcfd