Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – III

During the last articles of this mini-series

Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – I
Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – II

we saw how to set up a basic PyDev project in Eclipse which we coupled to a virtual Python environment. We modified the PYTHONPATH and added Python packages to our project with the help of “pip”. In addition we have prepared the PyDev console for interactive Python experiments such that we can use matplotlib in a Qt5 environment. Thus we got a reasonably equipped development environment to start with Python based experiments in “Machine-Learning” [ML] – and build up code much more systematically than just with Jupyter notebooks.

One thing that we expect from an IDE (besides editors and code organization) is a possibility of debugging complicated Python modules and classes. So, in the final article of this series we shall have a brief look at (local) debugging of Python code within PyDEV. To set up a remote debugging server for any clients of our Linux machine is no major problem – but it is beyond the scope of this article.

As with the previous articles experienced Python and PyDev developers won’t learn anything new. My target group is students of ML and people intrested in Python who have some experience with other programming languages.

A “stupid” test example

We need a simple test code example to learn working with the debugging features of PyDev. We open our Eclipse installation and change to the PyDev perspective.

Watch out for the activated symbol at the right upper corner – here a list of different perspectives (which you have used recently) are displayed.

During the last article we had (within the PyDev-Explorer) already created a directory “mytestcode” below our basic project “ml_1”. By default settings of the PyDev environment, actually, we have created not only a directory but a so called Python “package” – which is a collection of modules that have a common context and should be delivered together. (You find more about “modules” and “packages” in any reasonable book on Python).

Now we add a (test) module – i.e. a file with some Python code – to our “package”.

In the next window (not shown we provide a name (without a “.py”-ending), e.g. “stupid” and in the subsequent window we choose a Template of type “<Empty>”. All these actions lead to the creation of a Python file “stupid.py” for which the Python editor of PyDev is opened:

Here we enter some “stupid” code:

def 
test(x=-1, y=-1):
    
    print("\nhelloworld: let us multiply and divide two POSITIVE INTEGERS < 10")
    if (x < 0):
        a = input("Give an integer value for a: ")
    else:
        a = x    
    if (y < 0):
        b = input("Give an integer value for b: ")
    else:
        b = y    
    print("a =" + a + ", b = " + b)
    
    if (a > 10):
        a=0 
    if (b > 10):
        b=0 
    
    c = int(a) * int(b)
    print("c=" + str(c))
    
    d = int(a) / int(b)
    print("d = " + str(d))
    return [c, d] 

Of course, you see already traps and lines doomed for failure; some of these traps we want to explore below by debugging :-). Debug As >> Other >> Python Run

By the way, we see a major disadvantage of the Eclipse environment (which does not only affect PyDEV): With “Oxygen” the outline view missed its capability to analyze beyond the first level of the code’s node hierarchy.

Our code does not contain any direct executable commands. We add 2 lines to be able to run it as a “Python Run” at the Python prompt of a console.

res = test()
print("c=" + str(res[0]) + ", d = " + res[1])

After this you may notice that the variable re is now displayed in the “outline view”.

Preparing debugging

Lets try to run our code:
Within the PyDev perspective we open a PyDev console for the interpreter of our virtualenv “ml1” (see the last article about how to do this). There we “import” our “stupid” code and answer the questions by typing in values for the variables a, b and pressing “Enter” each time. We end up with a first error message:

Not unexpected. Ok, lets turn to debugging. Eclipse offers us a special “perspective” which supports debugging. We open it by using the menu “Window >> Perspective >> Open perspective >> Other…“.

In the next window we choose “Debug”:

This leads to a change of the screen layout. The console view shows in the lower part of the screen that we have opened a “Debug console”. (You will see this in some of the following pictures).

There are a lot of new buttons available in the icon bar at the top. One in the middle shows a picture of a “bug” with an arrow besides it: obviously, there are options in what way to debug.

We now first click into our editor window for the “stupid” code and then on the arrow besides the “bug” button; you get something like this:

In my case there is a long list of previous debug runs.
However, in your case the list may be empty as you may never have launched any debug runs before.

As indicated in the screenshot you choose “Debug As >> Other >> Python Run” and click.

Unfortunately, the console view will now display an error message regarding “cython speedups”; this may look similar to the following:

I took the screen shot from another fresh installation – without a virtual Python environment… So the details of the message (especially the path to the interpreter) may look differently.

When I set up my PyDev environment with my virtualenv “ml1” the actual recommendation was to run the command

"/projekte/GIT/ai/ml1/bin/python3.6" "/projekte/eclipse/plugins/org.python.pydev.core_6.4.2.201807021651/pysrc/setup_cython.py" build_ext --inplace

The double quotes around the first 2 parts of the command are important in some command environments! On a Linux command console they do not do any harm.

The reason for this command is that we need to compile and install some additional cython related c-programs to get the “speedups” mentioned in the error message. It is not necessary to understand the details of this in our context.

What you have to do now is to start a Linux terminal window. There you move to the directory of your virtual Python environment – in my case to “/projekte/GIT/ai/ml1”. There you launch the command “source bin/activate” to activate the “virtualenv” with its interpreter.

Then you enter the following commands – Do NOT forget to activate the virtualenv “ml1” with the second command !

myself@mytux:~> cd /projekte/GIT/ai/ml1/
myself@mytux:/projekte/GIT/ai/ml1> source bin/activate
(ml1) myself@mytux:/projekte/GIT/ai/ml1> "/projekte/GIT/ai/ml1/bin/python3.6" "/projekte/eclipse/plugins/org.python.pydev.core_6.4.2.201807021651/pysrc/setup_cython.py" build_ext --inplace
running build_ext
building '_pydevd_bundle.pydevd_cython' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/_pydevd_bundle
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g -DOPENSSL_LOAD_CONF -fwrapv -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g -fPIC -I/usr/include/python3.6m -c _pydevd_bundle/pydevd_cython.c -o build/temp.linux-x86_64-3.6/_pydevd_bundle/pydevd_cython.o
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/_pydevd_bundle
gcc -pthread -shared -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none build/temp.linux-x86_64-3.6/_pydevd_bundle/pydevd_cython.o -L/usr/lib64 -lpython3.6m -o build/lib.linux-x86_64-3.6/_pydevd_bundle/pydevd_cython.cpython-36m-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.6/_pydevd_bundle/pydevd_cython.cpython-36m-x86_64-linux-gnu.so -> _pydevd_bundle
running build_ext
building '_pydevd_frame_eval.pydevd_frame_evaluator' extension
creating build/temp.linux-x86_64-3.6/_pydevd_frame_eval
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g -DOPENSSL_LOAD_CONF -fwrapv -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g -fPIC -I/usr/include/python3.6m -c _pydevd_frame_eval/pydevd_frame_evaluator.c -o build/temp.linux-x86_64-3.6/_pydevd_frame_eval/pydevd_frame_evaluator.o
creating build/lib.linux-x86_64-
3.6/_pydevd_frame_eval
gcc -pthread -shared -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none build/temp.linux-x86_64-3.6/_pydevd_frame_eval/pydevd_frame_evaluator.o -L/usr/lib64 -lpython3.6m -o build/lib.linux-x86_64-3.6/_pydevd_frame_eval/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so                                                             
copying build/lib.linux-x86_64-3.6/_pydevd_frame_eval/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so -> _pydevd_frame_eval                                            
(ml1) myself@mytux:/projekte/GIT/ai/ml1> 

 

After this it is reasonable to restart Eclipse via the menu point “File >> Restart“. if you first want to clean up the started Debug runs you could first click on the double red square icon in the console view’s icon bar:

and afterward on the double “x”-icon:

If you looked a bit to the leftmost window “Debug” of the debug perspective during these actions, you would have seen some changes there :-). What you did was to forcefully terminate and remove all launched debug runs from the debug environment!

Then restart by “File >> Restart”. Your Eclipse application should automatically start again – with the perspective and files open that you last worked with.

Breakpoints

Now we again click into the editor window for “stupid.py” and move the mouse over the bug-like icon in the top icon bar

and click. Now you should get something like the following – without errors for the debugger:

Our program code has started automatically due to the first of our final two commands (“res=test()”).

Note that the console has two parts – an upper one, where output from the code is shown and a lower part with a Python prompt.
The lower part is for interactive command execution during debugging. We ignore for the time being and only work in the upper part. This is, by the way, the area where the “input” command of Python prompts us to enter values.

There we enter 2 values for our variables a and b and … Of course, we run again into our first error. I have not displayed this as we would learn nothing new from it.

The reason, of course, is: We debugged without having set breakpoints before.

Setting breakpoints is fortunately easy: We move to our PyDev editor view. On the left side we see line numbers (if not: right click on the outer left border stripe of the editor view to get related options!). We right-click on the left side stripe of the line where we want to set a breakpoint to force a stop of the code execution there (i.e. before execution the line’s command).

Hint: Setting and removing breakpoints can also be done by left double-clicking on the leftmost black stripe of the editor.

We add 3 breakpoints for a start – at line 8, 17 and 19 :

Debugging

Our debugger should be in a state where we can just re-debug the present code. There are 2 possibilities for doing so: In the icon bar of the debug console view we can click on the green “Relaunch”-button:

Or you right-click on the relevant entry on the leftmost view on the last debug runs (which is only one right now) and choose the “Relaunch” option in the context menu:

Code execution stops at our first breakpoint – the correspondent line gets marked in the editor; in the “Variables View” on the upper right we see the values of the variables (x, y) set so far:

How to resume code execution?

This is where the different “player” icons in the top icon menu bar enter the game. You should explore these buttons in detail; for our introduction we just need the green “Resume“-button (and maybe also the red “Terminate“-button):

By clicking on the “Resume”-button code-execution is performed until the next break-point; we enter two integers for a and b again, and the code stops at line 17:

In the variables view we see now that a and b are strings! It is therefore that line 19 will produce an error – as we have learned before. We resume and end at line 19:

Resuming code execution now would again lead to an error. Let us, therefore, change the code, cast a, b to integers and save the modified file. However, this leads to an error message:

The debug environment lost track of our python code file.

Unfortunately, resuming now leads to our old error – despite the file changes:

Hint:
Whilst debugging as a Python run even reloading the “stupid”-module would not help.

Actually, what we do have to is to relaunch the whole debug process.

Whilst in an un-terminated debug session we can do this by simply pressing the red button with the yellow resume arrow:

Doing so and moving from breakpoint to breakpoint leads again into trouble – however now at a different code line, namely the final one (line 32):

You know,of course, why. But the code of our function test() in between has worked flawlessly now.

Instead of adding another breakpoint or performing the necessary code change for the last line, we try a different way of debugging – namely by linking a standard PyDev console to the debugger.

To clean up our workspace we terminate and remove all running debug runs first. We could use the buttons of the debug console for it as we have learned it above. But this time you could also try and use the leftmost “Debug View” and right-click on the relevant run there to get an option “terminate and Remove”.

Connect a PyDev command console to Debug sessions

We open “Window >> Preferences >> PyDev >> Interactive Console” and activate the checkbox for the point “Connect Console to a debug session?

Then we provoke a new error in our code at line 27 by changing it to:

d = a / int(b)

In addition we remove our last 2 lines

res = test()
print("c=" + str(res[0]) + ", d = " + res[1])

Thereby, importing module “stupid” in a console will NOT cause direct code execution 🙂 .

Let us start a PyDev console for the interpreter of our virtualenv (see the last article if you have forgotten how to do it). This gives us:

As expected we get a PyDev-console with a Python prompt.
The interesting thing, however, is indicated on the left screen side:

The entries in the “Debug View” show that we obviously started some debug run! Actually, our console is now part of a debugging session.

We can type anything at the prompt; we may e.g. define a variable “myvar”. Unfortnately it is not shown in the variables view, but we can get its value at any time when (and if) we have the prompt available :

As we are in an interactive environment we must import our module and do something with the test-function (as no code is executed automatically):

from 
mytestcode import stupid
res = stupid.test()

This brings us to our first break point:

We add a breakpoint to line 27, which will lead to an error – and march from breakpoint to breakpoint until we reach line 27:

Variable “a” is a string and would cause trouble. Let us change the code to “d = int(a) / int(b)” and save the modified file. We again get an error :

pydev debugger: Unable to find module to reload: "stupid".   

If we now resume code execution we will run into the foreseen trouble:

But afterwards we still have an active Python prompt of our Pydev-console for our session. We should be able to reload the code.

For Python 3.6 the following is required to do so:

By using the up-arrow then we move backward through the command history, start “res=stupid.test()” again and move via the break-points to the end – no errors any more (at least for the chosen a- and b-values):

Of course our file “stupid.py” contains a lot more sources for errors. E.g. there is no check or exception handling for a division by zero. And we have not used the function’s parameters, yet. I leave it to the reader to experiment with respective debugging.

How to use the prompt in the normal Debug console?

Now that you know the basics of how to debug, there is one more thing worth mentioning:

When you debug something as a Python Run in the standard “Debug Console”, you may have noticed the (green) Python prompt in the lower part of the console view:

What is it good for?

Well, with it you can interactively change variables and do other things interactively within the context of the debugged object (here of function test()).

Important note:
To avoid inconsistent states you should use the interactive prompt only, when the debugger stopped at a breakpoint. In addition you should enter values for variables requested by “input”-statements only in the upper part of the debug console!

But as soon as the debugger stops you can interactively ask e.g. for values of variables:

As soon as you press Enter in the lower part the
command is reflect in the upper part of the Debug console and results are shown there, too. So, while the debugger stops at a break point, you can do a lot of interactive things; you may set some new variables and use them later on – this is very useful when you want to keep some intermediate results for later purposes during a debug session.

Such new, interactively set variables will, by the way, even be shown in the Variables View (watch out for variables “alp” and “ufo”):

Customization of the debug environment

Eventually, a hint regarding customization of the environment discussed so far.

You should now have a look at all sub-points of the the two following menu points under “Window >> Preferences“:

Of course, there are the sub-points of “PyDev”:
There you can detect a lot more properties of e.g. the “Interpreters” (e.g. of your virtualenv environments) and e.g. the “Interactive Console“.

But Another interesting point is “Run/Debug“:
After a while you may want to use some of the properties there to make life easier during debug sessions. See e.g. for the length of the list of the last debug runs menu point “Run/Debug >> Launching >> Size of recently launched application list”.

Although I cannot discuss it in this article:
Another thing you should become familiar with is the configuration of Debug (and unittest) Runs.
You find a screen for it when you open the combobox of the main debug icon in the top icon bar of Eclipse and click on the point “Debug configurations …” there.

Unit tests

Interactively debugging is nice and useful. Something that is equally important in the long run is, however, “unit testing”. There are two interesting mechanisms that Python provides for doing this: The “doctest”-module and “unittest”. Both are beyond the scope of our introduction to PyDev. You find more about these things on the Internet or in reasonable books on Python.

Note, however, that Python’s “unittest” is especially interesting as the Debug environment of Eclipse/PyDev provides some nice views and tools for it. The following links may give you a first introduction:

https://www.youtube.com/watch?v=1Lfv5tUGsn8
https://www.youtube.com/watch?v=fU7RHewj6dg
https://www.blog.pythonlibrary.org/2016/07/07/python-3-testing-an-intro-to-unittest/

Conclusion

In this mini series of articles I have tried to show that setting up Eclipse with PyDev is relatively easy. This gives anybody interested in doing Machine-Learning-experiments with Python, Tensorflow and Keras a reasonable and cost free environment where you can build up and test solid code. A starting point could be module code which you export from a Jupyter notebook after some first trial experiments.

I will provide some examples for a two folded approach with Jupyter notebooks and code refinement via PyDev later on in this
blog.

One may ask now, why do we need Jupyter notebooks at all for machine learning experiments. Well, one big advantage of a Jupyter environment is the fact that we can arrange blocks of Python commands in a cell and re-execute the code-blocks in a self-chosen order again. So, it is possible to redo experiments quickly in a modified way; Jupyter gives you – in my opinion – a bit more flexibility in executing preliminary code fragments in a quick and (admittedly) dirty way. In PyDev you would have to work with editors and maybe multiple files to something similar.

Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – II

Developing and organizing efficient code after preliminary experiments in machine learning [ML] requires an IDE. This mini-series of articles deals with the setup of a Python environment which supports Eclipse – and Jupyter notebooks. A key ingredient is “virtualenv”: it defines an encapsulated environment for a particular python interpreter together with a specific collection of library packages. In the last article

Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – I

we prepared such a virtual Python3 environment “ml1” at a path “/projekte/ai/ml1” and installed some of the basic ML packages there with the help of the “pip3”-mechanism. Within Eclipse we implemented the PyDev plugin. During the setup of a “Python project” we could refer to our “ml1“-environment by defining paths to the Python interpreter and library packages located there.

Changes of the PYTHONPATH from Eclipse/PyDev

To integrate our future own Python modules into interactive experiments we need to add the paths to our own Python file directories into the PYTHONPATH variable. We expect that this should be possible from within Eclipse – and indeed it is on the project level.

In the left Eclipse view of the “PyDev explorer” we add an example directory “mytestcode”; we do this by a right-click on “ml1″ >> New >> folder” and giving the new folder a name in the eventual popup

As soon as the new folder appears we right-click on the root folder of our project “ml_1” in the PyDeev package explorer; in the appearing window we click on “Properties” and get:

There, we choose “PyDev – PYTHONPATH”. By clicking on the button “Add source folder” we can add a folder, e.g. “mytestcode”.

From now on we can import modules in any interactive Python command environment from this directory.

Python console in Eclipse

To perform experiments within an IDE as Eclipse we need some interface to interactively run Python commands and programs. A basic interface for this purpose is a “console”. PyDev, of course, offers a special Python console. How to start it?

If you have chosen a Python perspective within Eclipse you may already see a view area with a console. We start, however, from a perspective where no console view is open, yet:

To add the console view area we use the menu point “Window >> Show View >> Console”.

This gives us:

n

We got a “Debug console” – not exactly, what we want right now. So, let us open a new console view:

Again a debug console – but we change this now to a PyDev console:

At last, we get a popup where we can choose between a number of defined Python interpreters for command execution. You should at least see 2 items here: A reference to the Linux-system’s Python installation’s interpreter plus a reference to the interpreter configuration of the virtual Python environment, which we had set up in the last article. We had given it the name “python_ml1”.

We chose it; in my case this results in the following view:

Ok, we have a Python prompt (>>>) – but a bunch of error messages, too… The error messages indicate that something to access the graphical environment is missing; PyDEV’s console actually has recognized that it needs an Qt5-based interface to the desktop.

The reason for this is that I had done some customization of the “PyDev” console beforehand; when you look at the choices of “Window >> Preferences” you may find something like this:

here, the setting for “Enable GUI event-loop integration” is interesting: I had chosen the option “PyQt5(qt5)” from the combobox. To me this seemed to be a natural choice on a Qt5-based KDE Linux desktop. Remember, I had the Qt5 python modules installed on my Linux system … Well, error messages nevertheless …

Does the console work at all? Can we use “matplotlib”?

We briefly test whether the Python console works at all:

Yes! And:

We actually do get a reasonable output from “matplotlib”! However, this is NOT based on a “Qt5”-backend, but “TkAgg” (which we can see by the graphical layout of buttons). Where does this come from? And why the complaint of our console about “Qt5”?

Let us try another option from the Combobox : Tkinter(tk).

And then starting yet another console:

Hey, no error messages! This is again a strong indication that some things are missing.

Enable Qt5!

A natural guess is that we need PyQt5 within our virtual environment. Have a look at the Interpreters by choosing
“Window >> Prefrences>> PyDev >> Interpreters >> Python Interpreter”.

There we find no path to the system’s directory for “site-packages”; only the path to thw “ml1”-environments site-package directory is included in the PYTHONPATH. Now, we use “pip” from within Eclipse. This can be done by choosing our “python_ml1” in the upper area and then clicking on “Packages”:

No PyQt5 there – but a button “Install/Uninstall with pip”; we confidently use it:

We terminate all our consoles, we reset the “Interactive console settings” for the GUI event loop integration” (see above) to “PyQt5” and start again a new console for our environment’s “python_ml1” interpreter:

Good! No error messages any more; and:

Yeah, that’s what we want!

Other matplotlib-settings

You should also be aware of the fact that the backend for “matplotlib” may also be defined in a specific configuration file of your environment. In my case we find the relevant file at “/projekte/GIT/ml_1/lib64/python3.6/site-packages/matplotlib/mpl-data/matplotlibrc“.

There you find a commented entry

# backend: :Agg” ,

which you could un-comment and set to a default of “Qt5Agg”. But this is only seldomly required:

Reading the information text in matplotlibrc, we see that Qt5Agg was
automatically chosen as the first working backend of a list of possible backends: MacOSX Qt5Agg Qt4Agg Gtk3Agg TkAgg WxAgg Agg.

By the way this together with the information at https://askubuntu.com/questions/1045720/what-is-a-good-default-backend-for-matplotlibexplains explains why TkAgg worked.

Console colors and command history

Via “Window >&gt: Preferences >> PyDev >> Interactive Console” we can adjust the console colors. I use the following settings to get a dark background:

Command history: The PyDev console, of course, also allows for scrolling through commands but the arrow-up/down-keys. The number of commands can be set via the option “Maximum number of lines to store in global history …”.

Conclusion

A basic Eclipse/PyDev environment which supports a “virtual Python environment (virtualenv) and graphical output in Qt5 quality is set up quickly. We can use it as a tool to collect, rectify and optimize code of experimental Jupyter sessions in Python source files.

In the next article

Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – III

we shall have a brief look at debugging local Python code in PyDev.

Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – I

When you enter the field of machine learning [ML] and Artificial Intelligence [AI] there is no way around Python. And whilst studying books like “A. Geron’s Machine Learning with SciKit-Learn & TensorFlow” [1] or F. Chollet’s “Deep learning with Python and Keras” [2] one understands quickly: You do not learn by reading, but by doing experiments.

For me this meant to both improve my basic Python knowledge and to set up a reasonable working environment on my Linux workstation (with Opensuse Leap Linux and KDE). The named books recommend using “Jupyter notebooks” – and I must say, Jupyter environments are fun to use. However, as soon as I started with more complex program variations I began missing an IDE. I think that in the end Python code must be organized in a more systematic way than during experiments with Jupyter notebooks. A Jupyter notebook serves one purpose, a Python IDE a supplemental one.

A natural choice for an IDE based on opensource tools is Eclipse with PyDev. After a basic setup I stumbled across two problems:

  • For projects a so called “virtual” Python environment is useful, which encapsulates a defined mix of Python and library versions. How to use “virtualenv” within PyDev and its Python specific console?
  • Quite often the results of ML/AI-experiments need to be represented in a graphical way. Browser based “Jupyter notebooks” make the use of graphics easy by using browser capabilities. But how to use Python’s matplotlib in my Opensuse/KDE/Eclipse environment?

In this article I address the steps to setup Eclipse/PyDev in such a way that both points are covered. I do this for an Opensuse Leap system, but a transfer to other Linux distributions should be simple. The group of readers I address is either ML-interested folks for whom Eclipse is a new environment or people as me who know Eclipse but not the PyDev plugin. People who already work with PyDev will probably not learn anything new.

Step 1: Install Eclipse

A basic Eclipse installation is a straightforward business on most Linux distributions ( see e.g.: https://simopr.wordpress.com/2016/05/26/install-eclipse-ide-on-opensuse-leap-42/). I will, therefore, not cover this topic in detail here. You first need to install a Java Runtime environment (on Opensuse via the RPM java-10-openjdk), if not yet provided by your distribution. A current version of Eclipse can be downloaded from the site
https://www.eclipse.org/downloads/packages/.
(Actually, I used my already installed Eclipse photon version 4.9.0 of September 2018 – which works pretty well for me. But the present 2019 RC3 candidate of Eclipse should work as well.)

To my knowledge there is no special Eclipse package for Python developers; as a PHP-developer I choose the package for PHP-developers for a basic Eclipse installation and install the required Python PyDev packages afterwards.

You download your chosen tar.gz-file from the Eclipse site named above, save it and then expand its contents into a suitable directory of your Linux system (in my case into “/projects/eclipse”). Then you can directly start the executable “eclipse”-file there – e.g. in a terminal.

Then you need to define your path for a “workspace” for your Python projects. Note that the workspace is not necessarily identical with a root directory for all your project files; a workspace instead gathers information on your configuration settings for Eclipse and defined projects. The project files themselves, however, can be located in a very different place – e.g. in a directory defined for your local GIT platform – in my case below “/projects/GIT/…”.

Eventually, you get a full fledged Eclipse IDE interface, which you
can customize (see “Window >> “Preferences”). This is beyond the scope of this article; I give however some hints regarding color. You can e.g. customize editor and console colors for specific programming languages within Eclipse.

However, regarding certain application control elements you may nevertheless run into trouble regarding the definition of colors; one reason is that on a Qt5-based KDE desktop the end result may depend both on Eclipse settings and also on desktop design schemes for GTK2/GTK3 applications as Eclipse. This type of dependency requires experiments. So, what exactly do I use?

Within Eclipse itself I use the “Dark Theme” – to avoid an eye sore whilst programming.

Regarding my KDE desktop I use a standard Breeze Desktop Scheme with Elegance-Design and the Standard Color Theme (with the activation flag for non-Qt-applications set). KDE application design elements, however, are taken from the Adwaita-Scheme. For GTK2 applications on KDE I prefer the Clearlooks-design, for GTK3 applications – as Eclipse (> 4.9.0) – again Adwaita. This combination gives me a sufficient foreground/background-contrast for control elements like checkboxes, radio buttons, …

A last convenience point: In a graphical desktop environment as KDE you will of course add some icon to your desktop (in my case with a reference to the file “projects/eclipse/eclipse”) to reduce the starting process to a click.

Step 2: Basic Python packages on the system level

I assume that you have already installed Python in your Linux-(Opensuse)-system. In my environment I use the Python 3.6 RPM-packages from the standard repositories for Opensuse Leap 15.0:
https://download.opensuse.org/distribution/ leap/15.0/repo/oss/
https://download.opensuse.org/update/ leap/15.0/oss/.

The number of available Python library packages is quite big; what libraries you should install depends on your programming objectives. You need at least the basic “python3” package. Another “must”, in my opinion, is the package “python3-pip“; it enables us to perform specific package installations for our “virtual Python environment” later on.

As a basic ingredient for graphics you may also install suitable libraries for your Linux desktop environment. In my case this is KDE – so I installed the packages “python3-qt5″, python-qt5-utils” and also “python3-qt5-devel” to be on the safe side. However, as we shall see we may need Qt5-packages within a project environment, too. That is where Python’s internal “pip” mechanism enters the game.

Below we shall perform the installation of the “virtualenv” package to demonstarte the usage of “pip” or “pip3” in a Python3-environment. As a first step I provide myself (i.e. user “myself”) with a current version of “pip3”:

myself@mytux:~> pip3 --version
pip 19.1.0 from /home/myself/.local/lib/python3.6/site-packages/pip (python 3.6)
myself@mytux:~> pip3 install --user --
upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl (1.4MB)
     |████████████████████████████████| 1.4MB 1.0MB/s 
Installing collected packages: pip
  Found existing installation: pip 19.1                                                                                                                                                 
    Uninstalling pip-19.1:                                                                                                                                                              
      Successfully uninstalled pip-19.1                                                                                                                                                 
Successfully installed pip-19.1.1                                                                                                                                                       
myself@mytux:~> pip3 --version
pip 19.1.1 from /home/myself/.local/lib/python3.6/site-packages/pip (python 3.6)

You see that the parameter “–user” already lead to a personal configuration of basic Python packages (within my home-directory). But we shall specify a project specific environment in the fourth step.

Step3: Working directory for our ML-project

We now define a base directory “ai” for future experiments.

myself@mytux:~> export AI_PATH ="/projekte/GIT/ai/"
myself@mytux:~> mkdir -p $AI_PATH

A sub-directory “ml1” shall provide the environment for a bunch of initial basic ML-experiments and related Python code files, libraries, Jupyter notebooks, etc.. We create this “ml1” directory as a base for a “virtual” Python environment.

Step 4: Prepare a virtual Python environment via virtualenv and working directories

Python installations allow for the definition of a so called “virtual environment” for projects via the “virtualenv” add-on. Among other things “virtualenv” lets you define a project specific configuration with Python and library versions in a consistent reproducible state. This in turn gives you a base for the “configuration management” of complex endeavors; therefore, I strongly recommend to make use of this feature – also in combination with PyDev: .

myself@mytux:~> pip3 install --user --upgrade virtualenv
Collecting virtualenv
  Downloading https://files.pythonhosted.org/packages/ca/ee/8375c01412abe6ff462ec80970e6bb1c4308724d4366d7519627c98691ab/virtualenv-16.6.0-py2.py3-none-any.whl (2.0MB)
     |████████████████████████████████| 2.0MB 1.6MB/s 
Installing collected packages: virtualenv
  Found existing installation: virtualenv 16.5.0
    Uninstalling virtualenv-16.5.0:
      Successfully uninstalled virtualenv-16.5.0
Successfully installed virtualenv-16.6.0
myself@mytux:~> virtualenv --version
16.6.0
myself@mytux:~>

Now we can use “virtualenv” to setup the virtual Python environment for “ml1” in our “ai”-directory:

myself@mytux:~> cd /projekte/GIT/ai/
myself@mytux:/projekte/GIT/ai> virtualenv ml1
Using base prefix '/usr'
  No LICENSE.txt / LICENSE found in source
New python executable in /projekte/GIT/ai/ml1/bin/python3
Also creating executable in /projekte/GIT/ai/ml1/bin/python
Installing setuptools, pip, wheel...
done.
myself@mytux:/projekte/GIT/ai> la ml1
insgesamt 20
drwxr-xr-x 5 myself users 4096 25. Mai 15:05 .
drwxr-xr-x 3 myself users 4096 25. Mai 15:05 ..
drwxr-xr-x 2 myself users 4096 25. Mai 15:05 bin
drwxr-xr-x 2 myself users 4096 25. Mai 15:05 include
drwxr-xr-x 3 myself users 4096 25. Mai 15:05 lib
lrwxrwxrwx 1 myself users    3 25. Mai 15:05 lib64 -> lib
myself@mytux:/projekte/GIT/ai> la ml1/bin
insgesamt 72
drwxr-xr-x 2 myself users  4096 25. Mai 15:05 .
ndrwxr-xr-x 5 myself users  4096 25. Mai 15:05 ..
-rw-r--r-- 1 myself users  2096 25. Mai 15:05 activate
-rw-r--r-- 1 myself users  1428 25. Mai 15:05 activate.csh
-rw-r--r-- 1 myself users  3052 25. Mai 15:05 activate.fish
-rw-r--r-- 1 myself users  1804 25. Mai 15:05 activate.ps1
-rw-r--r-- 1 myself users  1512 25. Mai 15:05 activate_this.py
-rw-r--r-- 1 myself users  1150 25. Mai 15:05 activate.xsh
-rwxr-xr-x 1 myself users   249 25. Mai 15:05 easy_install
-rwxr-xr-x 1 myself users   249 25. Mai 15:05 easy_install-3.6
-rwxr-xr-x 1 myself users   231 25. Mai 15:05 pip
-rwxr-xr-x 1 myself users   231 25. Mai 15:05 pip3
-rwxr-xr-x 1 myself users   231 25. Mai 15:05 pip3.6
lrwxrwxrwx 1 myself users     7 25. Mai 15:05 python -> python3
-rwxr-xr-x 1 myself users 10456 25. Mai 15:05 python3
lrwxrwxrwx 1 myself users     7 25. Mai 15:05 python3.6 -> python3
-rwxr-xr-x 1 myself users  2338 25. Mai 15:05 python-config
-rwxr-xr-x 1 myself users   227 25. Mai 15:05 wheel
myself@mytux:/projekte/GIT/ai> 

You see that a whole directory structure was established – with Python3 executables copied from our basic system installation. We can fully use this Python environment already on the command line (of a terminal window). However, we need to activate it so that its files and libs are really used:

myself@mytux:/projekte/GIT/ai/ml1> source bin/activate  
(ml1) myself@mytux:/projekte/GIT/ai/ml1> python3 
Python 3.6.5 (default, Mar 31 2018, 19:45:04) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello World!")
Hello World!
>>> quit()
(ml1) myself@mytux:/projekte/GIT/ai/ml1> pip3 install --upgrade jupyter                                                                                                                      
Collecting jupyter                                                                                                                                                                      
  Using cached https://files.pythonhosted.org/packages/83/df/0f5dd132200728a86190397e1ea87cd76244e42d39ec5e88efd25b2abd7e/jupyter-1.0.0-py2.py3-none-any.whl                            
Collecting notebook (from jupyter)
...
..Successfully built pyrsistent
Installing collected packages: Send2Trash, ipython-genutils, decorator, six, traitlets, jupyter-core, MarkupSafe, jinja2, pyzmq, python-dateutil, tornado, jupyter-client, backcall, pickleshare, wcwidth, prompt-toolkit, ptyprocess, pexpect, pygments, parso, jedi, ipython, ipykernel, prometheus-client, pyrsistent, attrs, jsonschema, nbformat, terminado, entrypoints, mistune, webencodings, bleach, testpath, defusedxml, pandocfilters, nbconvert, notebook, jupyter-console, widgetsnbextension, ipywidgets, qtconsole, jupyter
Successfully installed MarkupSafe-1.1.1 Send2Trash-1.5.0 attrs-19.1.0 backcall-0.1.0 bleach-3.1.0 decorator-4.4.0 defusedxml-0.6.0 entrypoints-0.3 ipykernel-5.1.1 ipython-7.5.0 ipython-genutils-0.2.0 ipywidgets-7.4.2 jedi-0.13.3 jinja2-2.10.1 jsonschema-3.0.1 jupyter-1.0.0 jupyter-client-5.2.4 jupyter-console-6.0.0 jupyter-core-4.4.0 mistune-0.8.4 nbconvert-5.5.0 nbformat-4.4.0 notebook-5.7.8 pandocfilters-1.4.2 parso-0.4.0 pexpect-4.7.0 pickleshare-0.7.5 prometheus-client-0.6.0 prompt-toolkit-2.0.9 ptyprocess-0.6.0 pygments-2.4.1 pyrsistent-0.15.2 python-dateutil-2.8.0 pyzmq-18.0.1 qtconsole-4.5.0 six-1.12.0 terminado-0.8.2 testpath-0.4.2 tornado-6.0.2 traitlets-4.3.2 wcwidth-0.1.7 webencodings-0.5.1 widgetsnbextension-3.4.2
(ml1) myself@mytux:/projekte/GIT/ai/ml1/include> cd ../bin
(ml1) myself@mytux:/projekte/GIT/ai/ml1/bin> la
insgesamt 152
drwxr-xr-x 2 myself users  4096 26. Mai 14:22 .
drwxr-xr-x 7 myself users  4096 26. Mai 14:22 ..
-rw-r--r-- 1 myself users  2096 25. Mai 15:05 activate
-rw-r--r-- 1 myself users  1428 25. Mai 15:05 activate.csh
-rw-r--r-- 1 myself users  3052 25. Mai 15:05 activate.fish
n-rw-r--r-- 1 myself users  1804 25. Mai 15:05 activate.ps1
-rw-r--r-- 1 myself users  1512 25. Mai 15:05 activate_this.py
-rw-r--r-- 1 myself users  1150 25. Mai 15:05 activate.xsh
-rwxr-xr-x 1 myself users   249 25. Mai 15:05 easy_install
-rwxr-xr-x 1 myself users   249 25. Mai 15:05 easy_install-3.6
-rwxr-xr-x 1 myself users   250 26. Mai 14:22 iptest
-rwxr-xr-x 1 myself users   250 26. Mai 14:22 iptest3
-rwxr-xr-x 1 myself users   243 26. Mai 14:22 ipython
-rwxr-xr-x 1 myself users   243 26. Mai 14:22 ipython3
-rwxr-xr-x 1 myself users   232 26. Mai 14:22 jsonschema
-rwxr-xr-x 1 myself users   238 26. Mai 14:22 jupyter
-rwxr-xr-x 1 myself users   252 26. Mai 14:22 jupyter-bundlerextension
-rwxr-xr-x 1 myself users   237 26. Mai 14:22 jupyter-console
-rwxr-xr-x 1 myself users   242 26. Mai 14:22 jupyter-kernel
-rwxr-xr-x 1 myself users   280 26. Mai 14:22 jupyter-kernelspec
-rwxr-xr-x 1 myself users   238 26. Mai 14:22 jupyter-migrate
-rwxr-xr-x 1 myself users   240 26. Mai 14:22 jupyter-nbconvert
-rwxr-xr-x 1 myself users   239 26. Mai 14:22 jupyter-nbextension
-rwxr-xr-x 1 myself users   238 26. Mai 14:22 jupyter-notebook
-rwxr-xr-x 1 myself users   240 26. Mai 14:22 jupyter-qtconsole
-rwxr-xr-x 1 myself users   259 26. Mai 14:22 jupyter-run
-rwxr-xr-x 1 myself users   243 26. Mai 14:22 jupyter-serverextension
-rwxr-xr-x 1 myself users   243 26. Mai 14:22 jupyter-troubleshoot
-rwxr-xr-x 1 myself users   271 26. Mai 14:22 jupyter-trust
-rwxr-xr-x 1 myself users   231 25. Mai 15:05 pip
-rwxr-xr-x 1 myself users   231 25. Mai 15:05 pip3
-rwxr-xr-x 1 myself users   231 25. Mai 15:05 pip3.6
-rwxr-xr-x 1 myself users   234 26. Mai 14:22 pygmentize
lrwxrwxrwx 1 myself users     7 25. Mai 15:05 python -> python3
-rwxr-xr-x 1 myself users 10456 25. Mai 15:05 python3
lrwxrwxrwx 1 myself users     7 25. Mai 15:05 python3.6 -> python3
-rwxr-xr-x 1 myself users  2338 25. Mai 15:05 python-config
-rwxr-xr-x 1 myself users   227 25. Mai 15:05 wheel

Looking into the lib-directory is also informative. I leave this to the user.

(ml1) myself@mytux:/projekte/GIT/ai/ml1/bin> cd ../lib/python3.6/site-package
(ml1) myself@mytux:/projekte/GIT/ai/ml1/lib/python3.6/site-packages> la

Step 5: Install some important libraries for ML studies

As we are occupied with installing packages, let us get some more packages typically required to do experiments for AI/ML:

(ml1) myself@mytux:/projekte/GIT/ai/ml1> pip3 install --upgrade matplotlib numpy pandas scipy scikit-learn
....

Step 6: Install PyDev for Eclipse

The previous steps were all on the level of the Linux-system and/or for a special Python environment for me as a user. But Eclipse does not know anything about Python, yet. We need a special Python environment within Eclipse with suitable editors, project and test environments, configuration options and so on for our Python based machine learning projects.

You find the necessary PyDev plugins for Eclipse at the site http://pydev.sf.net/updates/.

The easiest way to install PyDev is: Add this site to the update configuration of Eclipse – via the menu point “Help >> Install new software”. Click the “Add”-Button there. In the popup you provide a name for the site and its URL. Then you choose this site “to work with” and click on the relevant plugin “PyDev for Eclipse”. If you are a fan of Mylyn you also load the respective package.

Step 7: Change to a PyDev perspective within Eclipse

After having installed the PyDev packages we can start Eclipse and change
the layout by choosing a Python specific “perspective“.

We start with the menu point
“Window >> Perspective >> Open Perspective >> Other …”

Then we choose “PyDev” and end up with the a layout of Eclipse similar to the following (you may have some other position arrangements of the sub-windows):

On the left side you see some projects, which I had set up already. (As I integrate some of my Python experiments with PHP-programs the reader may detect some PHP-projects, too …). In the lower right part of the IDE we see a console view for interactive python commands. I come back to this point below.

Step 8: Add a Python project in Eclipse for our virtual environment ml1

We now create a new project which shall be related to our directory “/projekte/GIT/ai/ml1”. A right mouse click into the leftmost area gives us:

On the next popup we choose a “PyDev”-project type.

On the third screen we first enter our path “/projekte/GIT/ai/ml1” – with this setting we see all the modules and libraries loaded for our virtual environment in Eclipse, too.

The important interpreter setting – it decides on the usage of our virtualenv
Really interesting is the field for the choice of an “Interpreter“. Here we get the option to refer to our “virtual environment”. When we click on the blue link we can configure an interpreter and related path settings. On the opening popup window we enter the path to the interpreter of our ml1-environment, i.e. to “/projekte/GIT/ai/ml1/bin/python3.6“.

We go on and get

Important: We do not delete the references to the systems libraries here!

We move on and come back to our project definition window – we now
choose the interpreter “python_ml1” which we defined a minute ago.

On the next screen we do not yet have any other projects to be referenced.

So we finish and get our first Python3 project:

Enough for today. In the second article

Eclipse, PyDev, virtualenv and graphical output of matplotlib on KDE – II

of this series we shall use a Python-console within Eclipse for interactive coding and the display of results. We shall see that we need additional settings to get matplotlib to work.

Stay tuned …

Links

https://www.caktusgroup.com/blog/2011/08/31/getting-started-using-python-eclipse/