Corona App – Standortbestimmung

Posted on 22. June 2020 by eremo

Ich bin technologie-affin, aber ich bin kein Freund von Datenkraken. Ich finde eine Corona-App wichtig und richtig – aber wenn sie denn endlich eingeführt wird, dann doch bitte in Gänze mit offenen Karten.

Ich habe die deutsche Corona-App nun seit einer Woche auf meinem Android-Smartphone installiert und bin zunehmend verärgert. Zur Irritation beigetragen hat gestern die Lektüre einer Leser-Diskussion in der Zeit zum Thema; siehe die Debatte zum Artikel
https://www.zeit.de/digital/mobil/2020-06/tracing-app-corona-datenschutz-standortdaten-bluetooth-virus-bekaempfung.

Da überziehen sich die Debattanten mit dem Vorwurf der Unkenntnis [“keine Ahnung”, “Unwissenheit”, …] und sind z.T. stolz darauf, dass sie selbst (die Schlaumeier) wissen, dass man Zugriffsrechte (begrenzt) pro App regeln kann (übrigens auf die Gefahr hin, dass die betroffene App dann nicht mehr funktioniert). Scheingefechte unter Möchtegern-Gurus … und aus meiner Sicht eine Thema-Verfehlung.

Das eigentliche Thema ist, dass die Aktivierung der Corona-App vom Anwender die Freigabe der Standortbestimmung grundsätzlich und in pauschaler Weise verlangt, obwohl die Corona-App selbst die dadurch ermittelbaren Daten gar nicht nutzt.

Vorgeblicher Grund für die notwendige Freigabe: Nicht durchschaubare Android-Spezifika seit der Android Version 6.0, die ab der Version 8 aber wieder aufgeweicht wurden. Eine pauschale Freigabe der Standortbestimmung ist natürlich ein gefundenes Fressen für viele andere auf einem Android-Smartphone installierte Apps (darunter natürlich an dominanter Stelle etliche Standard-Apps von Google).

Man müsste also nach der Installation der Corona-App alle anderen Anwendungen Stück für Stück bzgl. der Rechte konfigurieren. Auf das notwendige Verfahren weist einen die Corona-App aber gar nicht hin! Und ich bezweifle, dass jeder Anwender jede installierte App auf die Rechtesetzung überprüfen würde.

Die Aktivierung der Corona-App bringt also (wirtschaftliche) Vorteile für Google und andere App-Ersteller mit sich, die weit über die Entwicklungsgelder für die Corona-App hinaus reichen. Man reibt sich verwundert die maskengereizten Wangen …

Ich habe in der vergangenen Woche mehrere Leute (8) – darunter auch 4 IT-ler – danach befragt, ob Ihnen bewusst sei, dass die Aktivierung der Corona-App auf ihrem Handy zu einer Freigabe der Standortbestimmung geführt habe. Sieben wussten davon nichts. Drei entgegneten mir, das sei ja gerade bei der Corona-App nicht der Fall. Eine hatte etwas während der Installation gelesen, aber dann weiter geklickt …

Wahrlich Zeit für eine “Standortbestimmung” in puncto Corona-App und zugehörige Vorinformationen durch Regierung und Presse. Gerade, weil ich selbst kein Android-Experte bin, möchte ich dabei ein paar Ungereimtheiten ansprechen …

Drei Perspektiven

Bevor ich mir – ähnlich wie manche Kommentierenden des Zeit-Artikels – anhören muss, ich würde ja auch sonst viele Daten durch Nutzung meines Smartphones an Google und Co weitergeben, stelle ich mal meine Sicht auf die Dinge dar:

Die Corona-Perspektive: Ich reise aufs beruflichen Gründen generell viel mit der Bahn. So war ich etwa die ganze letzte Woche unterwegs; quer durch Deutschland. In durchaus gut gefüllten Zügen, die die DB selbst als zu mehr als 50% ausgelastet angezeigt hat. Wegen meines Alters zähle ich leider bereits zu dem Personenkreis mit erhöhtem Corona-Risiko. Die Corona-App bewahrt mich natürlich nicht vor diesen Risiken, aber sie wäre doch ein wichtiger Puzzlestein im Bemühen, rechtzeitig einen Arzt zu kontaktieren, wenn eine bestimmtes Infektionsrisiko aus den erfassten Daten mit einer hohen Eintrittswahrscheinlichkeit bewertet wird. Die App böte auch die Chance, meine ältere
Lebensgefährtin, die aus verschiedenen Gründen zur Hochrisikogruppe gehört, zu warnen und geeignete Maßnahmen zu treffen. Deshalb habe ich mir am Dienstag vergangener Woche die deutsche Corona-App auf mein Smartphone installiert.

Die Smartphone-Perspektive: Mein privates Android-Smartphone ist relativ alt und mit Android 6 versehen. Es hat einen guten HW-Unterbau, viel Speicher, eine schnelle verschlüsselte Zusatzkarte und ist performanter als manches Smartphone von anderen Familienangehörigen mit Android 8. Ich benutze mein Smartphone primär zum Lesen in Zeitungen (per FF-Browser bei aktivierter VPN-Lösung). VLC ermöglicht mir dabei, Musik zu hören. Die Kommunikation mit anderen Mitbürgern erfolgt ausschließlich über Signal. Standortbestimmung, GPS, Bluetooth sind normalerweise abgeschaltet; der Zugang ins Internet über variierende VPN-Server ist regelmäßig aktiviert. Entfernbare Apps sind entfernt. Die Zugriffsberechtigungen von Apps sind auf ein Minimum reduziert (soweit möglich). Ich nutze weder Google Chrome noch Googles Suchmaschine am Handy für Standardsuchen im Browser; Noscript blockt Javascript, bis ich es zulasse. Ich nutze Facebook und Whatsapp (nach eingehendem Studium über 4 Monate hinweg) natürlich nicht, Youtube ist für mich auf dem Smartphone uninteressantes Terrain und deaktiviert.
Ich meine deshalb das, was man ohne Rooten des Smartphones tun kann, getan zu haben, um für Samsung und Google zumindest nicht unmittelbar überwachbar zu sein. Ja, ich höre schon: das ist eine Illusion. Stimmt, aber ich reduziere halt den direkten Zugriff, so gut es geht. Meine Frau hält mein Handy deshalb für nicht benutzbar – ein gutes Zeichen. Meine grundlegende Devise ist: Ich habe keine Freunde im Internet. Und Google ist mir bislang sicher nicht als Freund, sondern als kommerzielles Unternehmen mit kapitalistisch unterfüttertem Interesse an Daten zu meiner Person begegnet.

Die Perspektive auf unseren Staat in der aktuellen Krise: Ich hege kein fundamentales Misstrauen gegenüber unserem Staat und seinen Institutionen; eher gegenüber einzelnen Politikern und bestimmten Parteien. Ich bin sehr froh, in Deutschland zu leben und schätze unser Grundgesetz mit jedem Lebensalter mehr. Ich fand die Maßnahmen unserer Bundesregierung und der meisten Landesregierungen in Sachen Corona-Pandemie richtig. (Es schadet bei der Beurteilung dieser Maßnahmen übrigens nicht, selbst mal ein wenig auf Basis der bekannten Zahlen zu rechnen. Aber die Fähigkeit zum “Rechnen”, geschweige denn das Beherrschen von ein wenig Statistik, scheint ja bei vielen Mitbürgern, Moderatoren und Politikern nicht mehr zur Grundausstattung zu gehören … außer bei Fr. Dr. Merkel). Die Anstrengungen der Bundesregierung, eine App auf die Beine zu stellen, fand ich begrüßenswert. Den leider erst spät erfolgten Schritt in Richtung dezentrale Datenhaltung und Open Source auch.

Standortermittlung – ?

Es war klar, dass die Corona-App Bluetooth (Low Energy) nutzen würde. Die generelle Umsetzung von Bluetooth auf Smartphones durch die jeweiligen Hersteller ist bislang leider immer wieder durch viele Sicherheitslücken aufgefallen und weniger als solide Kommunikationstechnologie. Normalerweise aktiviert kein vernünftiger Mensch Bluetooth in einem dicht gefüllten Zugabteil. Aber in Coronazeiten muss man hier leider Kompromisse schließen. Das war – wie gesagt – von Anfang an klar und ist aus meiner Sicht auch hinreichend dargestellt und debattiert worden.

Dass man auf Android Smartphones aber pauschal die Standortbestimmung (inkl. GPS) aktivieren muss, das war nicht ausgemacht. In der öffentlichen Darstellung wurde im Vorfeld auch eher das Gegenteil betont. So traute ich meinen Augen nicht, als ich im Zuge der Installation und Aktivierung der Corona-App auf Seite 2 einer Popup-Meldung lesen musste:

“Für diese Funktionen [es ging um Bluetooth] sind die folgenden Berechtigungen erforderlich:
nGerätestandort
Diese Einstellung ist erforderlich, damit Bluetooth-Geräte in deiner Nähe gefunden werden können. Für Benachrichtigungen zu möglicher Begegnung mit Covid-19-Infizierten wird der Gerätestandort jedoch nicht genutzt. Andere Apps mit der Berechtigung zur Standortermittlung haben Zugriff auf den Gerätestandort.”

Ich hätte das beinahe überblättert. Zwar fiel mir das Wort “Gerätestandort” auf; aber aufgrund der Meldungen im Fernsehen und der Presse hatte ich die Erwartungshaltung, dass ich diesbzgl. lediglich darüber informiert würde, dass die Standortbestimmung nicht notwendig sei. Erst als ich erneut “Aktivieren” drücken sollte, wurde ich unschlüssig – die App oder was anderes? Und begann zu lesen …

Davon habe ich dann einem IT-Kollegen am Mittwoch vergangener Woche im Zug erzählt. Er hatte die App selbst installiert, hatte dabei allerdings über die zwei Erläuterungs-Popup-Seiten am Anfang zügig hinweggeklickt. Er wollte meine Darstellung erst nicht glauben. Bis ich ihm dann eine damals noch englische (!) Warnmeldung zeigte, die hochkam, wenn man die durch die Corona-App aktivierte Standortbestimmung (inkl. GPS) explizit wieder deaktivierte. Inzwischen (App-Version 1.0.4) ist die Meldung auf Deutsch da, aber zumindest auf meinem Gerät im Meldungsbereich nicht direkt in Gänze zu lesen. Da muss man dann auf dem Smartphone schon unter den eigenen “Google”-Einstellungen nachsehen:

“Benachrichtigungen zu möglicher Begegnung mit Covid-19-Infizierten. Diese Funktion ist deaktiviert”.

Tja, das ist wohl eindeutig. Keine Aktivierung der Standortbestimmung => dann auch keine Aktivierung der Benachrichtigung durch die Corona-App. War beim Kollegen dann auch so (trotz eines viel aktuelleren Handys). Richtig geglaubt hat der Kollege es aber erst, als ich ihm die Beschreibung der technischen Voraussetzungen unter den “Datenschutzinformationen” der App zeigte.
Interessanterweise findet man ja zur Standortbestimmung nichts unter “Häufige Fragen”, sondern eben unter “Datenschutzinformation” und dort weit unten unter Punkt 7 “Technische Voraussetzungen” – Punkt b) Android-Smartphones:

“Die Standortermittlung Ihres Smartphones muss aktiviert sein, damit ihr Gerät nach Bluetooth-Signalen anderer Smartphones sucht. Standortdaten werden dabei aber nicht erhoben. “

Eine fast gleichlautende Info erhält man übrigens auch, wenn man die Corona-App über deren eigenen Schalter deaktiviert und dann wieder reaktiviert.

Wirkt sich die Aktivierung der Standortbestimmung auf andere Apps aus? Oh ja! Am einfachsten kann man das über Google Maps oder Kompass-Anwendungen testen. Die fragen dann nicht mehr nach, ob z.B. GPS aktiviert werden soll; man hat das im Zuge der Aktivierung der Corona-App tatsächlich pauschal freigegeben (falls man anderweitig nicht andere Einschränkungen getroffen hat.) Was nicht heißt, das die Corona-App selbst Standortdaten verwendet. Dazu unten mehr.

Soviel erst mal zu den Fakten und zu dem, was einem die Corona-App selbst mitteilt.

Was zeigen Recherchen im Internet?

Man kann nun ein wenig im Internet recherchieren. Dabei wird man feststellen, dass man sich nicht getäuscht hat. Auf Android-Geräten ist zunächst die pauschale Standortfreigabe notwendig. Auch wenn die Corona-App selbst keine Standortdaten nutzt … Und was lernt man auf die Schnelle sonst noch?

Punkt 1: Google koppelt die “Standortbestimmung” (inkl. der generellen Freigabe von GPS für andere Apps) und die Nutzung von “Bluetooth Low Energy” faktisch und technisch aneinander. Das war und ist so beabsichtigt! Mindestens mal in Android 6.x und 7.X. Siehe:

stackoverflow.com/ questions/ 33045581/ location-needs-to-be-
enabled-for-bluetooth-low-energy-scanning-on-android-6-0
https://www.mobiflip.de/shortnews/corona-warn-app-geraetestandort-unter-android/

Man kann Bluetooth zwar separat aktivieren (inkl. Low Energy-Funktionalität [LE]), aber es nutzt dir nichts, wenn du nicht gleichzeitig eine Standortbestimmung zulässt. Grund: Bluetooth LE würde sonst angeblich gar nicht anfangen, nach anderen Bluetooth-Geräten zu suchen. Leider wird dir als Corona-App-Nutzer in der der Statuszeile des Android-Bildschirms nicht angezeigt, dass die Standortbestimmung aktiviert wurde. Das musst du dann schon selbst entdecken …

Interessant ist in diesem Zusammenhang die Erkenntnis, dass die zwei technisch zunächst getrennten Funktionalitäten der Bluetooth-LE-Aktivierung und der faktischen Freigabe einer Standortbestimmung in Apples iOS nicht miteinander verbunden worden. Der Bluetooth-Chip ist auch auf Android-Handys eine eigene Einheit und hat z.B. mit einer groben weiteren Standortbestimmung per Wifi oder GPS oder durch andere verortete Bluetooth-Empfänger-Geräte technisch nicht direkt etwas zu tun. Dennoch hat Google die Kopplung auf Betriebssystemebene (!) herbeigeführt. Ein Schelm, wer dabei Böses denkt ….

Punkt 2: Die fast verschämte Antwort, die dazu an verschiedenen Stellen im Internet kolportiert wird, lautet, dass man das deshalb so handhabe, weil ja auch über Bluetooth eine Standortbestimmung vorgenommen werden könne. Ich zitiere aus https://developer.android.com/guide/topics/connectivity/bluetooth-le:

“Because discoverable devices might reveal information about the user’s location, the device discovery process requires location access. If your app is being used on a device that runs Android 8.0 (API level 26) or higher, use the Companion Device Manager API. This API performs device discovery on your app’s behalf, so your app doesn’t need to request location permissions.”

Das ist schon eine seltsame Logik. Weil die Aktivierung eines technischen Senders indirekt und in Kombination mit anderen technischen Funktionalitäten oder anderen Geräten eine Standortbestimmung ermöglichen könnte, muss man die Standortbestimmung auf dem eigenen Gerät pauschal für alle anderen Apps freigeben? Aber ab Android 8 dann doch nicht mehr? Verarschen kann man sich alleine wirklich besser ….

Siehe hierzu auch:
https://android.stackexchange.com/ questions/ 160479/ why-do-i-need-to-turn-on-location-services-to-pair-with-a-bluetooth-device
https://www.t-online.de/ digital/ id_88069644/ corona-warn-app-android-standort-freigeben-so-verhindern-sie-google-tracking.html
https://www.smartdroid.de/ corona-warn-app-standortermittlung-kommt-von-google-ein-erklaerungsversuch/

Tja, Google, warum wurde ich eigentlich bisher beim Nutzen von Bluetooth LE darauf nicht aufmerksam gemacht? Wie man es auch dreht und wendet: Es gibt keine logische Begründung dafür, zwingend eine Standortbestimmung auch für andere Apps (inkl. GPS-Nutzung) freizugeben, weil Bluetooth ggf. eine Standortbestimmung möglich macht.

Punkt 3: Dennoch bleibt festzuhalten: Ja, es stimmt, man (ein anderer Bluetoothnutzer bzw. eine anderes Bluetooth-Gerät) kann über Bluetooth und z.B. Wifi indirekt eine Standortbestimmung deines Handys vornehmen. U.a. über die Beacon-Funktionalität. Auch Google konnte (bei aktiviertem Bluetooth auf deinem Geräte) wohl immer schon
deinen Standort über Dritte ermitteln, die sich in deiner Nähe mit aktiviertem Bluetooth, GPS oder WiFi befanden.

Punkt 4: Es bleibt die Frage, warum Google zwei, eigentlich völlig unabhängige Funktionen (nämlich Bluetooth LE und eine Standortbestimmung) seit Android 6 technisch so eng aneinander gekoppelt hat. Es wäre ja auch anders gegangen. Du aktivierst Bluetooth und erhältst eine Warnung, dass damit auch eine Standortbestimmungen möglich sind. Und eine Beschreibung der möglichen Mechanismen. So wurde das aber nicht aufgezogen … Zieht man allerdings für die Kopplung auch andere als nur technische Gründe in Betracht, so macht das Ganze plötzlich sehr viel Sinn; allerdings keinen technischen sondern einen ökonomischen. Zufall?

Punkt 5: Bluetooth und Bluetooth LE … Habe ich als Anwender eigentlich Kontrolle darüber, welche App was genau einsetzt? Und ob die App bei Bedarf nicht auch die Standortfreigabe auf irgendeine Rückfrage im Hintergrund aktiviert? Schalte ich (normales) Bluetooth an, ohne Standortfreigabe und GPS explizit zu aktivieren, sehe ich im Moment gerade 10 Geräte in unserem Wohnhaus mit aktiviertem Bluetooth. Es gibt für den Android-Laien somit vier Möglichkeiten, die Situation zu beurteilen:
(a) Entweder ist die Aktivierung der Standortbestimmung für Bluetooth und das Erkennen anderer Geräte technisch gar nicht erforderlich (entgegen Google’s eigener Verlautbarung für Bluetooth LE …). Zu dieser Variante passt: Es gibt unter den Bluetooth-Einstellungen die Möglichkeit, sich selbst aktiv sichtbar zu machen.
(b) Oder die Standortfreigabe wird im Hintergrund aktiviert – ohne dass ich als Anwender informiert werde.
(c) Eine dritte Variante ist, dass die Standortfreigabe für das aktive Suchen nach anderen Bluetooth-Geräten nur im Falle von “Bluetooth Low Energy” [BLE], wie für die Corona-App eingesetzt, verwendet wird – und in diesem Fall aber technisch unumgänglich ist.
(d) Über eine deutlich schlimmere Variante mag ich nicht weiter nachdenken.

Lieber Leser: Können wir uns darauf einigen, dass das ohne tiefere technische Einblicke in Android mindestens mal eine äußerst dubiose Sachlage ist? Die nun aber Google indirekt sehr zugute kommt?

Punkt 6: Nun nutzt ja die Corona-App selbst die im Prinzip ermittelbaren Standortdaten nicht. Da glaube ich mal jenen, die den Code der App bereits im Detail durchforstet haben (hoffentlich). Aber was macht Google ggf. auf anderen Wegen mit der aktivierten “groben” Standortbestimmung? Aus Diskussionen in Zeitungen (z.B. der SZ, der Zeit oder der Faz) entnimmt man nicht mehr als die Versicherung von Google, dass es die Daten im Kontext der Corona-App nicht erfasse. Soll ich das glauben? Hmm, es ist immerhin strafbewehrt ….

Punkt 7: Aber halt: Da fand sich in den Hinweisen der Corona-App selbst ja noch der kleine Satz am Schluss: “Andere Apps mit der Berechtigung zur Standortermittlung haben Zugriff auf den Gerätestandort.” Offenbar dürfen die anderen Apps die Standortdaten dann wohl auch nutzen. Der User hat ja schließlich die Standortbestimmung explizit freigegeben. Um sich gegen Corona zu schützen …

Aha! Wer dürfte sich da freuen? Google! 10 Millionen Downloads => 10 Millionen pauschal eine für Apps pauschal aktivierte Standortbestimmung frei Haus – durch Aktivierung der Corona-App! Nein, die Standortermittlung erfolgt nicht durch die Corona-App selbst – aber eben (potentiell) durch andere Apps, die über die Aktivierung der Corona App nun die Nutzung der Standortbestimmung freigeschaltet bekommen haben. Das ist so genial, dass man fast neidisch wird.

Punkt 8: Nun werden einige Schlaumeier sagen, man könne ja die Rechte der (vielen) anderen Apps über den “Anwendungsmanager” einschränken. Stimmt. Kann man. Darauf weist die Corona-App aber leider gar nicht hin. Und, hast du, Schlaumeier, das auch schon
mal gemacht? Jede Anwendung im Anwendungsmanager aufgerufen und die Rechte beschränkt? Das ist übrigens eine sehr lehrreiche Übung, die ich jedem Android-Nutzer dringend ans Herz lege … Bei der Corona-App selbst kannst du auf dem Weg übrigens nur die Berechtigung zur Nutzung der Kamera (wird bei Übermittlung eines Testergebnisses verwendet) abschalten …

Punkt 10: Ein wenig Nachforschen zeigt: Ein paar Einstellungen zur Corona-App können auch im Kontext der Google-Einstellungen gesetzt werden (man öffne die Android-Anwendung “Einstellungen” und dort “Google”!) . Da erfährt man übrigens auch definitiv, dass die Benachrichtigung bzgl. des Kontakts mit Infizierten deaktiviert wurde, wenn man die Standortfreigabe zwischenzeitlich – aus welchen Gründen auch immer – abgeschaltet hat. Und da überfällt mich dann wieder ein größeres Stirnrunzeln. Die Corona-App selbst zeigt dir das nämlich nicht an; die werkelt einfach weiter. Ein Hinweis darauf, dass die Entwickler davon ausgegangen sind, dass die Freigabe der Standortbestimmung nicht mehr deaktiviert wird?

Die App verhält sich bei explizit abgeschalteter Standortbestimmung seltsam

Ich habe die Standortbestimmung am Mittwochnachmittag letzter Woche abgeschaltet. Damals kam dann noch ein englischer Hinweis auf eine Deaktivierung der Benachrichtigung zum Infektionsrisiko. Die App selber meldete mir über ihren misslichen Zustand erst mal gar nichts. Sie lief mindestens drei Tage weiter, zeigte mir jeden Tag an dass sie aktiv sei (entgegen der Feststellung unter meinen Google-Einstellungen) und dass mein Risiko nach wie vor gering sei. Heute – am Montagvormittag – zeigte sie mir dann an, dass kein Risiko bestimmt werden könne, weil die App seit 3 Tage nicht “updaten” konnte. ????
Test: Standortbestimmung wieder freigeben. Corona-Benachrichtigung wieder in der App aktiviert. App läuft => geringes Risiko. Standortermittlung explizit deaktiviert => Corona-App läuft weiter, als sei nichts geschehen => “Geringes Risiko. 6 von 14 Tagen aktiv!”. Aber Meldung unter den Google-Einstellungen: Diese Funktion (gemeint ist die Risikobenachrichtigung) ist deaktiviert!

Ich würde das mindestens mal als Inkonsistenz bezeichnen … eigentlich ist es ein massiver Bug.

Auf Nachfrage bei anderen Nutzern der Corona-App hat sich übrigens herausgestellt, dass mancher der Befragten die Standortbestimmung tatsächlich einfach wieder deaktiviert hatte: Sobald sie nämlich zufällig entdeckten, dass sie aktiv war. Zwei dachten, sie hätten wohl vergessen, die Standortbestimmung nach einer Benutzung von Google Maps wieder abzuschalten. Die Meldung, die dann nicht vollständig lesbar war, hatten sie ignoriert. Sie konnten sie auch nicht unmittelbar der Corona-App zuordnen. Die “lief” ja noch und zeigte ein geringes Risiko an …. Oh mei …

Bewertung

Ich finde, das Ganze ist eine schwierige, bis üble Gemengelage. Das liegt zunächst nicht an der Corona-App selbst und schon gar nicht an unserer Regierung. Es liegt einzig und allein an Google – und deren Kopplung des Einsatzes von Bluetooth LE mit der Freigabe der Standortbestimmung – für alle anderen Apps. Diese Kopplung spielt Google nicht erst jetzt eindeutig ökonomisch in die Hände bzw. unterstützt Googles Geschäftsmodell massiv.

Ja, man kann zwar nach der Installation der Corona-App alle anderen Apps bzgl. der Rechte konfigurieren. Aber ehrlich: Wer macht das schon? Der normale Anwender weiß nicht mal, wo und wie er Zugriffsrechte verwalten kann. Die Corona-App macht einen darauf leider auch nicht aufmerksam.

Hinzu kommt auch, dass die Corona-App nichts anzeigt oder meldet, wenn die angeblich zwingend benötigte Freigabe der Standortbestimmung im Nachhinein abgestellt wird. Das ist entweder fahrlässig – oder aber Freigabe der Standortbestimmung wird entgegen aller Beteuerungen doch nicht benötigt. Keine der beiden Alternativen ist gut.

Was hätte ich erwartet?

Zunächst mal klare
und eindeutige Informationen über die notwendige Aktivierung der Standortbestimmung im Vorfeld der Veröffentlichung der App – durch die Regierung und auch durch Google. Später eine klare Information durch die App selbst. Plus Informationen für Interessierte, warum zwei technisch unterschiedliche Funktionen ab Android 6 überhaupt aneinander gekoppelt wurden.
Ich hätte erwartet, dass mir die pauschale Aktivierung der Standortbestimmung in der oberen Statusleiste von Android durch ein eindeutiges Symbol im Zuge der Aktivierung der Corona-App angezeigt wird. Am besten in einer Signalfarbe.
Ich hätte detaillierte Informationen dazu erwartet, dass man nach der Installation und Aktivierung der Corona-App andere Apps bzgl. deren Zugriffsrechte nachjustieren muss – und für den Normalverbraucher auch Infos dazu, wie man das macht.
Ich hätte grundsätzlich erwartet, dass Google eine technische Lösung anbietet, die eine Umgehung einer pauschalen Freigabe der Standortbestimmung ermöglicht. Das war schon lange überfällig. Warum hat man da seitens des Auftraggebers (= Regierung) nicht mehr Druck ausgeübt?
Ich hätte zumindest erwartet, dass mir die Corona-App anzeigt, dass sie nicht mehr richtig funktioniert, wenn ich die Standortbestimmung aus irgendwelchen Gründen abschalte.
Ich hätte mindestens mal vom CCC (Chaos Computer Club) eine Bewertung der pauschalen Freigabe der Standortbestimmung für andere Apps unter Android erwartet. Statt dessen: https://www.zdf.de/nachrichten/politik/corona-app-launch-100.html. Sicher ist Vieles vorbildlich gewesen … aber Open Source allein reicht nicht als Gütesiegel …

Fazit: Ich bin mit der App nicht zufrieden. Es bleibt das schale Gefühl, das Google uns alle mal wieder über den Tisch gezogen hat, auch wenn die Corona-App selbst sinnvoll ist. Meine Lehre aus dem Ganzen ist: Es wäre endlich an der Zeit, eine Alternative zu Android auf europäischer Ebene zu entwickeln. Aber das werde ich wohl nicht mehr erleben …

Habe ich die Corona-App jetzt abgeschaltet? Nein. Aber ich habe die Rechte anderer Apps weiter reduziert – mit interessanten Folgen. Jetzt hat mir Google eine freundliche Mail geschickt, ich möge nun doch bitte meine Google-Account-Einstellungen zur Nutzung personenbezogener Werbung vervollständigen … Ein Schelm, wer Böses denkt ….

A simple CNN for the MNIST dataset – III – inclusion of a learning-rate scheduler, momentum and a L2-regularizer

Posted on 13. June 2020 by eremo

In the last article of this series

A simple CNN for the MNIST datasets – II – building the CNN with Keras and a first test
A simple CNN for the MNIST datasets – I – CNN basics

we saw that we could easily create a Convolutional Neural Network [CNN] with Keras and apply it to the MNIST dataset. But this was only a quick win. Readers who followed me through the MLP-oriented article series may have noticed that we are far, yet, from having the full control over a variety of things we would use to optimize the classification algorithm. After our experience with MLPs you may rightfully argue that

both a systematic reduction of the learning rate in the vicinity of a (hopefully) global minimum of the multidimensional loss function hyperplane in the {weights, costs}-space
and the use of a L2- or L1-regularizer

could help to squeeze out the last percentage of accuracy of a CNN. Actually, in our first test run we did not even care about how the RMSProp algorithm invoked a learning rate and what the value(s) may have been. A learning rate “lr” was no parameter of our CNN setup so far. The same is true for the use and parameterization of a regularizer.

I find it interesting that some of the introductory books on “Deep Learning” with Keras do not discuss the handling of learning rates beyond providing a fixed (!) rate parameter on rare occasions to a SGD- or RMSProp-optimizer. Folks out there seem to rely more on the type and basic settings of an optimizer than on a variable learning rate. Within the list of books I appended to the last article, only the book of A. Geron gives you some really useful hints. So, we better find information within the online documentation of the Keras framework to get control over learning rates and regularizers.

Against my original plans, the implementation of schedulers for the learning rate and the usage of a L2-regularizer will be the main topic of this article. But I shall – as a side-step – also cover the use of “callbacks” for some interactive plotting during a training run. This is a bit more interesting than just plotting key data (loss, accuracy) after training – as we did for MLPs.

Learning rate scheduler and momentum

A systematic reduction of the learning rate or the momentum can be handled with Keras via a parameterization of a scheduler. Such a scheduler can e.g. be invoked by a chosen optimizer. As the optimizer came into our CNN with the model.compile()-function we would not be astonished, if we had to include the scheduler there, too. The Keras version of Tensorflow 2 offers a variety of (partially) configurable schedulers. And there are, in addition, multiple different ways how to integrate a systematic decline of the learning rate into a Keras CNN model. I choose a very simple method for this article, which is, however, specific for the TF2 variant of Keras (tensorflow.keras). The book of Geron discusses other alternatives (e.g. the invocation of a scheduler via a so called callback for batches; see below).

The optimizer for the gradient calculation directly accepts parameters for an (initial and constant) learning rate (parameter “learning_rate”) and a momentum (parameter “momentum”). An example with the SGD-optimizer would be:

optimizer =  keras.optimizers.SGD(learning_rate=0.001, momentum=0.5)
cnn.compile(optimizer=optimizer, ....

If you just provide a parameter “learning_rate=0.001” then the optimizer will use a constant learning rate. However, if we provide a scheduler object – an instance of a scheduler class – then we can control a certain decline of the learning rate. What scheduler classes are available? You find them here:
keras.io/api/optimizers/ learning rate schedules/

A simple scheduler which allows for the “power scheduling” which we implemented for our MLPs is the scheduler “InverseTimeDecay“. It can be found under the library branch optimizers.schedules. We take this scheduler as an example in this article.

But how do we deliver the scheduler object to the optimizer? With tensorflow.keras and TF > 2.0 this is simply done by providing it as the parameter (object) for the learning_rate to the optimizer, as e.g. i the following code example:

from tensorflow.keras import optimizers
from tensorflow.keras.optimizers import schedules
...
...
scheduled_lr_rate = schedules.InverseTimeDecay(lr_init, lr_decay_steps, lr_decay_rate) 
optimizer=optimizers.RMSprop(learning_rate=scheduled_lr_rate, momentum=my_momentum)
...

Here “lr_init” defines an initial learning rate (as 0.001), “lr_decay_steps” a number of steps, after which the rate is changed, and lr_decay_rate a decay rate to be applied in the formula of the scheduler. Note that different schedulers take different parameters – so look them up before applying them. Also check that your optimizer accepts a scheduler object as a parameter …

The scheduler instance works as if we had a function returning an output – a learning rate – for an input, which corresponds to a number of “steps“:

learning_rate=scheduler(steps) .

Now what do we mean by “steps”? Epochs? No, actually a step here corresponds practically to the result of an iterator over batches. Each optimization “step” in the sense of a weight adjustment after a gradient analysis for a mini-batch defines a step for the scheduler. If you want to change the learning rate on the level of epochs only, then you must either adjust the “decay_step”-parameter or write a so called “callback-function” invoked after each epoch and control the learning rate yourself.

Momentum
Note, by the way, that in addition to the scheduler we also provided a value for the “momentum” parameter an optimizer may use during the gradient evolution via some specific adaptive formula. How the real weight changes are calculated based on gradient development and momentum parameters is of course specific for an optimizer.

Required import statements

We use the Jupyter cells of the code we built in the last article as far as possible. We need to perform some more imports to work with schedulers, regularizers and plotting. You can exchange the contents of the first Jupyter cell with the following statements:

Jupyter Cell 1:

  
import numpy as np
import scipy
import time 
import sys 

from sklearn.preprocessing import StandardScaler
import tensorflow as tf
from tensorflow import keras as K
from tensorflow.python.keras import backend as B 
from tensorflow.keras import models
from tensorflow.keras import layers
from tensorflow.keras import regularizers
from tensorflow.keras import optimizers
from tensorflow.keras.optimizers import schedules
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.datasets import mnist

from tensorflow.python.client import device_lib

import matplotlib as mpl
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
import matplotlib.patches as mpat 

import os

Then we have two unchanged cells:

Jupyter Cell 2:

#gpu = False 
gpu = True
if gpu: 
#    GPU = True;  CPU = False; num_GPU = 1; num_CPU = 1
    GPU = True;  CPU = False; num_GPU = 1; num_CPU = 1
else: 
    GPU = False; CPU = True;  num_CPU = 1; num_GPU = 0

config = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=6,
                        inter_op_parallelism_threads=1, 
                        allow_soft_placement=True,
                        device_count = {'CPU' : num_CPU,
                                        'GPU' : num_GPU}, 
                        log_device_placement=True

                       )
config.gpu_options.per_process_gpu_memory_fraction=0.35
config.gpu_options.force_gpu_compatible = True
B.set_session(tf.compat.v1.Session(config=config))

and

Jupyter Cell 3:

# load MNIST 
# **********
#@tf.function
def load_Mnist():
    mnist = K.datasets.mnist
    (X_train, y_train), (X_test, y_test) = mnist.load_data()

    #print(X_train.shape)
    #print(X_test.shape)

    # preprocess - flatten 
    len_train =  X_train.shape[0]
    len_test  =  X_test.shape[0]
    X_train = X_train.reshape(len_train, 28*28) 
    X_test  = X_test.reshape(len_test, 28*28) 

    #concatenate
    _X = np.concatenate((X_train, X_test), axis=0)
    _y = np.concatenate((y_train, y_test), axis=0)

    _dim_X = _X.shape[0]

    # 32-bit
    _X = _X.astype(np.float32)
    _y = _y.astype(np.int32)

    # normalize  
    scaler = StandardScaler()
    _X = scaler.fit_transform(_X)

    # mixing the training indices - MUST happen BEFORE encoding
    shuffled_index = np.random.permutation(_dim_X)
    _X, _y = _X[shuffled_index], _y[shuffled_index]

    # split again 
    num_test  = 10000
    num_train = _dim_X - num_test
    X_train, X_test, y_train, y_test = _X[:num_train], _X[num_train:], _y[:num_train], _y[num_train:]

    # reshape to Keras tensor requirements 
    train_imgs = X_train.reshape((num_train, 28, 28, 1))
    test_imgs  = X_test.reshape((num_test, 28, 28, 1))
    #print(train_imgs.shape)
    #print(test_imgs.shape)

    # one-hot-encoding
    train_labels = to_categorical(y_train)
    test_labels  = to_categorical(y_test)
    #print(test_labels[4])
    
    return train_imgs, test_imgs, train_labels, test_labels

if gpu:
    with tf.device("/GPU:0"):
        train_imgs, test_imgs, train_labels, test_labels= load_Mnist()
else:
    with tf.device("/CPU:0"):
        train_imgs, test_imgs, train_labels, test_labels= load_Mnist()

Include the regularizer via the layer definitions of the CNN

A “regularizer” modifies the loss function via a sum over contributions delivered by a (common) function of the weight at every node. In Keras this sum is split up into contributions at the different layers, which we define via the “model.add()”-function. You can build layers with or without regularization contributions. Already for our present simple CNN case this is very useful:

In a first trial we only want to add weights to the hidden and the output layers of the concluding MLP-part of our CNN. To do this we have to provide a parameter “kernel_regularizer” to “model.add()”, which specifies the type of regularizer to use. We restrict the regularizers in our example to a “l2”- and a “l1”-regularizer, for which Keras provides predefined functions/objects. This leads us to the following change of the function “build_cnn_simple()”, which we used in the last article:

Jupyter Cell 4:

# Sequential layer 
model of our CNN
# ***********************************

# just for illustration - the real parameters are fed later 
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
li_conv_1   = [32, (3,3), 0] 
li_conv_2   = [64, (3,3), 0] 
li_conv_3   = [64, (3,3), 0] 
li_Conv     = [li_conv_1, li_conv_2, li_conv_3]
li_pool_1   = [(2,2)]
li_pool_2   = [(2,2)]
li_Pool     = [li_pool_1, li_pool_2]
li_dense_1  = [70, 0]
li_dense_2  = [10, 0]
li_MLP      = [li_dense_1, li_dense_2]
input_shape = (28,28,1)

# important !!
# ~~~~~~~~~~~
cnn = None
x_optimizers = None 

def build_cnn_simple(li_Conv, li_Pool, li_MLP, input_shape, 
                     my_regularizer=None, 
                     my_reg_param_l2=0.01, my_reg_param_l1=0.01 ):

    use_regularizer = True
    if my_regularizer == None:
        use_regularizer = False  
    
    # activation functions to be used in Conv-layers 
    li_conv_act_funcs = ['relu', 'sigmoid', 'elu', 'tanh']
    # activation functions to be used in MLP hidden layers  
    li_mlp_h_act_funcs = ['relu', 'sigmoid', 'tanh']
    # activation functions to be used in MLP output layers  
    li_mlp_o_act_funcs = ['softmax', 'sigmoid']

    # dictionary for regularizer functions
    d_reg = {
        'l2': regularizers.l2,  
        'l1': regularizers.l1
    }
    if use_regularizer: 
        if my_regularizer not in d_reg:
            print("regularizer " + my_regularizer + " not known!")
            sys.exit()
        else: 
            regul = d_reg[my_regularizer] 
        if my_regularizer == 'l2':
            reg_param = my_reg_param_l2
        elif my_regularizer == 'l1':
            reg_param = my_reg_param_l1
    
    
    # Build the Conv part of the CNN
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    num_conv_layers = len(li_Conv)
    num_pool_layers = len(li_Pool)
    if num_pool_layers != num_conv_layers - 1: 
        print("\nNumber of pool layers does not fit to number of Conv-layers")
        sys.exit()
    rg_il = range(num_conv_layers)

    # Define a sequential model
    # ~~~~~~~~~~~~~~~~~~~~~~~~~
    cnn = models.Sequential()

    for il in rg_il:
        # add the convolutional layer 
        num_filters = li_Conv[il][0]
        t_fkern_size = li_Conv[il][1]
        cact        = li_conv_act_funcs[li_Conv[il][2]]
        if il==0:
            cnn.add(layers.Conv2D(num_filters, t_fkern_size, activation=cact, input_shape=input_shape))
        else:
            cnn.add(layers.Conv2D(num_filters, t_fkern_size, activation=cact))
        
        # add the pooling layer 
        if il < num_pool_layers:
            t_pkern_size = li_Pool[il][0]
            cnn.add(layers.MaxPooling2D(t_pkern_size))
            

    # Build the MLP part of the CNN
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    num_mlp_layers = len(li_MLP)
    rg_im = range(num_mlp_layers)

    cnn.add(layers.Flatten())

    for im in rg_im:
        # add the dense layer 
        n_nodes = li_MLP[im][0]
        if im < num_mlp_layers - 1:  
            m_act   =  li_mlp_h_act_funcs[li_MLP[im][1]]
            if use_regularizer:
                cnn.add(layers.Dense(n_nodes, activation=m_act, kernel_regularizer=regul(reg_param)))
            else:
                cnn.add(layers.Dense(n_nodes, activation=m_act))
        else: 
            m_act   =  li_mlp_o_act_funcs[li_MLP[im][1]]
            if use_regularizer:
                cnn.add(layers.Dense(n_nodes, activation=m_act, kernel_regularizer=regul(reg_param)))
            else:
                cnn.add(layers.Dense(n_nodes, activation=m_act))
                
    return cnn

I have used a dictionary to enable an indirect call of the regularizer object. The invocation of a regularizer happens in
the statements:

cnn.add(layers.Dense(n_nodes, activation=m_act, kernel_regularizer=regul(reg_param)))

We only use a regularizer for MLP-layers in our example.

Note that I used predefined regularizer objects above. If you want to use a regularizer object defined and customized by yourself you can either define a simple regularizer function (which accepts the weights as a parameter) or define a child class of the “keras.regularizers.Regularizer”-class. You find hints how to do this at https://keras.io/api/layers/regularizers/ and in the book of Aurelien Geron (see the literature list at the end of the last article).

Although I have limited the use of a regularizer in the code above to the MLP layers you are of course free to apply a regularizer also to convolutional layers. (The book of A. Geron gives some hints how to avoid code redundancies in chapter 11).

Include a scheduler via a parameter to the optimizer of the CNN

As we saw above a scheduler for the learning rate can be provided as a parameter to the optimizer of a CNN. As the optimizer objects are invoked via the function “model.compile()”, we prepare our own function “my_compile()” with an appropriate interface, which then calls “model.compile()” with the necessary parameters. For indirect calls of predefined scheduler objects we again use a dictionary:

Jupyter Cell 5:

  
# compilation function for further parameters 
def my_compile(cnn, 
               my_loss='categorical_crossentropy', my_metrics=['accuracy'], 
               my_optimizer='rmsprop', my_momentum=0.0, 
               my_lr_sched='powerSched', 
               my_lr_init=0.01, my_lr_decay_steps=1, my_lr_decay_rate=0.00001  ):
    
    # dictionary for the indirct call of optimizers 
    d_optimizer = {
        'rmsprop': optimizers.RMSprop,  
        'nadam':   optimizers.Nadam,
        'adamax':  optimizers.Adamax
    }
    if my_optimizer not in d_optimizer:
        print("my_optimzer" + my_optimizer + " not known!")
        sys.exit()
    else: 
        optim = d_optimizer[my_optimizer]
        
    use_scheduler = True
    if my_lr_sched == None:
        use_scheduler = False
        print("\n No scheduler will be used")
    
    # dictionary for the indirct call of scheduler funcs 
    d_sched = {
        'powerSched' : schedules.InverseTimeDecay,  
        'exponential': schedules.ExponentialDecay
    }
    if use_scheduler:
        if my_lr_sched not in d_sched:
            print("lr scheduler " + my_lr_sched + " not known!")
            sys.exit()
        else: 
            sched = d_sched[my_lr_sched] 
    
    if use_scheduler:
        print("\n lr_init = ", my_lr_init, " decay_steps = ", my_lr_decay_steps, " decay_rate = ", my_lr_decay_rate)
        scheduled_lr_rate   = sched(my_lr_init, my_lr_decay_steps, my_lr_decay_rate)
        optimizer = optim(learning_rate=scheduled_lr_rate, momentum=my_momentum)  
    else:
        print("\n lr_init = ", my_lr_init)
        optimizer=optimizers.RMSprop(learning_rate=my_lr_init, momentum=my_momentum)
    
    cnn.compile(optimizer=optimizer, loss=my_loss, metrics=my_metrics)  
    
    return cnn

You see that, for the time being, I only offered the choice between the predefined schedulers “InverseTimeDecay” and “ExponentialDecay“. If no scheduler name is provided then only a constant learning rate is delivered to the chosen optimizer.

You find some basic information on optimizers and schedulers here:
https://keras.io/api/optimizers/
nhttps://keras.io/api/optimizers/ learning rate schedules/.

Note that you can build your own scheduler object via defining a child class of “keras.callbacks.LearningRateScheduler” and provide with a list of other callbacks to the “model.fit()“-function; see
https://keras.io/api/callbacks/ learning rate scheduler/.
or the book of Aurelien Geron for more information.

Two callbacks – one for printing the learning rate and an other for plotting during training

As I am interested in the changes of the learning rate with the steps during an epoch I want to print them after each epoch during training. In addition I want to plot key data produced during the training of a CNN model.

For both purposes we can use a convenient mechanism Keras offers – namely “callbacks”, which can either be invoked at the beginning/end of the treatment of a mini-batch or at the beginning/end of each epoch. Information on callbacks is provided here:
https://keras.io/api/callbacks/
https://keras.io/guides/ writing your own callbacks/

It is best to just look at examples to get the basic points; below we invoke two callback objects which provide data for us at the end of each epoch.

Jupyter Cell 6:

  
# Preparing some callbacks 
# **************************

# Callback class to print information on the iteration and the present learning rate 
class LrHistory(K.callbacks.Callback):
    def __init__(self, use_scheduler):
        self.use_scheduler = use_scheduler
        
    def on_train_begin(self, logs={}):
        self.lr = []

    def on_epoch_end(self, epoch, logs={}):
        if self.use_scheduler: 
            optimizer = self.model.optimizer
            iterations = optimizer.iterations
            present_lr = optimizer.lr(iterations)   
        else:
            present_lr = optimizer.lr
        self.lr.append(present_lr)
        #print("\npresent lr:", present_lr) 
        tf.print("\npresent lr: ", present_lr)
        tf.print("present iteration:",  iterations)


# Callback class to support interactive printing  
class EpochPlot(K.callbacks.Callback):
    def __init__(self, epochs, fig1, ax1_1, ax2_2):
    #def __init__(self, epochs):
        self.fig1  = fig1
        self.ax1_1 = ax1_1
        self.ax1_2 = ax1_2
        self.epochs = epochs
        rg_i = range(epochs)
        self.lin92 = []
        for i in rg_i:
            self.lin92.append(0.992)

    def on_train_begin(self, logs={}):
        self.train_loss = []
        self.train_acc  = []
        self.val_loss   = []
        self.val_acc    = []
        
    def on_epoch_end(self, epoch, logs={}):
        if epoch == 0:
            for key in logs.keys():
                print(key)
        self.train_loss.append(logs.get('loss'))
        self.train_acc.append(logs.get('accuracy'))
        self.val_loss.append(logs.get('val_loss'))
        self.val_acc.append(logs.get('val_accuracy'))
        if len(self.train_acc) > 0: 
            self.ax1_1.clear()
            self.ax1_1.set_xlim (0, self.epochs+1)
            self.ax1_1.set_ylim (0.97, 1.001)
            self.ax1_1.plot(self.train_acc, 'g' )
            self.ax1_1.plot(self.val_acc,   'r' )
            self.ax1_1.plot(self.lin92,     'b', linestyle='--' )
            
            self.ax1_2.clear()
            self.ax1_2.set_xlim (0, self.epochs+1)
            self.ax1_2.set_ylim (0.0, 0.2)
            self.ax1_2.plot(self.train_loss, 'g' )
     
       self.ax1_2.plot(self.val_loss,   'r' )
 
            self.fig1.canvas.draw()

From looking at the code we see that a callback can be defined as a child class of a base class “keras.callbacks.Callback” or of some specialized predefined callback classes listed under “https://keras.io/api/callbacks/“. They are all useful, but perhaps the classes “ModelCheckpoint”, “LearningRateScheduler” (see above) and “EarlyStopping” are of direct practical interest for people who want to move beyond the scope of this article. In the above example I, however, only used the base class “callbacks.Callback”.

Printing steps, iterations and the learning rate
Let us look at the first callback, which provides a printout of the learning rate. This is a bit intricate: Regarding the learning rate I have already mentioned that a scheduler changes it after each operation on a batch; such a step corresponds to a variable “iterations” which is counted up by the optimizer during training after the treatment of each mini-batch. [Readers from my series on MLPs remember that gradient descent can be based on a gradient evaluation over the samples of mini-batches – instead of using a gradient evaluation based on all samples (batch gradient descent or full gradient descent) or of each individual sample (stochastic gradient descent).]

As we defined a batch size = 64 for the fit()-method of our CNN in the last article the number of optimizer iterations (=steps) per epoch is quite big: 60000/64 => 938 (with a smaller last batch).

Normally, the constant initial learning rate of an optimizer could be retrieved in a callback by a property “lr” as in “self.model.optimizer.lr”. However, in our example this attribute now points to a method “schedule()” of the scheduler object. (See schedule() method of the LearningRateScheduler class). We must provide the number of “iterations” (= number of steps) as a parameter to this schedule()-function and take the returned result as the present lr-value.

Our own callback class “LrHistory(K.callbacks.Callback)” gets three methods which are called at different types of “events” (Javascript developers should feel at home 🙂 ):

At initialization (__init__()) we provide a parameter which defines whether we used a scheduler at all, or a constant lr.
At the beginning of the training (on_train_begin()) we just instantiate a list, which we later fill with retrieved lr-values; we could use this list for some evaluation e.g. at the end of the training.
At the end of each epoch (on_epoch-end()) we determine the present learning rate via calling the scheduler (behind the “lr” attribute) – if we used one – for the number of iterations passed.

This explains the following statements:

optimizer = self.model.optimizer
iterations = optimizer.iterations
present_lr = optimizer.lr(iterations)

Otherwise we just print the constant learning rate used by the optimizer (e.g. a default value).

Note that we use the “tf.print()” method to do the printing and not the Python “print()” function. This is basically done for convenience reasons: We thus avoid a special evaluation of the tensor and a manual transformation to a Numpy value. Do not forget: All data in Keras/TF are basically tensors and, therefore, the Python print() function would print extra information besides the expected numerical value!

Plotting at the end of an epoch

With our MLP we never used interactive plots in our Jupyter notebooks. We always plotted after training based on arrays or lists filled during
the training iterations with intermediately achieved values of accuracy and loss. But seeing the evolution of key metric data during training is a bit more rewarding; we get a feeling for what happens and can stop a training run with problematic hyperparameters before we waste too much valuable GPU or CPU time. So, how do we do change plots during training in our Jupyter environment?

The first point is that we need to prepare the environment of our Jupyter notebook; we do this before we start the training run. Below you find the initial statements of the respective Jupyter cell; the really important statement is the call of the ion-function: “plt.ion()”. For some more information see
https://linux-blog.anracom.com/2019/12/26/matplotlib-jupyter-and-updating-multiple-interactive-plots/
and links at the end of the article.

 
# Perform a training run 
# ********************
# Prepare the plotting 
# The really important commands for interactive (=intermediate) plot updating
%matplotlib notebook
plt.ion()

#sizing
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 8
fig_size[1] = 3

# One figure 
# -----------
fig1 = plt.figure(1)
#fig2 = plt.figure(2)

# first figure with two plot-areas with axes 
# --------------------------------------------
ax1_1 = fig1.add_subplot(121)
ax1_2 = fig1.add_subplot(122)
fig1.canvas.draw()

The interesting statements are the first two. The statements for setting up the plot figures should be familiar to my readers already. The statement “fig1.canvas.draw()” leads to the display of two basic coordinate systems at the beginning of the run.

The real plotting is, however, done with the help of our second callback “EpochPlot()” (see the code given above for Jupyter cell 6). We initialize our class with the number of epochs for which our training run shall be performed. We also provide the addresses of the plot figures and their two coordinate systems (ax1_1 and ax1_2). At the beginning of the training we provide empty lists for some key data which we later (after each epoch) fill with calculated values.

Which key or metric data can the training loop of Keras provide us with?

Keras fills a dictionary “logs” with metric data and loss values. What the metric data are depends on the loss function. At the following link you find a discussion of the metrics you can use with respect to different kinds of ML problems: machinelearningmastery.com custom-metrics-deep-learning-keras-python/.

For categorical_crossentropy and our metrics list [‘accuracy’] we get the following keys

loss, accuracy, val_loss, val_accuracy

as we also provided validation data. We print this information out at epoch == 0. (You will note however, that epoch 0 is elsewhere printed out as epoch 1. So, the telling is a bit different inside Keras and outside).

Then have a look at the parameterization of the method on_epoch_end(self, epoch, logs={}). This should not disturb you; logs is only empty at the beginning, later on a filled logs-dictionary is automatically provided.

As you see form the code we retrieve the data from the logs-dictionary at the end of each epoch. The rest are plain matplot-commands The “clear()”-statements empty the coordinate areas; the statement “self.fig1.canvas.draw()” triggers a redrawing.

The new training function

To do things properly we now need to extend our training function which invokes the other already defined functions as needed:

Jupyter Cell 7:

# Training 2 - with test data integrated 
# *****************************************
def train( cnn, build, train_imgs, train_labels, 
           li_Conv, li_Pool, li_MLP, input_shape, 
           reset=True, epochs=5, batch_size=64, 
           my_loss='categorical_crossentropy', my_metrics=['accuracy'], 
           my_regularizer=None, 
           my_reg_param_l2=0.01, my_reg_param_l1=0.01, 
           my_optimizer='rmsprop', my_momentum=0.0, 
           my_lr_sched=None, 
           my_lr_init=0.001, my_lr_decay_steps=1, my_lr_decay_rate=0.00001,
           fig1=None, ax1_1=None, ax1_2=None
):
    
    if build:
        # build cnn layers - now with regularizer - 200603 by ralph
        cnn = build_cnn_simple( li_Conv, li_Pool, li_MLP, input_shape, 
                                my_regularizer = my_regularizer, 
                                my_reg_param_l2 = my_reg_param_l2, my_reg_param_l1 = my_reg_param_l1)
        
        # compile - now with lr_scheduler - 200603 by ralph
        cnn = my_compile(cnn=cnn, 
                         my_loss=my_loss, my_metrics=my_metrics, 
                         my_optimizer=my_optimizer, my_momentum=my_momentum, 
                         my_lr_sched=my_lr_sched,
                         my_lr_init=my_lr_init, my_lr_decay_steps=my_lr_decay_steps, 
                         my_lr_decay_rate=my_lr_decay_rate)        
        
        # save the inital (!) weights to be able to restore them  
        cnn.save_weights('cnn_weights.h5') # save the initial weights 
        
    # reset weights(standard)
    if reset and not build:
        cnn.load_weights('cnn_weights.h5')
 
    # Callback list 
    # ~~~~~~~~~~~~~
    use_scheduler = True
    if my_lr_sched == None:
        use_scheduler = False
    lr_history = LrHistory(use_scheduler)
    callbacks_list = [lr_history]
    if fig1 != None:
        epoch_plot = EpochPlot(epochs, fig1, ax1_1, ax1_2)
        callbacks_list.append(epoch_plot)
    
    start_t = time.perf_counter()
    if reset:
        history = cnn.fit(train_imgs, train_labels, initial_epoch=0, epochs=epochs, batch_size=batch_size, verbose=1, shuffle=True, 
                  validation_data=(test_imgs, test_labels), callbacks=callbacks_list) 
    else:
        history = cnn.fit(train_imgs, train_labels, epochs=epochs, batch_size=batch_size, verbose=1, shuffle=True, 
                validation_data=(test_imgs, test_labels), callbacks=callbacks_list ) 
    end_t = time.perf_counter()
    fit_t = end_t - start_t
    
    return cnn, fit_t, history, x_optimizer  # we return cnn to be able to use it by other functions in the Jupyter later on

Note, how big our interface became; there are a lot of parameters for the control of our training. Our set of configuration parameters is now very similar to what we used for MLP training runs before.

Note also how we provided a list of our callbacks to the “model.fit()”-function.

Setting up a training run

Now , we are only a small step away from testing our modified CNN-setup. We only need one further cell:

Jupyter Cell 8:

# Perform a training run 
# ********************

# Prepare the plotting 
# The really important command for interactive (=interediate) plot updating
%matplotlib notebook
plt.ion()

#sizing
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 8
fig_size[1] = 3

# One figure 
# -----------
fig1 = plt.figure(1)
#fig2 = plt.figure(2)

# first figure with two plot-areas with axes 
# --------------------------------------------
ax1_1 = fig1.add_subplot(121)
ax1_2 = fig1.add_subplot(122)
nfig1.canvas.draw()

# second figure with just one plot area with axes
# -------------------------------------------------
#ax2 = fig2.add_subplot(121)
#ax2_1 = fig2.add_subplot(121)
#ax2_2 = fig2.add_subplot(122)
#fig2.canvas.draw()

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Parameterization of the training run 

build = False
build = True
if cnn == None:
    build = True
    x_optimizer = None 
batch_size=64
epochs=80
reset = False # we want training to start again with the initial weights

my_loss    ='categorical_crossentropy'
my_metrics =['accuracy']

my_regularizer = None
my_regularizer = 'l2'
my_reg_param_l2 = 0.0009
my_reg_param_l1 = 0.01


my_optimizer      = 'rmsprop'       # Present alternatives:  rmsprop, nadam, adamax 
my_momentum       = 0.6             # momentum value 
my_lr_sched       = 'powerSched'    # Present alternatrives: None, powerSched, exponential 
#my_lr_sched       = None           # Present alternatrives: None, powerSched, exponential 
my_lr_init        = 0.001           # initial leaning rate  
my_lr_decay_steps = 1               # decay steps = 1 
my_lr_decay_rate  = 0.001           # decay rate 


li_conv_1   = [64, (3,3), 0] 
li_conv_2   = [64, (3,3), 0] 
li_conv_3   = [128, (3,3), 0] 
li_Conv     = [li_conv_1, li_conv_2, li_conv_3]
li_pool_1   = [(2,2)]
li_pool_2   = [(2,2)]
li_Pool     = [li_pool_1, li_pool_2]
li_dense_1  = [120, 0]
#li_dense_2  = [30, 0]
li_dense_3  = [10, 0]
li_MLP      = [li_dense_1, li_dense_2, li_dense_3]
li_MLP      = [li_dense_1, li_dense_3]
input_shape = (28,28,1)

try: 
    if gpu:
        with tf.device("/GPU:0"):
            cnn, fit_time, history, x_optimizer  = train( cnn, build, train_imgs, train_labels, 
                                            li_Conv, li_Pool, li_MLP, input_shape, 
                                            reset, epochs, batch_size, 
                                            my_loss=my_loss, my_metrics=my_metrics, 
                                            my_regularizer=my_regularizer, 
                                            my_reg_param_l2=my_reg_param_l2, my_reg_param_l1=my_reg_param_l1,  
                                            my_optimizer=my_optimizer, my_momentum = 0.8,  
                                            my_lr_sched=my_lr_sched, 
                                            my_lr_init=my_lr_init, my_lr_decay_steps=my_lr_decay_steps, 
                                            my_lr_decay_rate=my_lr_decay_rate,  
                                            fig1=fig1, ax1_1=ax1_1, ax1_2=ax1_2
                                            )
        print('Time_GPU: ', fit_time)  
    else:
        with tf.device("/CPU:0"):
            cnn, fit_time, history = train( cnn, build, train_imgs, train_labels, 
                                            li_Conv, li_Pool, li_MLP, input_shape, 
                                            reset, epochs, batch_size, 
                                            my_loss=my_loss, my_metrics=my_metrics, 
                                            my_regularizer=my_regularizer, 
                                            my_reg_param_l2=my_reg_param_l2, my_reg_param_l1=my_reg_param_l1,  
                                            my_optimizer=my_optimizer, my_momentum = 0.8, 
                                            my_lr_sched=my_lr_sched, 
                                            my_lr_init=my_lr_init, my_lr_decay_steps=my_lr_decay_steps, 
                                            my_lr_decay_rate=my_lr_decay_rate,  
                                            fig1=fig1, ax1_1=ax1_1, ax1_2=ax1_2
                                            )
        print('Time_CPU: ', fit_time)  
except SystemExit:
    print("stopped due to exception")

Results

Below I show screenshots taken during training for the parameters defined above.

The left plot shows the achieved accuracy; red for the validation set (around 99.32%). The right plot shows the decline of the loss function relative to the original value. The green lines are for the training data, the red for the validation data.

Besides the effect of seeing the data change and the plots evolve during data, we can also take the result with us that we have improved the accuracy already from 99.0% to 99.32%. When you play around with the available hyperparameters a bit you may find that 99.25% is a reproducible accuracy. But in some cases you may reach 99.4% as best values.

The following plots have best values around 99.4% with an average above 99.35%.

Epoch 13/80
935/938 [============================>.] - ETA: 0s - loss: 0.0096 - accuracy: 0.9986
present lr:  7.57920279e-05
present iteration: 12194
938/938 [==============================] - 6s 6ms/step - loss: 0.0096 - accuracy: 0.9986 - val_loss: 0.0238 - val_accuracy: 0.9944
Epoch 19/80
937/938 [============================>.] - ETA: 0s - loss: 0.0064 - accuracy: 0.9993
present lr:  5.3129319e-05
present iteration: 17822
938/938 [==============================] - 6s 6ms/step - loss: 0.0064 - accuracy: 0.9993 - val_loss: 0.0245 - val_accuracy: 0.9942
Epoch 22/80
930/938 [============================>.] - ETA: 0s - loss: 0.0056 - accuracy: 0.9994
present lr:  4.6219262e-05
present iteration: 20636
938/938 [==============================] - 6s 6ms/step - loss: 0.0056 - accuracy: 0.9994 - val_loss: 0.0238 - val_accuracy: 0.9942
Epoch 30/80
937/938 [============================>.] - ETA: 0s - loss: 0.0043 - accuracy: 0.9996
present lr:  3.43170905e-05
present iteration: 28140
938/938 [==============================] - 6s 6ms/step - loss: 0.0043 - accuracy: 0.9997 - val_loss: 0.0239 - val_accuracy: 0.9941
Epoch 55/80
937/938 [============================>.] - ETA: 0s - loss: 0.0028 - accuracy: 0.9998
present lr:  1.
90150222e-05
present iteration: 51590
938/938 [==============================] - 6s 6ms/step - loss: 0.0028 - accuracy: 0.9998 - val_loss: 0.0240 - val_accuracy: 0.9940
Epoch 69/80
936/938 [============================>.] - ETA: 0s - loss: 0.0025 - accuracy: 0.9999
present lr:  1.52156063e-05
present iteration: 64722
938/938 [==============================] - 6s 6ms/step - loss: 0.0025 - accuracy: 0.9999 - val_loss: 0.0245 - val_accuracy: 0.9940
Epoch 70/80
933/938 [============================>.] - ETA: 0s - loss: 0.0024 - accuracy: 0.9999
present lr:  1.50015e-05
present iteration: 65660
938/938 [==============================] - 6s 6ms/step - loss: 0.0024 - accuracy: 0.9999 - val_loss: 0.0245 - val_accuracy: 0.9940
Epoch 76/80
937/938 [============================>.] - ETA: 0s - loss: 0.0024 - accuracy: 0.9999
present lr:  1.38335554e-05
present iteration: 71288
938/938 [==============================] - 6s 6ms/step - loss: 0.0024 - accuracy: 0.9999 - val_loss: 0.0237 - val_accuracy: 0.9943

Parameters of the last run were

build = True
if cnn == None:
    build = True
    x_optimizer = None 

batch_size=64
epochs=80
reset = True 

my_loss    ='categorical_crossentropy'
my_metrics =['accuracy']
my_regularizer = None
my_regularizer = 'l2'
my_reg_param_l2 = 0.0008
my_reg_param_l1 = 0.01

my_optimizer      = 'rmsprop'       # Present alternatives:  rmsprop, nadam, adamax 
my_momentum       = 0.5            # momentum value 
my_lr_sched       = 'powerSched'    # Present alternatrives: None, powerSched, exponential 
#my_lr_sched       = None           # Present alternatrives: None, powerSched, exponential 
my_lr_init        = 0.001           # initial leaning rate  
my_lr_decay_steps = 1               # decay steps = 1 
my_lr_decay_rate  = 0.001           # decay rate 

li_conv_1   = [64, (3,3), 0] 
li_conv_2   = [64, (3,3), 0] 
li_conv_3   = [128, (3,3), 0] 
li_Conv     = [li_conv_1, li_conv_2, li_conv_3]
li_pool_1   = [(2,2)]
li_pool_2   = [(2,2)]
li_Pool     = [li_pool_1, li_pool_2]
li_dense_1  = [120, 0]
#li_dense_2  = [30, 0]
li_dense_3  = [10, 0]
li_MLP      = [li_dense_1, li_dense_2, li_dense_3]
li_MLP      = [li_dense_1, li_dense_3]
input_shape = (28,28,1)

It is interesting that RMSProp only requires small values of the L2-regularizer for a sufficient stabilization – such that the loss curve for the validation data does not rise again substantially. Instead we observe an oscillation around a minimum after the learning rate as decreased sufficiently.

Addendum, 15.06.2020

An attentive reader has found out that I have cheated a bit: I have used 64 maps at the first convolutional layer instead of 32 according to the setup in the previous article. Yes, sorry. To compensate for this I include the plot of a run with 32 maps at the first convolution. The blue line marks an accuracy of 99.35%. You see that we can get above it with 32 maps, too.

The parameters were:

build = True
if cnn == None:
    build = True
    x_optimizer = None 
batch_size=64
epochs=80
reset = False # reset the initial weight values to those saved?  

my_loss    ='categorical_crossentropy'
my_metrics =['accuracy']

my_regularizer = None
my_regularizer = 'l2'
my_reg_param_l2 = 0.001
#my_reg_param_l2 = 0.01
my_reg_param_l1 = 0.01

my_
optimizer      = 'rmsprop'       # Present alternatives:  rmsprop, nadam, adamax 
my_momentum       = 0.9           # momentum value 
my_lr_sched       = 'powerSched'    # Present alternatrives: None, powerSched, exponential 
#my_lr_sched       = None           # Present alternatrives: None, powerSched, exponential 
my_lr_init        = 0.001           # initial leaning rate  
my_lr_decay_steps = 1               # decay steps = 1 
my_lr_decay_rate  = 0.001           # decay rate 

li_conv_1    = [32, (3,3), 0] 
li_conv_2    = [64, (3,3), 0] 
li_conv_3    = [128, (3,3), 0] 
li_Conv      = [li_conv_1, li_conv_2, li_conv_3]
li_Conv_Name = ["Conv2D_1", "Conv2D_2", "Conv2D_3"]
li_pool_1    = [(2,2)]
li_pool_2    = [(2,2)]
li_Pool      = [li_pool_1, li_pool_2]
li_Pool_Name = ["Max_Pool_1", "Max_Pool_2", "Max_Pool_3"]
li_dense_1   = [120, 0]
#li_dense_2  = [30, 0]
li_dense_3   = [10, 0]
li_MLP       = [li_dense_1, li_dense_2, li_dense_3]
li_MLP       = [li_dense_1, li_dense_3]
input_shape  = (28,28,1)

Epoch 15/80
926/938 [============================>.] - ETA: 0s - loss: 0.0095 - accuracy: 0.9988
present lr:  6.6357e-05
present iteration: 14070
938/938 [==============================] - 4s 5ms/step - loss: 0.0095 - accuracy: 0.9988 - val_loss: 0.0268 - val_accuracy: 0.9940
Epoch 23/80
935/938 [============================>.] - ETA: 0s - loss: 0.0067 - accuracy: 0.9995
present lr:  4.42987512e-05
present iteration: 21574
938/938 [==============================] - 4s 5ms/step - loss: 0.0066 - accuracy: 0.9995 - val_loss: 0.0254 - val_accuracy: 0.9943
Epoch 35/80
936/938 [============================>.] - ETA: 0s - loss: 0.0049 - accuracy: 0.9996
present lr:  2.95595619e-05
present iteration: 32830
938/938 [==============================] - 4s 5ms/step - loss: 0.0049 - accuracy: 0.9996 - val_loss: 0.0251 - val_accuracy: 0.9945

Another stupid thing, which I should have mentioned:
I have not yet found a simple way of how to explicitly reset the number of iterations, only,

without a recompilation
or reloading a fully saved model at iteration 0 instead of reloading only the weights
or writing my own scheduler class based on epochs or batches.

With the present code you would have to (re-) build the model to avoid starting with a large number of “iterations” – and thus a small learning-rate in a second training run. My “reset”-parameter alone does not help.

By the way: Shape of the weight matrices

Some of my readers may be tempted to have a look at the weight tensors. This is possible via the optimizer and a callback. Then they may wonder about the dimensions, which are logically a bit different from the weight matrices I have used in my code for MLPs in another article series.

In the last article of this CNN series I have mentioned that Keras takes care of configuring the weight matrices itself as soon as it got all information about the layers and layers’ nodes (or units). Now, this still leaves some open degrees of handling the enumeration of layers and matrices; it is plausible that the logic of the layer and weight association must follow a convention. E.g., you can associate a weight matrix with the receiving layer in forward propagation direction. This is what Keras and TF2 do and it is different from what I did in my MLP code. Even if you have fixed this point, then you still can discuss the row/column-layout (shape) of the weight matrix:

Keras and TF2 require the “input_shape” i.e. the shape of the tensor which is fed into the present layer. Regarding the weight matrix it is the number of nodes (units) of the previous layer (in FW direction) which is interesting – normally this is given by the 2nd dimension number of the tensors shape. This has to be combined with the number of units in the present
layer. The systematics is that TF2 and Keras define the weight matrix between two layers according to the following convention for two adjacent layers L_N and L_(N+1) in FW direction:

L_N with A nodes (6 units) and L_(N+1) with B nodes (4 units) give a shape of the weight matrix as (A,B). [(6,4)]

See the references at the bottom of the article.

Note that this is NOT what we would expect from our handling of the MLP. However, the amount of information kept in the matrix is, of course, the same. It is just a matter of convention and array transposition for the required math operations during forward and error backward propagation. If you (for reasons of handling gradients) concentrate on BW propagation then the TF2/Keras convention is quite reasonable.

Conclusion

The Keras framework gives you access to a variety of hyperparameters you can use to control the use of a regularizer, the optimizer, the decline of the learning rate with batches/epochs, the momentum. The handling is a bit unconventional, but one gets pretty fast used to it.

We once again learned that it pays to invest a bit in a variation of such parameters, when you want to fight for the last tenth of percents beyond 99.0% accuracy. In the case of MNIST we can drive accuracy to 99.35%.

With the next article of this series

A simple CNN for the MNIST dataset – IV – Visualizing the activation output of convolutional layers and maps

we turn more to the question of visualizing some data at the convolutional layers to better understand their working.

Links

Layers and the shape of the weight matrices
https://keras.io/guides/sequential_model/
https://stackoverflow.com/questions/ 44791014/ understanding-keras-weight-matrix-of-each-layers
https://keras.io/guides/ making new layers and models via subclassing/

Learning rate and adaptive optimizers
https://towardsdatascience.com/learning-rate-schedules-and-adaptive-learning-rate-methods-for-deep-learning-2c8f433990d1
https://machinelearningmastery.com/ understand-the-dynamics-of-learning-rate-on-deep-learning-neural-networks/
https://faroit.com/keras-docs/2.0.8/optimizers/
https://keras.io/api/optimizers/
https://keras.io/api/optimizers/ learning_rate_schedules/
https://www.jeremyjordan.me/nn-learning-rate/

Keras Callbacks
https://keras.io/api/callbacks/
https://keras.io/guides/ writing_your_own_callbacks/
https://keras.io/api/callbacks/ learning_rate_scheduler/

Interactive plotting
https://stackoverflow.com/questions/ 39428347/ interactive-matplotlib-plots-in-jupyter-notebook
https://matplotlib.org/3.2.1/api/ _as_gen/ matplotlib.pyplot.ion.html
https://matplotlib.org/3.1.3/tutorials/ introductory/ usage.html

Further articles in this series

A simple CNN for the MNIST dataset – XI – Python code for filter visualization and OIP detection
A simple CNN for the MNIST dataset – X – filling some gaps in filter visualization
A simple CNN for the MNIST dataset – IX – filter visualization at a convolutional layer
A simple CNN for the MNIST dataset – VIII – filters and features – Python code to visualize patterns which activate a map strongly
A simple CNN for the MNIST dataset – VII – outline of steps to visualize image patterns which trigger filter maps
A simple CNN for the MNIST dataset – VI – classification by activation patterns and the role of the CNN’s MLP part
A simple CNN for the MNIST dataset – V – about the difference of activation patterns and features

A simple CNN for the MNIST datasets – II – building the CNN with Keras and a first test

Posted on 31. May 2020 by eremo

I continue with my series on first exploratory steps with CNNs. After all the discussion of CNN basics in the last article,

A simple CNN for the MNIST datasets – I,

we are well prepared to build a very simple CNN with Keras. By simple I mean simple enough to handle the MNIST digit images. The Keras API for creating CNN models, layers and activation functions is very convenient; a simple CNN does not require much code. So, the Jupyter environment is sufficient for our first experiment.

An interesting topic is the use of a GPU. After a somewhat frustrating experience with a MLP on the GPU of a NV 960 GTX in comparison to a i7 6700K CPU I am eager to see whether we get a reasonable GPU acceleration for a CNN. So, we should prepare our code to use the GPU. This requires a bit of preparation.

We should also ask a subtle question: What do we expect from a CNN in comparison to a MLP regarding the MNIST data? A MLP with 2 hidden layers (with 70 and 30 nodes) provided over 99.5% accuracy on the training data and almost 98% accuracy on a test dataset after some tweaking. Even with basic settings for our MLP we arrived at a value over 97.7% after 35 epochs – below 8 secs. Well, a CNN is probably better in feature recognition than a cluster detection algorithm. But we are talking about the last 2 % of remaining accuracy. I admit that I did not know what to expect …

A MLP as an important part of a CNN

At the end of the last article I had discussed a very simple layer structure of convolutional and pooling layers:

Layer 0: Input layer (tensor of original image data, 3 layers for color channels or one layer for a gray channel)
Layer 1: Conv layer (small 3×3 kernel, stride 1, 32 filters => 32 maps (26×26), overlapping filter areas)
Layer 2: Pooling layer (2×2 max pooling => 32 (13×13) maps,
a map node covers 4×4 non overlapping areas per node on the original image)
Layer 3: Conv layer (3×3 kernel, stride 1, 64 filters => 64 maps (11×11),
a map node covers 8×8 overlapping areas on the original image (total effective stride 2))
Layer 4: Pooling layer (2×2 max pooling => 64 maps (5×5),
a map node covers 10×10 areas per node on the original image (total effective stride 5), some border info lost)
Layer 5: Conv layer (3×3 kernel, stride 1, 64 filters => 64 maps (3×3),
a map node covers 18×18 areas per node (effective stride 5), some border info lost )

This is the CNN structure we are going to use in the near future. (Actually, I followed a suggestion of Francois Chollet; see the literature list in the last article). Let us assume that we somehow have established all these convolution and pooling layers for a CNN. Each layer producse some “feature“-related output, structured in form of a tensors. This led to an open question at the end of the last article:

Where and by what do we get a classification of the resulting data with respect to the 10 digit categories of the MNIST images?

Applying filters and extracting “feature hierarchies” of an image alone does not help without a “learned” judgement about these data. But the answer is very simple:

Use a MLP after the last Conv layer and feed it with data from this Conv layer!

When we think in terms of nodes and artificial neurons, we could say: We just have to connect the “nodes” of the feature maps of layer 5
in our special CNN with the nodes of an input layer of a MLP. As a MLP has a flat input layer we need to prepare 9×64 = 576 receiving “nodes” there. We would use weights with a value of “1.0” along these special connections.

Mathematically, this approach can be expressed in terms of a “flattening” operation on the tensor data produced by the the last Conv data. In Numpy terms: We need to reshape the multidimensional tensor containing the values across the stack of maps at the last Conv2D layer into a long 1D array (= a vector).

From a more general perspective we could say: Feeding the output of the Conv part of our CNN into a MLP for classification is quite similar to what we did when we pre-processed the MNIST data by an unsupervised cluster detection algorithm; also there we used the resulting data as input to an MLP. There is one big difference, however:

The optimization of the network’s weights during training requires a BW propagation of error terms (more precisely: derivatives of the CNN’s loss function) across the MLP AND the convolutional and pooling layers. Error BW propagation should not be stopped at the MLP’s input layer: It has to move from the output layer of the MLP back to the MLP’s input layer and from there to the convolutional and pooling layers. Remember that suitable filter kernels have to be found during (supervised) training.

If you read my PDF on the error back propagation for a MLP
PDF on the math behind Error Back_Propagation
and think a bit about its basic recipes and results you quickly see that the “input layer” of the MLP is no barrier to error back propagation: The “deltas” discussed in the PDF can be back-propagated right down to the MLP’s input layer. Then we just apply the chain rule again. The partial derivatives at the nodes of the input layer with respect to their input values are just “1”, because the activation function there is the identity function. The “weights” between the last Conv layer and the input layer of the MLP are no free parameters – we do not need to care about them. And then everything goes its normal way – we apply chain rule after chain rule for all nodes of the maps to determine the gradients of the CNN’s loss function with respect to the weights there. But you need not think about the details – Keras and TF2 will take proper care about everything …

But, you should always keep the following in mind: Whatever we discuss in terms of layers and nodes – in a CNN these are only fictitious interpretations of a series of mathematical operations on tensor data. Not less, not more ..,. Nodes and layers are just very helpful (!) illustrations of non-cyclic graphs of mathematical operations. KI on the level of my present discussion (MLPs, CNNs) “just” corresponds to algorithms which emerge out of a specific deterministic approach to solve an optimization problem.

Using Tensorflow 2 and Keras

Let us now turn to coding. To be able to use a Nvidia GPU we need a Cuda/Cudnn installation and a working Tensorflow backend for Keras. I have already described the installation of CUDA 10.2 and CUDNN on an Opensuse Linux system in some detail in the article Getting a Keras based MLP to run with Cuda 10.2, Cudnn 7.6 and TensorFlow 2.0 on an Opensuse Leap 15.1 system. You can follow the hints there. In case you run into trouble on your Linux distribution try everything with Cuda 10.1.

Some more hints: TF2 in version 2.2 can be installed by the Pip-mechanism in your virtual Python environment (“pip install –upgrade tensorflow”). TF2 contains already a special Keras version – which is the one we are going to use in our upcoming experiment. So, there is no need to install Keras separately with “pip”. Note also that, in contrast to TF1, it is NOT necessary to install a separate package “tensorflow-gpu”. If all these things are new to you: You find some articles on creating an adequate ML test and development environment based on Python/PyDev/Jupyter somewhere else in this blog.

Imports and settings for CPUs/GPUs

We shall use a Jupyter notebook to perform the basic experiments; but I recommend strongly to consolidate your code in Python files of an Eclipse/PyDev environment afterwards. Before you start your virtual Python environment from a Linux shell you should set the following environment variables:

$>export OPENBLAS_NUM_THREADS=4 # or whatever is reasonable for your CPU (but do not use all CPU cores and/or hyper threads                            
$>export OMP_NUM_THREADS=4                                
$>export TF_XLA_FLAGS=--tf_xla_cpu_global_jit
$>export XLA_FLAGS=--xla_gpu_cuda_data_dir=/usr/local/cuda
$>source bin/activate                                     
(ml_1) $> jupyter notebook

Required Imports

The following commands in a first Jupyter cell perform the required library imports:

import numpy as np
import scipy
import time 
import sys 
import os

import tensorflow as tf
from tensorflow import keras as K
from tensorflow.python.keras import backend as B 
from keras import models
from keras import layers
from keras.utils import to_categorical
from keras.datasets import mnist
from tensorflow.python.client import device_lib

from sklearn.preprocessing import StandardScaler

Do not ignore the statement “from tensorflow.python.keras import backend as B“; we need it later.

The “StandardScaler” of Scikit-Learn will help us to “standardize” the MNIST input data. This is a step which you should know already from MLPs … You can, of course, also experiment with different normalization procedures. But in my opinion using the “StandardScaler” is just convenient. ( I assume that you already have installed scikit-learn in your virtual Python environment).

Settings for CPUs/GPUs

With TF2 the switching between CPU and GPU is a bit of a mess. Not all new parameters and their settings work as expected. As I have explained in the article on the Cuda installation named above, I, therefore, prefer to an old school, but reliable TF1 approach and use the compatibility interface:

#gpu = False 
gpu = True
if gpu: 
    GPU = True;  CPU = False; num_GPU = 1; num_CPU = 1
else: 
    GPU = False; CPU = True;  num_CPU = 1; num_GPU = 0

config = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=6,
                        inter_op_parallelism_threads=1, 
                        allow_soft_placement=True,
                        device_count = {'CPU' : num_CPU,
                                        'GPU' : num_GPU}, 
                        log_device_placement=True

                       )
config.gpu_options.per_process_gpu_memory_fraction=0.35
config.gpu_options.force_gpu_compatible = True
B.set_session(tf.compat.v1.Session(config=config))

We are brave and try our first runs directly on a GPU. The statement “log_device_placement” will help us to get information about which device – CPU or GP – is actually used.

Loading and preparing MNIST data

We prepare a function which loads and prepares the MNIST data for us. The statements reflect more or less what we did with the MNIST dat when we used them for MLPs.

  
# load MNIST 
# **********
def load_Mnist():
    mnist = K.datasets.mnist
    (X_train, y_train), (X_test, y_test) = mnist.load_
data()

    #print(X_train.shape)
    #print(X_test.shape)

    # preprocess - flatten 
    len_train =  X_train.shape[0]
    len_test  =  X_test.shape[0]
    X_train = X_train.reshape(len_train, 28*28) 
    X_test  = X_test.reshape(len_test, 28*28) 

    #concatenate
    _X = np.concatenate((X_train, X_test), axis=0)
    _y = np.concatenate((y_train, y_test), axis=0)

    _dim_X = _X.shape[0]

    # 32-bit
    _X = _X.astype(np.float32)
    _y = _y.astype(np.int32)

    # normalize  
    scaler = StandardScaler()
    _X = scaler.fit_transform(_X)

    # mixing the training indices - MUST happen BEFORE encoding
    shuffled_index = np.random.permutation(_dim_X)
    _X, _y = _X[shuffled_index], _y[shuffled_index]

    # split again 
    num_test  = 10000
    num_train = _dim_X - num_test
    X_train, X_test, y_train, y_test = _X[:num_train], _X[num_train:], _y[:num_train], _y[num_train:]

    # reshape to Keras tensor requirements 
    train_imgs = X_train.reshape((num_train, 28, 28, 1))
    test_imgs  = X_test.reshape((num_test, 28, 28, 1))
    #print(train_imgs.shape)
    #print(test_imgs.shape)

    # one-hot-encoding
    train_labels = to_categorical(y_train)
    test_labels  = to_categorical(y_test)
    #print(test_labels[4])
    
    return train_imgs, test_imgs, train_labels, test_labels

if gpu:
    with tf.device("/GPU:0"):
        train_imgs, test_imgs, train_labels, test_labels= load_Mnist()
else:
    with tf.device("/CPU:0"):
        train_imgs, test_imgs, train_labels, test_labels= load_Mnist()

Some comments:

Normalization and shuffling: The “StandardScaler” is used for data normalization. I also shuffled the data to avoid any pre-ordered sequences. We know these steps already from the MLP code we built in another article series.
Image data in tensor form: Something, which is different from working with MLPs is that we have to fulfill some requirements regarding the form of input data. From the last article we know already that our data should have a tensor compatible form; Keras expects data from us which have a certain shape. So, no flattening of the data into a vector here as we were used to with MLPs. For images we, instead, need the width, the height of our images in terms of pixels and also the depth (here 1 for gray-scale images). In addition the data samples are to be indexed along the first tensor axis.
This means that we need to deliver a 4-dimensional array corresponding to a TF tensor of rank 4. Keras/TF2 will do the necessary transformation from a Numpy array to a TF2 tensor automatically for us. The corresponding Numpy shape of the required array is:
(samples, height, width, depth)
Some people also use the term “channels” instead of “depth”. In the case of MNIST we reshape the input array – “train_imgs” to (num_train, 28, 28, 1), with “num_train” being the number of samples.
The use of the function “to_categorical()”, more precisely “tf.keras.utils.to_categorical()”, corresponds to a one-hot-encoding of the target data. All these concepts are well known from our study of MLPs and MNIST. Keras makes life easy regarding this point …
The statements “with tf.device(“/GPU:0”):” and “with tf.device(“/CPU:0”):” delegate the execution of (suitable) code on the GPU or the CPU. Note that due to the Python/Jupyter environment some code will of course also be executed on the CPU – even if you delegated execution to the GPU.

If you activate the print statements the resulting output should be:

(60000, 
28, 28)
(10000, 28, 28)
(60000, 28, 28, 1)
(10000, 28, 28, 1)
[0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]

The last line proves the one-hot-encoding.

The CNN architecture – and Keras’ layer API

Now, we come to a central point: We need to build the 5 central layers of our CNN-architecture. When we build our own MLP code we used a special method to build the different weight arrays, which represented the number of nodes via the array dimensions. A simple method was sufficient as we had no major differences between layers. But with CNNs we have to work with substantially different types of layers. So, how are layers to be handled with Keras?

Well, Keras provides a full layer API with different classes for a variety of layers. You find substantial information on this API and different types of layers at
https://keras.io/api/layers/.

The first section which is interesting for our experiment is https://keras.io/api/ layers/ convolution_layers/ convolution2d/.
You do not need to read much to understand that this is exactly what we need for the “convolutional layers” of our simple CNN model. But how do we instantiate the Conv2D class such that the output works seamlessly together with other layers?

Keras makes our life easy again. All layers are to be used in a purely sequential order. (There are much more complicated layer topologies you can build with Keras! Oh, yes …). Well, you guess it: Keras offers you a model API; see:
https://keras.io/api/models/.

And there we find a class for a “sequential model” – see https://keras.io/api/ models/sequential/. This class offers us a method “add()” to add layers (and create an instance of the related layer class).

The only missing ingredient is a class for a “pooling” layer. Well, you find it in the layer API documentation, too. The following image depicts the basic structure of our CNN (see the left side of the drawing), as we designed it (see the list above):

Keras code for the Conv and pooling layers

The convolutional part of the CNN can be set up by the following commands:

Convolutional part of the CNN

# Sequential layer model of our CNN
# ***********************************

# Build the Conv part of the CNN
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# Choose the activation function for the Conv2D layers 
conv_act_func = 1
li_conv_act_funcs = ['sigmoid', 'relu', 'elu', 'tanh']
cact = li_conv_act_funcs[conv_act_func]

# Build the Conv2D layers 
cnn = models.Sequential()
cnn.add(layers.Conv2D(32, (3,3), activation=cact, input_shape=(28,28,1)))
cnn.add(layers.MaxPooling2D((2,2)))
cnn.add(layers.Conv2D(64, (3,3), activation=cact))
cnn.add(layers.MaxPooling2D((2,2)))
cnn.add(layers.Conv2D(64, (3,3), activation=cact))

Easy, isn’t it? The nice thing about Keras is that it cares about the required tensor ranks and shapes itself; in a sequential model it evaluates the output of a already defined layer to guess the shape of the tensor data entering the next layer. Thus we have to define an “input_shape” only for data entering the first Conv2D layer!

The first Conv2D layer requires, of course, a shape for the input data. We must also tell the layer interface how many filters and “feature maps” we want to use. In our case we produce 32 maps by first Conv2D layer and 64 by the other two Conv2D layers. The (3×3)-parameter defines the filter area size to be covered by the filter kernel: 3×3 pixels. We define no “stride”, so a stride of 1 is automatically used; all 3×3 areas lie close to each other and overlap each other. These parameters result in 32 maps of size 26×26 after the first convolution. The size of the maps of the other layers are given in the layer list at the beginning of this article.

In addition you saw from the code that we chose an activation function via an index of a Python list of reasonable alternatives. You find an explanation of all the different activation functions in the literature. (See also: wikipedia: Activation function). The “sigmoid” function should be well known to you already from my other article series on a MLP.

Now, we have to care about the MLP part of the CNN. The code is simple:

MLP part of the CNN

# Build the MLP part of the CNN
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# Choose the activation function for the hidden layers of the MLP 
mlp_h_act_func = 0
li_mlp_h_act_funcs = ['relu', 'sigmoid', 'tanh']
mhact = li_mlp_h_act_funcs[mlp_h_act_func]

# Choose the output function for the output layer of the MLP 
mlp_o_act_func = 0
li_mlp_o_act_funcs = ['softmax', 'sigmoid']
moact = li_mlp_o_act_funcs[mlp_o_act_func]

# Build the MLP layers 
cnn.add(layers.Flatten())
cnn.add(layers.Dense(70, activation=mhact))
#cnn.add(layers.Dense(30, activation=mhact))
cnn.add(layers.Dense(10, activation=moact))

This all is very straight forward (with the exception of the last statement). The “Flatten”-layer corresponds to the MLP’s inout layer. It just transforms the tensor output of the last Conv2D layer into the flat form usable for the first “Dense” layer of the MLP. The first and only “Dense layer” (MLP hidden layer) builds up connections from the flat MLP “input layer” and associates it with weights. Actually, it prepares a weight-tensor for a tensor-operation on the output data of the feeding layer. Dense means that all “nodes” of the previous layer are connected to the present layer’s own “nodes” – meaning: setting the right dimensions of the weight tensor (matrix in our case). As a first trial we work with just one hidden layer. (We shall later see that more layers will not improve accuracy.)

I choose the output function (if you will: the activation function of the output layer) as “softmax“. This gives us a probability distribution across the classification categories. Note that this is a different approach compared to what we have done so far with MLPs. I shall comment on the differences in a separate article when I find the time for it. At this point I just want to indicate that softmax combined with the “categorical cross entropy loss” is a generalized version of the combination “sigmoid” with “log loss” as we used it for our MLP.

Parameterization

The above code for creating the CNN would work. However, we want to be able to parameterize our simple CNN. So we include the above statements in a function for which we provide parameters for all layers. A quick solution is to define layer parameters as elements of a Python list – we then get one list per layer. (If you are a friend of clean code design I recommend to choose a more elaborated approach; inject just one parameter object containing all parameters in a structured way. I leave this exercise to you.)

We now combine the statements for layer construction in a function:

  
# Sequential layer model of our CNN
# ***********************************

# just for illustration - the real parameters are fed later 
# 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
li_conv_1   = [32, (3,3), 0] 
li_conv_2   = [64, (3,3), 0] 
li_conv_3   = [64, (3,3), 0] 
li_Conv     = [li_conv_1, li_conv_2, li_conv_3]
li_pool_1   = [(2,2)]
li_pool_2   = [(2,2)]
li_Pool     = [li_pool_1, li_pool_2]
li_dense_1  = [70, 0]
li_dense_2  = [10, 0]
li_MLP      = [li_dense_1, li_dense_2]
input_shape = (28,28,1)

# important !!
# ~~~~~~~~~~~
cnn = None

def build_cnn_simple(li_Conv, li_Pool, li_MLP, input_shape ):

    # activation functions to be used in Conv-layers 
    li_conv_act_funcs = ['relu', 'sigmoid', 'elu', 'tanh']
    # activation functions to be used in MLP hidden layers  
    li_mlp_h_act_funcs = ['relu', 'sigmoid', 'tanh']
    # activation functions to be used in MLP output layers  
    li_mlp_o_act_funcs = ['softmax', 'sigmoid']

    
    # Build the Conv part of the CNN
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    num_conv_layers = len(li_Conv)
    num_pool_layers = len(li_Pool)
    if num_pool_layers != num_conv_layers - 1: 
        print("\nNumber of pool layers does not fit to number of Conv-layers")
        sys.exit()
    rg_il = range(num_conv_layers)

    # Define a sequential model 
    cnn = models.Sequential()

    for il in rg_il:
        # add the convolutional layer 
        num_filters = li_Conv[il][0]
        t_fkern_size = li_Conv[il][1]
        cact        = li_conv_act_funcs[li_Conv[il][2]]
        if il==0:
            cnn.add(layers.Conv2D(num_filters, t_fkern_size, activation=cact, input_shape=input_shape))
        else:
            cnn.add(layers.Conv2D(num_filters, t_fkern_size, activation=cact))
        
        # add the pooling layer 
        if il < num_pool_layers:
            t_pkern_size = li_Pool[il][0]
            cnn.add(layers.MaxPooling2D(t_pkern_size))
            

    # Build the MLP part of the CNN
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    num_mlp_layers = len(li_MLP)
    rg_im = range(num_mlp_layers)

    cnn.add(layers.Flatten())

    for im in rg_im:
        # add the dense layer 
        n_nodes = li_MLP[im][0]
        if im < num_mlp_layers - 1:  
            m_act   =  li_mlp_h_act_funcs[li_MLP[im][1]]
        else: 
            m_act   =  li_mlp_o_act_funcs[li_MLP[im][1]]
        cnn.add(layers.Dense(n_nodes, activation=m_act))

    return cnn

We return the model “cnn” to be able to use it afterwards.

How many parameters does our CNN have?

The layers contribute with the following numbers of weight parameters:

Layer 1: (32 x (3×3)) + 32 = 320
Layer 3: 32 x 64 x (3×3) + 64 = 18496
Layer 5: 64 x 64 x (3×3) + 64 = 36928
MLP : (576 + 1) x 70 + (70 + 1) x 10 = 41100

Making a total of 96844 weight parameters. Our standard MLP discussed in another article series had (784+1) x 70 + (70 + 1) x 30 + (30 +1 ) x 10 = 57390 weights. So, our CNN is bigger and the CPU time to follow and calculate all the partial derivatives will be significantly higher. So, we should definitely expect some better classification data, shouldn’t we?

Compilation

Now comes a thing which is necessary for models: We have not yet defined the loss function and the optimizer or a learning rate. For the latter Keras can choose a proper value itself – as soon as it knows the loss function. But we should give it a reasonable loss function and a suitable optimizer for gradient descent. This is the main purpose of the “compile()“-function.

cnn.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])

Although TF2 can already analyze the graph of tensor operations for partial derivatives, it cannot guess the beginning of the chain rule sequence.

As we have multiple categories “categorial_crossentropy” is a good choice for the loss function. We should also define which optimized gradient descent method is used; we choose “rmsprop” – as this method works well in most cases. A nice introduction is given here: towardsdatascience: understanding-rmsprop-faster-neural-network-learning-62e116fcf29a. But see the books mentioned in the last article on “rmsprop”, too.

Regarding the use of different metrics for different tasks see machinelearningmastery.com / custom-metrics-deep-learning-keras-python/. In case of a classification problem, we are interested in the categorical “accuracy”. A metric can be monitored during training and will be recorded (besides aother data). We can use it for plotting information on the training process (a topic of the next article).

Training

Training is done by a function model.fit() – here: cnn.fit(). This function accepts a variety of parameters explained here: https://keras.io/ api/ models/ model_training_apis/ #fit-method.

We now can combine compilation and training in one function:

# Training 
def train( cnn, build=False, train_imgs, train_labels, reset, epochs, batch_size, optimizer, loss, metrics,
           li_Conv, li_Poo, li_MLP, input_shape ):
    if build:
        cnn = build_cnn_simple( li_Conv, li_Pool, li_MLP, input_shape)
        cnn.compile(optimizer=optimizer, loss=loss, metrics=metrics)        
        cnn.save_weights('cnn_weights.h5') # save the initial weights 
    # reset weights
    if reset and not build:
        cnn.load_weights('cnn_weights.h5')
    start_t = time.perf_counter()
    cnn.fit(train_imgs, train_labels, epochs=epochs, batch_size=batch_size, verbose=1, shuffle=True) 
    end_t = time.perf_counter()
    fit_t = end_t - start_t
    return cnn, fit_t  # we return cnn to be able to use it by other functions

Note that we save the initial weights to be able to load them again for a new training – otherwise Keras saves the weights as other model data after training and continues with the last weights found. The latter can be reasonable if you want to continue training in defined steps. However, in our forthcoming tests we repeat the training from scratch.

Keras offers a “save”-model and methods to transfer data of a CNN model to files (in two specific formats). For saving weights the given lines are sufficient. Hint: When I specify no path to the file “cnn_weights.h5” the data are – at least in my virtual Python environment – saved in the directory where the notebook is located.

First test

In a further Jupyter cell we place the following code for a test run:

  
# Perform a training run 
# ********************
build = False     
if cnn == None:
    build = True
batch_size=64
epochs=5
reset = True # we want training to start again with the initial weights

optimizer='rmsprop' 
loss='categorical_crossentropy'
metrics=['accuracy']

li_conv_1   = [32, (3,3), 0] 
li_conv_2   = [64, (3,3), 0] 
li_conv_3   = [64, (3,3), 0] 
li_Conv     = [li_conv_1, li_conv_2, li_conv_3]
li_pool_1   = [(2,2)]
li_pool_2   = [(2,2)]
li_Pool     = [li_pool_1, li_pool_2]
li_dense_1  = [70, 0]
li_dense_2  = [10, 0]
li_MLP      = [li_dense_1, li_dense_2]
input_shape = (28,28,1)

try: 
    if gpu:
        with tf.device("/GPU:0"):
            cnn, fit_time = train( cnn, build, train_imgs, train_
labels, 
                                   reset, epochs, batch_size, optimizer, loss, metrics, 
                                   li_Conv, li_Pool, li_MLP, input_shape)
        print('Time_GPU: ', fit_time)  
    else:
        with tf.device("/CPU:0"):
            cnn, fit_time = train( cnn, build, train_imgs, train_labels, 
                                   reset, epochs, batch_size, optimizer, loss, metrics, 
                                   li_Conv, li_Pool, li_MLP, input_shape)
        print('Time_CPU: ', fit_time)  
except SystemExit:
    print("stopped due to exception")

You recognize the parameterization of our train()-function. What results do we get ?

Epoch 1/5
60000/60000 [==============================] - 4s 69us/step - loss: 0.1551 - accuracy: 0.9520
Epoch 2/5
60000/60000 [==============================] - 4s 69us/step - loss: 0.0438 - accuracy: 0.9868
Epoch 3/5
60000/60000 [==============================] - 4s 68us/step - loss: 0.0305 - accuracy: 0.9907
Epoch 4/5
60000/60000 [==============================] - 4s 69us/step - loss: 0.0227 - accuracy: 0.9931
Epoch 5/5
60000/60000 [==============================] - 4s 69us/step - loss: 0.0184 - accuracy: 0.9948
Time_GPU:  20.610678611003095

And a successive check on the test data gives us:

We can ask Keras also for a description of the model:

Accuracy at the 99% level

We got an accuracy on the test data of 99%! With 5 epochs in 20 seconds – on my old GPU.
This leaves us a very good impression – on first sight …

Conclusion

We saw today that it is easy to set up a CNN. We used a simple MLP to solve the problem of classification; the data to its input layer are provided by the output of the last convolutional layer. The tensor there has just to be “flattened”.

The level of accuracy reached is impressing. Well, its also a bit frustrating when we think about the efforts we put into our MLP, but we also get a sense for the power and capabilities of CNNs.

In the next post
A simple CNN for the MNIST dataset – III – inclusion of a learning-rate scheduler, momentum and a L2-regularizer
we will care a bit about plotting. We at least want to see the same accuracy and loss data which we used to plot at the end of our MLP tests.

Linux-Blog – Dr. Mönchmeyer / anracon

Notes about Linux, ML and some simple math …

Author Archives: eremo

Corona App – Standortbestimmung

Drei Perspektiven

Standortermittlung – ?

Was zeigen Recherchen im Internet?

Die App verhält sich bei explizit abgeschalteter Standortbestimmung seltsam

Bewertung

A simple CNN for the MNIST dataset – III – inclusion of a learning-rate scheduler, momentum and a L2-regularizer

Learning rate scheduler and momentum

Required import statements

Include the regularizer via the layer definitions of the CNN

Include a scheduler via a parameter to the optimizer of the CNN

Two callbacks – one for printing the learning rate and an other for plotting during training

Plotting at the end of an epoch

The new training function

Setting up a training run

Results

Addendum, 15.06.2020

By the way: Shape of the weight matrices

Conclusion

Links

Further articles in this series

A simple CNN for the MNIST datasets – II – building the CNN with Keras and a first test

A MLP as an important part of a CNN

Using Tensorflow 2 and Keras

Imports and settings for CPUs/GPUs

Required Imports

Settings for CPUs/GPUs

Loading and preparing MNIST data

The CNN architecture – and Keras’ layer API

Keras code for the Conv and pooling layers

Parameterization

How many parameters does our CNN have?

Compilation

Training

First test

Accuracy at the 99% level

Conclusion