TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – I

Today server services should offer network connectivity for clients with encryption. On Linux StartTLS based services are common – for LDAP, email/groupware servers as well as web servers. To set up SSL/TLS/StartTLS based services we need certificates and encryption keys issued by a central CA – which we trust. Administering your own local CA and server certificates can be a bit challenging without graphical tools – even in smaller networks with a dozen server instances.

In our networks with mainly Opensuse and Debian servers I had used YaST’s CA-module to create a CA and server certificates signed by this CA. The stupid thing is that the required “yast2-ca”-module and its RPM are missing since Opensuse Leap 15.0. This was not a major problem so far; the update processes respected existing certificates, of course. However, some days ago two of my central server certificates – namely the one for my LDAP-server and an Apache2-server – expired. This in turn lead to a breakdown of several other services on other (virtual) machines: SSSD, IMAP, Postfix (SMTP, because these services use the LDAP server among other things as a backend for user authentication. (SSSD itself provides a TLS connection to LDAP.)

The Opensuse documentation cha.security.yast_ca.html is really misleading because it claims to be valid for Opensuse Leap 15.1 – which it is not, as there still is no yast-ca-module available. For me this kind of policy of Opensuse is unbelievable; doesn’t Leap provide the basic platform for SLES? How shall SLES admins in smaller companies tackle the resulting problems? Buy a PKI tool? Everybody talks about secuirty …. but SuSE (???)

I wanted some cost free alternative for my own network – and as a first trial I went for “TinyCA2“.

This became more of an adventure than expected. Part of the hurdles were due to Opensuse specific settings – but also due to the very many different configuration files which had to be adapted for the certificate of my new CA – which came in addition to my old one. (I did not yet want to give up the old CA as some (virtual) servers still have valid server certificates from it.) Another obstacle appeared when Opensuse deleted any new files in “/etc/ssl/certs” after a system restart. Also the GUI of TinyCA2 has some strange “features” regarding default values, which I did not become aware of during my first trials. In addition it seemed to be necessary to replace SHA1 by SHA256. And in the end I got e.g. Apache running with its new server certificate, but not e.g. the slapd.service – due to a access rights problem which was difficult to see.

In this article I shall describe most of the required steps for switching to a TinyCA2 CA and adjusting server settings. I shall concentrate on some simple services as examples. But I hope the general pattern of how to proceed will become clear and help others, who work with Opensuse, to save a bit of time.

Installing and patching TinyCA2

Both Opensuse Leap 15.0 and Opensuse Leap 15.1 provide RPMs for TinyCA2 version 0.7.5. Which is from 2015. If you have a look at GitHub (see the link in the last section of this article) you may also find some (important) patches. On an Opensuse system You install the RPM easily with the help of YaST (yast2). After the installation you find the Perl files of TinyCA2 in the directory “/usr/share/TinyCA2/lib“.

When you start TinyCA2 via the command “tinyca2 &” the first thing you may stumble across is the fact that (among other digests) MD5 and SHA1 are offered as hashing algorithms. Look at the bottom part of the following screenshot:

(By the way: The layout – especially the icons – may look different on your system. It depends on your graphical desktop and your settings for GTK applications)

You see that we get a variety of hashing algorithms offered under the category “Digest”. Most of them are regarded insecure today. So, even in a semiprofessional environment you would like to see something better – e.g. SHA256. Fortunately, another guy (Bill Thorsteinson) had the same problem and he has created a patch for TinyCA2 which enables SHA256. You find the patch at
https://www.systemajik.com/tinyca-sha2/.

Let us try this out; on my ssh session to my central server “myserv” (this is the one with LDAP):

myserv:~ # mkdir /extras/Updates/tinyca
myserv:~ # wget https://www.systemajik.com/wp-uploads/2014/10/tinyca_sha256.patch_.txt -O /central/Updates/tinyca/tinyca_sha256.patch_.txt
--2019-07-20 12:44:57--  https://www.systemajik.com/wp-uploads/2014/10/tinyca_sha256.patch_.txt
Resolving www.systemajik.com (www.systemajik.com)... 206.47.13.3, 2001:470:1f11:b22::8
Connecting to www.systemajik.com (www.systemajik.com)|206.47.13.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4863 (4.7K) [text/plain]
Saving to: ‘/central/Updates/tinyca/tinyca_sha256.patch_.txt’

/central/Updates/tinyca/tinyca_sha256.pat 100%[=================>]   4.75K  --.-KB/s    in 0s      

2019-07-20 12:44:58 (138 MB/s) - ‘/central/Updates/tinyca/tinyca_sha256.patch_.txt’ saved [4863/4863]

myserv:~ # 
myserv:~ # cd /usr/share/TinyCA2/lib
myserv:/usr/share/TinyCA2/lib # cp /extras/Updates/tinyca/tinyca_sha256.patch_.txt .
myserv:/usr/share/TinyCA2/lib # patch --verbose -p1 < tinyca_sha256.patch_.txt
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From e5e25e55f8da2b4d2bad584f2145ca0ff6b3a92a Mon Sep 17 00:00:00 2001
|From: Bill Thorsteinson <bill.git@systemajik.com>
|Date: Thu, 30 Oct 2014 22:26:47 -0400
|Subject: [PATCH] Apply changes
|
|---
...
...
|--- a/REQ.pm
|+++ b/REQ.pm
--------------------------
patching file REQ.pm
Using Plan A...
Hunk #1 succeeded at 59.
Hunk #2 succeeded at 426.
Hmm...  Ignoring the trailing garbage.
done

Note: The “-p1” in “patch –verbose -p1 < tinyca_sha256.patch_.txt” reads “-pONE” and not “-pL” with a small L-letter.

A “tinyca2” command now produces:

Much better !

Importing the old CA from Opensuse?

If you play around with the menus of TinyCA2 you find an option to import other CAs. Could this work with my old YaST-CA? To make a long story short – I did not succeed with this. The reasons are still unclear to me …. TinyCA could not read the relevant information.

So, I really was forced to set up a new CA – with all consequences as issuing and deploying new server certificates and the (trusted) CA certificate on my servers (and the CA cert also on my client machines). Which even in my little network (12 servers- thanks to virtualization) is painstaking …

Creating a new TinyCA2 based CA

Let us create a new CA with TinyCA2. We have some freedom regarding the “common name“. I choose a reference to my main internal domain “anraconc.de” – so my common name is: “anraconc-CA”. (It only looks like an official Internet domain; but actually it is an internal domain, only, and my DNS server is configured accordingly.)

Important hint:
Change the settings for Keylength and Digest by explicitly clicking first on other values and then the real choice again! If you do not change anything explicitly you may get a surprise regarding default values. They may not be what is indicated. Seems to be a bug. Do not disregard this hint if you want to save time ….

Now, a click on the “OK”-button gives us:

We set the keyUsage to “critical” (this certificate extension is used by some applications). And we eventually get all the information about our CA certificate:

The data – and especially the private key – can be found in the directory “/root/.TinyCA/anraconc-CA/”. TinyCA2 creates such a directory for every main CA. (If you use sub-CAs you will find respective directories below it).

myserv:~/.TinyCA # cd anraconc-CA/
myserv:~/.TinyCA/anraconc-CA # la
total 44
drwx------ 7 root root 4096 Jul 20 13:13 .
drwx------ 5 root root 4096 Jul 20 13:12 ..
-rw------- 1 root root 3311 Jul 20 13:13 cacert.key
-rw------- 1 root root 2504 Jul 20 13:13 cacert.pem
drwx------ 2 root root 4096 Jul 20 13:12 certs
drwx------ 2 root root 4096 Jul 20 13:13 crl
-rw------- 1 root root    0 Jul 20 13:12 index.txt
drwx------ 2 root root 4096 Jul 20 13:12 keys
drwx------ 2 root root 4096 Jul 20 13:12 newcerts
-rw------- 1 root root 3872 Jul 20 13:13 openssl.cnf
drwx------ 2 root root 4096 Jul 20 13:12 req
-rw------- 1 root root    2 Jul 20 13:12 serial

Hint: You should make a backup of the CA directories on a periodic basis.

Now, you can export the CA certificate in form of a standard pem-file to some intermediate place where you gather your own certificates and keys – in my case this is a directory “/etc/certs” – which so far survived any Opensuse upgrades. Depending on what else you intend to save there (private keys?), you should make this place accessible to root only! We click on the second to last icon in the icon row of TinyCA2:

Note: In general you have any freedom here to give the exported file any kind of name – whatever you like. However, it is a good policy to use the “common name” which you gave to the CA certificate. See below for the reason.

Place the CA certificate at a central location for trusted CAs

We can now export this certificate file with public information to servers into directories where we gather the public certificates (keys) of all trusted CAs. Of course we need to do this on the server “myserv”, too, as some services may refer to it. In my age my first guess is “/etc/ssl/certs“; old habit form a decade ago where this directory was used more frequently.

myserv:~/.TinyCA/anraconc-CA # cp /etc/certs/anraconc-CA.pem /etc/ssl/certs
myserv:~/.TinyCA/anraconc-CA # chmod 640 /etc/ssl/certs/anraconc-CA.pem 
myserv:~/.TinyCA/anraconc-CA # 

A wrong decision in the end – see below. But for our present session this will work.

Note: If we would create Sub-CAs we would have to export all respective pem files to such a central location – the whole CA-chain must be reflected there for the verification of a service whose “server certificate” has been issued by a sub-CA. I do not use Sub-CAs in this article – but it my be necessary in your organization!

Create a server certificate

Now, we need to create “server certificates” or even service specific certificates. It depends on your policy of how far you want to discriminate services.

In this article I follow the path of a server wide central “server certificate” for all the services implemented there. As examples we shall later have a look at a local OpenLDAP service and a local Apache web server. My central server “myserv” with OpenLDAP has a FQDN of “myserv.anraconc.de”.

Important note: You must use the FQDN as a “common name” in server certificates – consistent with DNS settings. Otherwise you may risk warnings of security aware applications that the server certificate does not fit the server!

in our TinyCA2 window we click on the tab “Certificates” and then on the empty sheet icon:

We fill in the required data. As keys protected by a password may cause trouble for services during automated system startups we try to leave the password-fields empty:

But this approach is not accepted!

So we type in some lengthy password – and set the options for Digest and Algorithm again explicitly – by clicking a bit around first (see above). Then we click on “OK” and get:

Think a bit about the validity! Actually, the length of the validity period should be somewhat shorter than the period for your CA! E.g. 5 years. Otherwise you will get a warning. Eventually:

If you click on the tab “Keys” you will see a related (private) key, too.

Important Note:
We have taken a shortcut here. You could have started in a different way – namely via a certificate “request”. In a first step you would then have issued such a “request” under the tab “Requests” and filled out an initial form there. Afterwards, you explicitly need to sign the requested certificate with the CA’s signature. You get the option for signing by right-clicking on the request entry after its creation. This approach also leads to a valid certificate.

Exporting the server certificate and the key to a central location on the Opensuse system

We need to export the certificate and the (private) key to some
save location on our server “myserv” – only accessible to root (and maybe read accessible to some special system user). On an Opensuse Leap system the location for server certificates should be “/etc/ssl/servercerts“. For exporting the certificate we right-click on the entry:

Then we switch to the tab “Keys” and do the same there:

Important note:
At this point we get an option to export the key without a password. I have chosen this option. This implicates security risks – your exported private key is protected by nothing afterwards. So be very careful where you save it and with which access rights. On the other side such a key will allow for automated service starts – otherwise someone would have to provide the password during startup. I do not want to deepen the discussion here. But be careful with unprotected private keys!

You saw that I exported into my intermediate directory “/etc/certs”. There we change rights for security reasons to:

myserv:/etc/ssl # chmod 600 /etc/certs/myserv-anraconc-key.pem 

Note:
If you instead export your key with the password there is a way to get rid of it afterwards:

myserv:/etc/certs # openssl rsa -in /etc/certs/myserv-anraconc-key.pem -out /etc/certs/myserv-anraconc-key_new.pem
Enter pass phrase for /etc/certs/myserv-anraconc-key.pem:
writing RSA key
myserv:/etc/certs # cp /etc/certs/myserv-anraconc-key_new.pem /etc/certs/myserv-anraconc-key.pem

This creates a key without password! Actually I recommend to use this approach – we do not know details of TinyCA2’s export procedure, but “openssl” will always create the key in the format required for further processing.

Now, before any further steps, I make a backup of everything existing in the folder “/etc/ssl/servercerts“. This is important! If you loose your previous certificates and keys they are gone and you have no chance to get services running with them.

Then we overwrite the existing entries:

 
myserv:/etc/ssl # cp /etc/certs/myserv-anraconc-cert.pem /etc/ssl/servercerts/servercert.pem 
myserv:/etc/ssl # cp /etc/certs/myserv-anraconc-key.pem /etc/ssl/servercerts/serverkey.pem 
myserv:/etc/ssl # chmod 600 /etc/ssl/servercerts/serverkey.pem 

Note: The last step is of fundamental importance due to security reasons! See the discussion below if this leads to trouble for some services and the related system user.

You may try “644” in the beginning to avoid any problems with system users running special services. But if you do this then DO NOT FORGET to restrict the read rights again in the end and after your tests.

Note that replacing the contents of /etc/ssl/severcerts” will probably lead to a breakdown of all services which base their TLS/SSL functionality on these files. In most cases the reason for this will be that the configuration will refer to a wrong CA-certificate. Therefore, you must reconfigure your local services step by step.

Conclusion

Enough for today. We have seen how TinyCA2 can be patched for SHA256, how it can be used to create a CA and server certificates. In the next article

TinyCA2 as a replacement for YaST’s CA-tools on Opensuse Leap servers with TLS/SSL – II

we shall reconfigure an Apache and an LDAP service to work with the new server certificate. And I shall show how we can make the CA-certificate permanently available in “/etc/ssl/certs” – across any server reboot. Stay tuned !

Links

CAs and Certificates – general information
https://wiki.ubuntuusers.de/CA/
stackexchange.com: what-role-do-hashes-play-in-tls-ssl-certificate-validation
stackexchange.com how-does-ssl-tls-work
robpol86.com root certificate authority
https://roll.urown.net/ca/ca_cert.html

TinyCA2
https://github.com/glennie/tinyca2
linux-magazin 2008: eigene zertifikatsstelle mit tinyca

TinyCA2 patches
https://www.systemajik.com/tinyca-sha2/.

Critical
stackexchange.com: which-properties-of-a-x-509-certificate-should-be-critical-and-which-not

 

Opensuse 15 – present libldb1 mismatch with sssd and samba in the Update repository

I updated packages on one of my servers today to the latest versions available in SUSE’s Update repository for Leap 15. Unfortunately this ended with errors whilst starting the sssd-service:

sssd[9042]: ldb: failed to initialise module /usr/lib64/ldb/samba/acl.so : Unavailable

In addition a module mismatch was indicated. After some analysis I found that version 1.2.3.-lp150.7.2 of the required library “libldb1” (of the Update repository) does not work together with the latest versions of sss and samba.

Use “libldb1” version 1.2.3.-lp150.2.3.1 instead (until the SuSE people correct this glitch)!

Mail-server-upgrade to Opensuse Leap 15 – and some hours with authentication trouble …

Some of my readers know that the mail server in my private LAN resides in an encrypted KVM/qemu guest. It provides IMAP- and SMTP-services to mail-clients – and receives emails from several hosted servers on the Internet via a fetchmail system in a DMZ. It was/is my policy that any access to a mail service account on my private mail server requires authentication against user entries on a separate LDAP-server. The accounts the IMAP/SMTP-service-daemons deal with are not to be confused with Linux user accounts: On the mail server NO Linux user accounts are required for mail handling. I regard the absence of standard Linux user accounts on the mail-server as a small, but effective contribution to security. No mail user can login into the mail server as a Linux user. Nevertheless authentication for the use of mail services is required.

Upgrade top Leap 15 resulted in unusable mail services

The manual upgrade of the mail server system from OS Leap 42.3 to Leap 15 seemed to work perfectly. I got no serious warnings. As root I could login to the virtual KVM guest via ssh – and systemd had started all required services: Fetchmail operated on external servers in the DMZ and transferred new mails to the upgraded mail server. There my postfix queues and secondary services for virus and spam checks seemed to work perfectly. Regarding IMAP I could also see that new entries for new mails appeared in various email accounts (more precisely: in related spool directories). Everything looked great …

However, the big surprise came pretty soon: No mail user could login to the IMAP-service – neither with Kmail, nor Mutt, nor Thunderbird. Access to the SMTP service for sending email was not possible either. System messages appeared on the GUIs saying that there was an authentication error. A very unpleasant situation which required analysis ….

In the meantime I had to start a backup copy of the old mailserver guest installation. On this front
virtualization proved its strengths and simplicity – I had made a copy of the whole KVM guest before the upgrade and just had to include the copy as a virtualization domain into the KVM setup of the virtualization host. But, as the upgraded server had already processed some new mails, I needed to transfer these mails between the servers and reconstruct and reindex some IMAP accounts … A simple, but a task which can be done by some scripts ….

Error analysis

It took me quite a while to figure out where the origin of the upgrade problem was located. I first checked my local firewall protocols both on the mail server and on the LDAP server. Nothing special there … Then I checked the LDAP access protocols – there were successful data requests from various systems – but astonishingly not from the mail server. So it seemed that the mail services did not even try to authenticate their users … strange ….
I then created a new IMAP test user based on a new local Linux user account, i.e. with an entry in “/etc/passwd” and “/etc/shadow”. Guess what? This user could work without any problems. So, obviously the communication of the mail services with the LDAP service for authentication was not triggered or failed in a strange way. I, therefore, tried a manual data request to the LDAP server from the mail server – and got a perfect answer. So, there seemed to be no fundamental problem with the required sssd-daemon and client-configurations. What else was wrong? A bit of despair was in the air …

The role of PAM in the game

Then I tried to remember what I had done in the past to separate the email accounts from the Linux user accounts. Over the last years I had actually forgotten how I had treated this problem. After some reading in old diaries my attention eventually turned to PAM …

The PAM-layer offers Linux admins the possibility of fine-tuning access/authentication conditions for the usage
of a service; among other things you can request certain PAM-modules to authenticate a user via given credentials against defined resources. A sequential series of required, sufficient or optional criteria can be set up. ( When a service needs additional Linux user account information during a session NSS can in addition request information from name resolution backends as “/etc/passwd”. However, this is NOT required for mail services. )

Linux mail services often use the SASL-framework and the saslauthd daemon to perform authentication. SASL can communicate with a variety of authentication backends directly – but it can also use PAM as an intermediate layer. And, in fact, I had used PAM to access my independent LDAP-server with mail account credentials via the PAM-module for SSSD.

I have described my simple approach in a previous article
https://linux-blog.anracom.com/2014/03/15/cyrus-imap-mit-sasl-pam-sssd-und-ldap-opensuse-12-313-1-ii/
Unfortunately, only in German. I, therefore, summarize the basic ideas shortly (see also the graphics in the referenced article).

The things one has to do to force IMAP and/or a SMTP services to use LDAP, but other services to use local authentication resources is:

  • We direct local login-services, ssh-services, etc. on the server via PAM to local resources, only.
  • We eliminate any UIDs corresponding to email account names from local resources as “/etc/passwd” and “/etc/shadow” (and NIS if used).
  • We do not create any local home-directories for email account names.
  • We eliminate LDAP as a usable NSS resource from nsswitch.conf; i.e. we eliminate all LDAP references there.
  • We setup special PAM files in “/etc/pam.d” for the IMAP-service and the SMTP-service. These files will necessarily include pam-modules for the “sss“-service (pam_sss.so).
  • We select entries on the LDAP server for users which for any reasons shall NOT or NO LONGER get access to the mail service accounts, set the “host”-attribute for these entries and configure sssd to use the host-attribute. Or deny certain user names access to the IMAP-service by the help of IMAP -configuration files; cyrus e.g offers the maintenance of a list “userdeny_db”.

This strategy worked very conveniently already on an Opensuse 12.3 platform. Examples for the required special files “/etc/pam.d/imap” and “/etc/pam.d/smtp” are given in the article named above. The mix and sequence of the statements there is due to the fact that locally defined system users as “cyrus” must get access to the IMAP-service, too.

What had happened during the Leap 15 upgrade?

My configuration had survived multiple upgrades of the virtualized server in the past. But not the one to Leap 15! What had happened?

After having remembered the importance of the PAM configuration, I eventually had a look into the directory “/etc/pam.d” on the upgraded system and compared it to the respective directory on the original system. Whereas during previous upgrades “personal” PAM configuration files for services had been respected, the upgrade to Leap 15 simply had removed my PAM configuration files for SMTP and IMAP – without a backup! Incredible, but true! Actually and astonishingly, the contents of some other standard PAM files had been transferred to a backup ….

A restoration of my PAM files for the mail services lead to an immediate success – authentication for the access of mail accounts on the upgraded server was possible again. Some hours in the hell for Linux admins came to an end.

Conclusion

Never (!) trust a standard upgrade of a whole Linux system procedure to keep configuration files in “/etc/pam.d” untouched. Keep an admin or system diary for unusual approaches. And – of course: Full
backups of critical systems before backups are a MUST ….

Opensuse Leap, KVM/QEMU-guests, KMS – problems with QXL/Spice after upgrade to Leap 15

Many services in my LAN are provided by virtual KVM/QEMU-guests of a dedicated KVM-server-host. More special services are provided by local KVM-guests on Workstations. All my virtualized systems get normally equipped with a qxl/spice-combination to provide a graphical interface – which can be used on the KVM-host or by remote spice clients. A direct graphical access via a spice client is required only seldomly (besides ssh connections) – but in some cases or situations it is useful, if not mandatory.

Recently, I upgraded both the central KVM-Server, its guests and also guests on workstations from Opensuse Leap 42.3 to Leap 15.0. Unfortunately, after the upgrade no graphical access was possible any longer to my guest-systems via spice-clients (as virt-viewer or the graphical spice-console of virt-manager).

After some tests I found out that this was due to missing KMS on the guest systems – present qxl-modules, however, do require KMS. But, if you update/install from an ISO image “KMS” may not be compatible with the graphical SuSE installer. And due to previous configurations “nomodeset” may be a kernel parameter used in the installation before the upgrade. Here is the story …

The problem: Unreadable, freezing interfaces on spice-clients

Normally, I upgrade by a 5-step sequence: 1) Update all packets, 2) reduce repositories to the Update- and OSS-repository, 3) switch to the repositories of the new distribution, 4) “zypper dup –download-only”, 5) “zypper –no-refresh dup” (see e.g. how-to-upgrade-from-opensuse-leap-422-to-423/).

The KVM server-host itself gave me no major problem during its upgrade. Also the KVM-guests – with their server-services – seemed to work well. Most often, I access the KVM-guest-systems via ssh to perform regular administration tasks. So, I did not even notice for a while that something was wrong with the qxl/spice-configuration. But when I used “virt-manager” in an “ssh -X” session from my workstation to the KVM-server and tried to open the graphical console for a guest there, I got an unreadable virtual screen, which froze instantly and did no longer react to any input – special commands sent via “virt-manager” to switch to a non-graphical console terminal were ignored. The same happened with “virt-viewer” and/or when I executed virt-manager directly on the graphical screen of the KVM-server.

Independent test with a new installation of a “Leap 15”-guest-system

To find out more I tested a new installation of Leap 15 on my Leap 15 workstation as a KVM-server. I chose a guest system configuration with standard equipment – among other things qxl/spice-components. The installation was started via the “virt-installer” component of virt-manager. I used an ISO-image of the Leap 15 installation image.

First thing I stumbled across was that I had to use a “No KMS” for the text console setting on the first screen of the Opensuse installer (see the options for the graphical setup there; F3-key). Otherwise the installer froze during the first “udev” checks. I knew this effect already from installations on some physical systems. Note that the choice of “No KMS” leads to an kernel parameter entry “nomodeset” in the command line for the kernel call in the Grub2 configuration (see the file /etc/default/grub).

Such a full new installation lead into a void. SuSE’s graphical installer itself worked perfectly with high resolution. However, after a restart of the freshly installed guest-system the switch to the graphical screen lead to a flickering virtual screen and the graphical display of the SDDM login manager never appeared.

Still and luckily, it was possible to login as root and execute a “init 3” command – which brought me to a working, non-flickering ASCII console interface (TTY1).

I had experienced this
type of behavior before, too, on some real physical systems. I recommend Leap users to be prepared: activate ssh-services during installation and open the firewall for ssh-ports! The SuSE-installer allows for such settings on its summary screen for the installation configuration. This gives you a chance to switch to a working text console (init 3) from a remote SSH-command line – if the graphical console does not allow for any input.

Tests 2: Upgrade to latest KVM / QEMU / libvirt – versions of the “Virtualization” repository

An installation of the cutting edge versions of KVM/QEMU and spice sever and client software did not change anything – neither on my dedicated KVM-server-system and its upgraded guests nor on my workstation with the fresh Leap 15 test-guest.

Repair of the older upgraded guest-systems

The emulated “hardware” of my older (upgraded) guest-systems, which stem from an OS 13.2 era, is a bit different from the equipment of the newest test guest. On these older systems I still had the choice to use a cirrus, vga or vmwvga graphics. Interestingly, these drivers do not provide the performance of qxl – but they worked at least with the virtual spice client displays.

One central question was whether the QXL driver – more precisely the corresponding kernel module “qxl” – was loaded at all. Answer for the guests on the central KVM-server: No, it was not.

So, on the first guest I simply tried “modprobe qxl” – and hey – the module loaded. An “init 5” then gave me the aspired graphical screen.

Later I checked that actually no “nomodeset” parameter was set in these guests. So, something went “wrong” with the system’s startup-configuration during the upgrade procedure. I have no real clue why – as the qxl-module was loaded without problems on the previous Leap 42.3 installations.

For Opensuse Leap the wrapper-script “mkinitrd” transfers a running configuration with its loaded kernel modules via “dracut” into a suitable and permanent initramfs/initrd- and systemd-startup configuration. So, issue “mkinitrd” after a successful test of a qxl/spice-interface.

Repair of the freshly installed Leap 15 guest-system

On the freshly installed Leap 15 guest client on my workstation things were slightly different: The qxl-module did not load there. A “modinfo qxl” shows you parameters you can apply. The right thing to do there was to try

modprobe qxl modeset=1

This worked! Then I eliminated the “nomodeset”-parameter from the “GRUB_CMDLINE_LINUX_DEFAULT”-entry in the file “/etc/default/grub” and started “mkinitrd” to get a stable permanent startup-configuration (via dracut) which booted the guest into graphical mode afterwards.

Adjustment of the TTYs and KDE

As soon as you have a reasonable configuration, you may want to adjust the screen dimensions of the text consoles tty1 to tty6, the sddm-login-screen and the KDE or Gnome screen. These are topics for separate articles. See, however, previous articles in this blog on KVM guests for some hints.

For the text console TTYs try reasonable settings for the following entries in the “/etc/default/grub” – below e.g. for a resolution of “1680×1050”:

GRUB_CMDLINE_LINUX_DEFAULT=” …. OTHER PARAMETERS …… video=1680×1050″
GRUB_GFXMODE=”1680×1050″
GRUB_GFXPAYLOAD=keep

(see https://www.suse.com/de-de/support/kb/doc/?id=7017979)

Do not forget to execute “mkinitrd” again afterwards!

For KDE adjustments a user can use the command “systemsettings5” and then specify screen resolutions in the dialog for “display and monitors”.

Conclusion

The graphical installer of Opensuse, but also upgrade-procedures on working virtual KVM/QEMU guests with qxl/spice can lead to situations where KMS is or
must be deactivated both for physical and virtual systems. As a consequence the “qxl-module” may not be loadable automatically afterwards. This can lead to failures during the start of a graphical qxl/spice-interface for local and remote spice-clients.

The remedy is to remove any “nomodeset”-parameter which may be a part of the entry for “GRUB_CMDLINE_LINUX_DEFAULT” in the file “/etc/default/grub”.

For tests of the qxl-driver try loading the qxl-module with “modprobe qxl modeset=1”. After a successful start of a graphical interface use the command “mkinitrd” (whilst the qxl-module is loaded!) to establish a permanent configuration which loads “qxl” during system start.

Installation Opensuse Leap 15 auf Laptop – Grafik Probleme, Optimus

Einer meiner Laptops ist in die Jahre gekommen (letzte BIOS-Generation; Optimus-System). Auf diesem System stand eine Neu-Installation von Opensuse Leap 15.0 an (kein Upgrade). Interessant ist dann immer wieder das Verhalten der Grafikarten – hier einer Nvidia und einer im Prozessor integrierten Intel-Karte. Schon früher gab es im Zuge von Upgrades immer wieder Probleme, die Bumblebee-Unterstützung zum Laufen zu bringen. Wegen der Neuinstallation musste ich die ersten Hürden diesmal schon zu Beginn der Installation überwinden. Ohne zusätzliche Maßnahmen lief danach auch der i915-Treiber für die in den Prozessor integrierte Intel-Grafik-Karte nicht so, wie ich das erwartet hätte. Bei der Bumblebee-Installation gab es ferner eine kleine, aber wichtige Neuerung im Zusammenhang mit bbswitch. Ohne deren Beachtung konnte ich optirun/primusrun für 3D-Anwendungen nicht zum Laufen bringen.

Ich stelle nachfolgend die wichtigsten Schritte für einen erfolgreichen Grafik-Setup auf einem solchen System zusammen.

“No KMS” bei den Vorgaben für die Textkonsole(n) am Startschirm für den SuSE-Installer

SuSE bietet auf dem Startschirm der Installation etliche Optionen für die Grafik-Einstellungen an (F3-Taste). Während ich beim ersten und letzten Punkt des aufklappenden Menüs maximale Auflösungen wählen konnte, musste ich bei der Textkonsole zwingend “Keine KMS” wählen.

Installer Size => 1920×1200
Text Console Size => Keine KMS (ggf. zwingend auf Optimus-System)
Video BIOS Size => 1920 x 1080

Tat ich das nicht, starb das System bei der HW-Erkennung mit udev (Stage 6/6 nach dem Laden des Kernels); der Installer landete im Nirwana (schwarzer Schirm mit Cursor) und weigerte sich, seine Arbeit fortzusetzen. Über die genaue Ursache des Problems mag ich hier nicht spekulieren (Optimus => Zugriff auf Nvidia-Karte ohne hinreichenden Treiber?). Siehe bei Interesse zu KMS und dem Suse-Installer die Links am Ende des Beitrags.

Durch Abschalten von KMS im Installer wird dem Kernel der Parameter “nomodeset” übergeben. Danach lief bei mir der grafische Installer in der gewünschten hohen Auflösung. Der Parameter “nomodeset” wird im Zuge der Installation in der Grub2-Konfiguration verankert (s. die Datei “/etc/default/grub” und die resultierenden Einträge in der “/boot/grub2/grub.cfg”).

Desktop-SW KDE Plasma 5 über Pattern installieren und Nouveau-Treiber blacklisten

Um die Kontrolle über die Installation zu behalten, empfiehlt es sich, schrittweise vorzugehen und nicht sofort einen grafischen Desktop zu installieren. Erledigen kann man das im Installer bzw. nach der Installation in YaST durch Abwahl bzw. Auswahl bestimmter Paketgruppen und “Schemata“, von denen SuSE etliche anbietet. Im Installer wählt man geeignete Pakte für eine kleinere Installation aus – z.B. für eine einfache Server-Basis-Installation. Bootet unser System danach erfolgreich in den Multiuser-Modus (mit 6 aktiven Textkonsolen; Wechsel mit Ctrl-Alt-F1 bis Ctrl-Alt-F6), so können wir z.B. einen KDE-Desktop unter YaSTs “Software Management” mittels des Schemas “KDE Plasma5 Desktop Environment” installieren.

Ich gehe dabei gemäß folgender Schritte vor:

1) Installieren des KDE 5 Plasma Schemas
Installieren des KDE Plasma Schemas über YaST >> “Software installieren und löschen” >> Anzeigen >> Schemata >> “KDE Plasma 5 Desktop Environment”.
Wir prüfen, dass das Paket “Mesa-dri-nouveau” nicht installiert wurde. Wenn doch, deinstallieren wir es.

2) Nouveau-Treiber “blacklisten
Öffnen der Datei “/etc/modprobe.d/50-blacklist.conf“. Hinzufügen folgender Zeilen am Ende (ggf. mit erläuternden Kommentaren):

blacklist nouveau
options
nouveau modeset=0

3) mkinitrd
Ausführen von “sudo mkinitrd” an einem Terminal. Dies ist notwendig, um evtl. nouveau-Treiber-Komponenten auch aus dem “initramfs” zu entfernen.

4) Reboot und Wechsel in das “graphical.target” von systemd
Nach einem Reboot sollte es möglich sein, in den Grafikmodus zu wechseln; dort wird einem der grafische Login-Schirm (bei KDE: SDDM) angeboten. Landet man am Ende des Hochfahrens nicht direkt auf dem grafischen Terminal (Ctrl-Alt-F7), weil das systemd “default.target” noch das “multiuser.target” ist, muss man auf einem Konsolen-Terminal als User root “init 5” absetzen. Später, wenn KDE Plasma korrekt läuft, kann man dann z.B. über Yasts “Services Manager” das “default.target” von systemd ändern. (Wer das lieber auf der Kommandozeile macht, sollte sich mit den Kommandos “systemctl get-default” und “systemctl set-default” befassen.)

Das Arbeiten mit dem grafischen Desktop ging dann in meinem Fall leider überhaupt nicht flüssig; man merkt das bereits am trägen Aufbau des Login-Schirms – vor allem aber nach einem Login und der Darstellung des KDE-Desktops. Es fehlt die 2D-Grafik-Beschleunigung, da der i915-Treiber für die in den Prozessor integrierte Graka noch nicht richtig funktioniert. Der erfordert offenbar KMS, das wir ja aber bei der Installation abschalten mussten (s.o.). Das können wir nun korrigieren.

Kernelparameter “nomodeset” entfernen, mkinitrd, Reboot und Laden des i915-Moduls prüfen

Wir unterbinden jetzt die Übergabe des Parameters “nomodeset” an den Kernel über die Grub2-Konfiguration. Dazu editieren wir die Datei “/etc/default/grub” und entfernen dort den Wert “nomodeset” im Eintrag für “GRUB_CMDLINE_LINUX_DEFAULT”. Der Eintrag sieht danach etwa wie folgt aus:

GRUB_CMDLINE_LINUX_DEFAULT=’splash=silent quiet showopts resume=/dev/mapper/cr-swap’

Um das wirksam zu machen, setzen wir erneut “mkinitrd” ab und initiieren dann einen Reboot-Vorgang (init 6). Bevor wir uns im Anschluss am SDDM-Login-Schirm einloggen, überprüfen wir an einem Konsolen-Terminal, ob das i915-Modul ordnungsgemäß geladen ist; das sollte dann etwa so aussehen:

 
mytux:~ # lsmod | grep "i915\|video"
i915                 1953792  7
i2c_algo_bit           16384  1 i915
drm_kms_helper        200704  1 i915
drm                   438272  5 i915,drm_kms_helper
video                  45056  2 msi_wmi,i915

Am grafischen Terminal (Ctrl-Alt-F7) ist dann ein erster Zwischenerfolg zu vermelden: Das Arbeiten mit dem Desktop geht nun flüssig. Für 2D-beschleunigtes KDE reicht die integrierte Intel Graka völlig aus 🙂 .

Bumblebee installieren

Wir wollen nun die Optimus-Konfiguration für 3D-Anwendungen nutzen. Wie schon früher für Leap 42.3 beschrieben (s.
Upgrade Laptop to Opensuse 42.3, Probleme mit Bumblebee und VMware WS 12.5, Workarounds )
gilt, dass man nur das Repository unter “http://download.opensuse.org/repositories/X11:/” für seine Distribution nutzen sollte:

https://download.opensuse.org/repositories/X11:/Bumblebee/openSUSE_Leap_15.0/

Sonst gar nichts! Das Nvidia-Community-Repository ist normalerweise nicht erforderlich.

Im Gegensatz zu Leap 42.3 gehen wir mit folgenden Schritten vor, nachdem wir das Bumblebee-Repository unter YaSTs “Software Management” eingebunden haben:

Schritt 1 – Paketinstallation: Wir installieren die folgenden Pakete:

VirtualGL, bumblebee, dkms, bbswitch, bbswitch-kmp-default

Achtet bitte darauf, dass ihr die zueinander passenden Versionen aus dem Bumblebee-
Repository (und nicht aus dem Leap-15-Update Repository) installiert. Nützlich ist auch das Paket “Mesa-demo-x”, das die Testprogramme “glxgears” und “glxspheres” als Testprogramme beinhaltet. Wir prüfen zudem, dass die Pakete “x11-tools” und “xf86-video-nv” installiert sind. Wenn nicht: Installieren!

Schritt 2 – Aktivierung dkms: Wir aktivieren nun den “dkms”-Service mittels der Kommandos “systemctl enable dkms; systemctl start dkms”. Im Gegensatz zur Leap 42.3-Installation schadet dieser Service diesmal nicht 🙂 ; er garantiert später vielmehr die Neukompilation der Module bei Kerneländerungen oder Modul-Reinstallationen.

Schritt 3 – Gruppe “bumblebee” anlegen: Dann legen wir (z.B. mit YaSTs “User and Group Management”) eine User-Gruppe “bumblebee” an. Bei mir geschah dies bei der Bumblebee-Installation leider nicht automatisch. Dieser Gruppe ordnen wir dann diejenigen User zu, die später “optirun” und “primusrun” für den Start von 3D-Applikationen verwenden dürfen (ggf. auch root).

Schritt 4 – Wichtige Modifikation der Datei “etc/modprobe.d/50-bbswitch.conf”
Ein entscheidender Schritt ist nun, dass wir die Datei “etc/modprobe.d/50-bbswitch.conf” editieren; sie sollte einen Eintrag von genau der Form

options bbswitch load_state=-1 unload_state=1

beinhalten. Wichtig ist das “-1” bei load_state !!

Schritt 5 – Aktivieren des bumblebee-Services

systemctl enable bumblebeed.service
systemctl start bumblebeed.service

Das “d” am Ende von “bumblebeed” ist wichtig; es handelt sich um einen Dämon.

Wir prüfen dann, ob das bbswitch-Module ordnungsgemäß erzeugt wurde. Ein “find / -name “*bbswitch*” sollte etwa Folgendes zeigen:

 
mytux:~ # find / -name "*bbswitch*"
/usr/share/doc/packages/bbswitch
/usr/lib/modules-load.d/bbswitch.conf
/sys/module/bbswitch
/etc/modprobe.d/50-bbswitch.conf
/etc/modprobe.d/50-bbswitch.conf.rpmsave
/proc/acpi/bbswitch
/lib/modules/4.12.14-lp150.11-default/updates/bbswitch.ko
/lib/modules/4.12.14-lp150.12.28-default/weak-updates/updates/bbswitch.ko

“bbswitch.ko” sollte für alle installierten Kernelvarianten vorhanden sein; zwingend aber für den aktuell verwendeten Kernel. Treten hier Fehler auf, so hat dies ggf. damit zu tun, dass schon früher einmal nvidia-Module (über andere Repositories) erzeugt wurden. Die entsprechenden RPM-Pakete sind zu löschen; danach muss man manuell nach nvidia-Kernel-Modulen suchen und diese ebenfalls manuell löschen. Anschließend sollte man die bbswitch-Pakte reinstallieren.

Wir probieren dann aus, ob sich das erzeugte bbswitch-Modul mittels

modeprobe bbswitch

laden lässt. Dies sollte fehlerfrei möglich sein.

Nvidia-Treiber installieren

Nun installieren wir die Pakete “nvidia-bumblebee, nvidia-bumblebee-32bit”. Dies sollte eine Weile dauern, da die Treiber runtergeladen und kompiliert werden müssen. Am Ende dieser Installation sollte ein “find / -name “*nvidia*” etwa Folgendes zeigen; der Output ist länglich, aber für einen Vergleich mit eurem eigenen System geeignet:

 
mytux:~ #
/usr/lib64/libnvidia-cfg.so.1
/usr/lib64/libnvidia-opencl.so.1
/usr/lib64/libnvidia-opencl.so.410.93
/usr/lib64/libnvidia-ml.so.1
/usr/lib64/libnvidia-glvkspirv.so
/usr/lib64/nvidia
/usr/lib64/nvidia/libnvidia-gtk3.so.410.93
/usr/lib64/nvidia/xorg/modules/extensions/libglxserver_nvidia.so
/usr/lib64/nvidia/xorg/modules/extensions/libglxserver_nvidia.so.410.93
/usr/lib64/libnvidia-compiler.so.410.93
/usr/lib64/vdpau/libvdpau_nvidia.so.1
/usr/lib64/vdpau/libvdpau_nvidia.so.410.93
/usr/lib64/libnvidia-
ptxjitcompiler.so.410.93
/usr/lib64/libnvidia-opencl.so
/usr/lib64/libnvidia-ml.so.410.93
/usr/lib64/libnvidia-tls.so.410.93
/usr/lib64/libnvidia-glvkspirv.so.410.93
/usr/lib64/xorg/modules/drivers/nvidia_drv.so
/usr/lib64/libnvidia-ptxjitcompiler.so.1
/usr/lib64/libnvidia-ml.so
/usr/lib64/libnvidia-fatbinaryloader.so.410.93
/usr/lib64/libnvidia-cfg.so.410.93
/usr/lib64/libnvidia-glcore.so.410.93
/usr/bin/nvidia-settings
/usr/bin/nvidia-smi
/usr/bin/nvidia-bug-report.sh
/usr/bin/nvidia-xconfig
/usr/share/pixmaps/nvidia-settings.png
/usr/share/nvidia
/usr/share/nvidia/nvidia-application-profiles-410.93-key-documentation
/usr/share/doc/nvidia-utils
/usr/share/doc/nvidia
/usr/share/doc/nvidia/nvidia                                                                                                                                        
/usr/share/doc/packages/nvidia-bumblebee                                                                                                                            
/usr/share/doc/packages/nvidia-bumblebee/README.nvidia-bumblebee                                                                                                    
/usr/share/doc/packages/nvidia-bumblebee-32bit                                                                                                                      
/usr/share/doc/packages/nvidia-bumblebee-32bit/README.nvidia-bumblebee-32bit
/usr/share/applications/nvidia-settings.desktop
/usr/share/vulkan/icd.d/nvidia_icd.x86_64.json
/usr/share/vulkan/icd.d/nvidia_icd.i586.json
/usr/share/man/man1/nvidia-xconfig.1.gz
/usr/share/man/man1/nvidia-settings.1.gz
/usr/share/man/man1/nvidia-smi.1.gz
/usr/share/licenses/nvidia-utils
/usr/share/licenses/kernel-firmware/LICENCE.nvidia
/usr/share/licenses/nvidia
/usr/share/licenses/nvidia/nvidia
/usr/src/linux-4.12.14-lp150.12.28/arch/arm64/boot/dts/nvidia
/usr/src/linux-4.12.14-lp150.12.28/drivers/video/fbdev/nvidia
/usr/src/linux-4.12.14-lp150.12.28/drivers/video/fbdev/nvidia/nvidia.c
/usr/src/linux-4.12.14-lp150.12.28/drivers/char/agp/nvidia-agp.c
/usr/src/linux-4.12.14-lp150.12.28/drivers/net/ethernet/nvidia
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/i2c/nvidia,tegra186-bpmp-i2c.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/i2c/nvidia,tegra20-i2c.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/iommu/nvidia,tegra20-gart.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/iommu/nvidia,tegra30-smmu.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/fuse/nvidia,tegra20-fuse.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra-audio-wm9712.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra-audio-sgtl5000.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra-audio-rt5677.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra20-das.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra30-i2s.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra30-hda.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra-audio-alc5632.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra20-ac97.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra30-ahub.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra20-i2s.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra-audio-wm8903.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra-audio-trimslice.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/
bindings/sound/nvidia,tegra-audio-wm8753.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra-audio-rt5640.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/sound/nvidia,tegra-audio-max98090.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/misc/nvidia,tegra20-apbmisc.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/cpufreq/nvidia,tegra124-cpufreq.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/usb/nvidia,tegra124-xusb.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/usb/nvidia,tegra20-ehci.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/pwm/nvidia,tegra20-pwm.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/rtc/nvidia,tegra20-rtc.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/pinctrl/nvidia,tegra210-pinmux.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/pinctrl/nvidia,tegra20-pinmux.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/pinctrl/nvidia,tegra124-dpaux-padctl.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/pinctrl/nvidia,tegra124-pinmux.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/pinctrl/nvidia,tegra124-xusb-padctl.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/pinctrl/nvidia,tegra30-pinmux.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/pinctrl/nvidia,tegra114-pinmux.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/thermal/nvidia,tegra124-soctherm.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/timer/nvidia,tegra20-timer.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/timer/nvidia,tegra30-timer.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/ata/nvidia,tegra124-ahci.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/gpio/nvidia,tegra186-gpio.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/gpio/nvidia,tegra20-gpio.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/serial/nvidia,tegra20-hsuart.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/dma/nvidia,tegra20-apbdma.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/mmc/nvidia,tegra20-sdhci.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/pci/nvidia,tegra20-pcie.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/phy/nvidia,tegra20-usb-phy.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/phy/nvidia,tegra124-xusb-padctl.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/gpu/nvidia,gk20a.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/bus/nvidia,tegra210-aconnect.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/bus/nvidia,tegra20-gmi.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/spi/nvidia,tegra114-spi.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/spi/nvidia,tegra20-slink.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/spi/nvidia,tegra20-sflash.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/display/tegra/nvidia,tegra114-mipi.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/display/tegra/nvidia,tegra20-host1x.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/interrupt-controller/nvidia,tegra20-ictlr.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/input/nvidia,tegra20-kbc.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/mailbox/
nvidia,tegra186-hsp.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/memory-controllers/nvidia,tegra30-mc.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/memory-controllers/nvidia,tegra124-emc.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/firmware/nvidia,tegra186-bpmp.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/clock/nvidia,tegra124-car.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/clock/nvidia,tegra114-car.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/clock/nvidia,tegra210-car.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/clock/nvidia,tegra124-dfll.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/clock/nvidia,tegra20-car.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/clock/nvidia,tegra30-car.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra30-actmon.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra30-mc.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra20-mc.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra20-emc.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra20-flowctrl.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/arm/tegra/nvidia,nvec.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra186-pmc.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra20-pmc.txt
/usr/src/linux-4.12.14-lp150.12.28/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra20-ahb.txt
/usr/src/nvidia-410.93
/usr/src/nvidia-410.93/nvidia
/usr/src/nvidia-410.93/nvidia/nvidia.Kbuild
/usr/src/nvidia-410.93/nvidia/nvidia-sources.Kbuild
/usr/src/nvidia-410.93/nvidia-modeset
/usr/src/nvidia-410.93/nvidia-modeset/nvidia-modeset-linux.c
/usr/src/nvidia-410.93/nvidia-modeset/nvidia-modeset-os-interface.h
/usr/src/nvidia-410.93/nvidia-modeset/nvidia-modeset.Kbuild
/usr/src/nvidia-410.93/nvidia-drm
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-encoder.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-priv.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm.Kbuild
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-fb.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-drv.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-gem-nvkms-memory.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-gem-user-memory.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-connector.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-helper.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-prime-fence.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-gem-nvkms-memory.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-gem.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-gem.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-os-interface.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-utils.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-modeset.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-encoder.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-modeset.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-helper.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-ioctl.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-connector.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-utils.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-fb.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-crtc.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-conftest.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-dma-fence-helper.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-linux.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-crtc.h
/usr/src/nvidia-
410.93/nvidia-drm/nvidia-drm-gem-user-memory.h
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-drv.c
/usr/src/nvidia-410.93/nvidia-drm/nvidia-drm-prime-fence.h
/usr/src/nvidia-410.93/nvidia-uvm
/usr/src/nvidia-410.93/nvidia-uvm/nvidia-uvm-sources.Kbuild
/usr/src/nvidia-410.93/nvidia-uvm/nvidia-uvm.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia_icd.x86_64.json
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-settings
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-smi
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-xconfig.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia_drv.so
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia.icd
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-modprobe
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-installer.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-settings.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia_icd.json.template
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-fbc.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/nvidia.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/nvidia-sources.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-modeset
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-modeset/nvidia-modeset-linux.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-modeset/nvidia-modeset-os-interface.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-modeset/nvidia-modeset.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-encoder.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-priv.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-fb.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-drv.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem-nvkms-memory.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem-user-memory.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-connector.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-helper.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-prime-fence.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem-nvkms-memory.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-os-interface.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-utils.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-modeset.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-encoder.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-modeset.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-helper.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-ioctl.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-connector.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-utils.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-fb.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-crtc.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-conftest.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-dma-fence-helper.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-linux.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-crtc.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem-user-memory.h
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-drv.c
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-prime-fence.h
/usr/
src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-uvm
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-uvm/nvidia-uvm-sources.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-uvm/nvidia-uvm.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/libglxserver_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/10_nvidia_wayland.json
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-opencl.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-glsi.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-settings
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-smi
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-xconfig.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia_drv.so
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia.icd
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-modprobe
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-installer.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-settings.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia_icd.json.template
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-fbc.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/nvidia.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/nvidia-sources.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-modeset
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-modeset/nvidia-modeset-linux.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-modeset/nvidia-modeset-os-interface.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-modeset/nvidia-modeset.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-encoder.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-priv.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-fb.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-drv.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem-nvkms-memory.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem-user-memory.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-connector.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-helper.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-prime-fence.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem-nvkms-memory.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-os-interface.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-utils.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-modeset.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-encoder.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-modeset.c
n/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-helper.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-ioctl.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-connector.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-utils.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-fb.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-crtc.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-conftest.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-dma-fence-helper.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-linux.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-crtc.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-gem-user-memory.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-drv.c
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-drm/nvidia-drm-prime-fence.h
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-uvm
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-uvm/nvidia-uvm-sources.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/kernel/nvidia-uvm/nvidia-uvm.Kbuild
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libglxserver_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/10_nvidia_wayland.json
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-opencl.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-glsi.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-modprobe.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-bug-report.sh
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-cuda-mps-server
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-debugdump
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-application-profiles-410.93-rc
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-persistenced
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-fbc.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-opencl.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-glsi.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-compiler.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libvdpau_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-ptxjitcompiler.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-eglcore.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libGLESv2_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libGLESv1_CM_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-ml.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-tls.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-glvkspirv.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libGLX_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libEGL_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-ifr.so.410.93
n/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-fatbinaryloader.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-encode.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/tls/libnvidia-tls.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/32/libnvidia-glcore.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-installer
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-rtcore.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-compiler.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-gtk3.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-gtk2.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libvdpau_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-wfb.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-ptxjitcompiler.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-eglcore.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-cbl.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libGLESv2_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-persistenced.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-persistenced-init.tar.bz2
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-settings.desktop
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libGLESv1_CM_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-cuda-mps-control.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-application-profiles-410.93-key-documentation
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-ml.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-tls.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-glvkspirv.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libGLX_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/html/nvidia-ml.html
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/html/nvidia-debugdump.html
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/html/nvidiasettings.html
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/html/nvidia-smi.html
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/html/nvidia-persistenced.html
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-smi.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libEGL_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-settings.png
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-cuda-mps-control
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-xconfig
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-ifr.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-fatbinaryloader.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/10_nvidia.json
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-cfg.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-encode.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/tls/libnvidia-tls.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-glcore.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/libnvidia-egl-wayland.so.1.1.0
/usr/src/NVIDIA-Linux-x86_64-410.93/NVIDIA-Linux-x86_64-410.93/nvidia-drm-outputclass.conf
/usr/src/NVIDIA-
Linux-x86_64-410.93/nvidia-modprobe.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-bug-report.sh
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-cuda-mps-server
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-debugdump
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-application-profiles-410.93-rc
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-persistenced
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-fbc.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-opencl.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-glsi.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-compiler.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libvdpau_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-ptxjitcompiler.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-eglcore.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libGLESv2_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libGLESv1_CM_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-ml.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-tls.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-glvkspirv.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libGLX_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libEGL_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-ifr.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-fatbinaryloader.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-encode.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/tls/libnvidia-tls.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/32/libnvidia-glcore.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-installer
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-rtcore.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-compiler.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-gtk3.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-gtk2.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libvdpau_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-wfb.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-ptxjitcompiler.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-eglcore.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-cbl.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libGLESv2_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-persistenced.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-persistenced-init.tar.bz2
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-settings.desktop
/usr/src/NVIDIA-Linux-x86_64-410.93/libGLESv1_CM_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-cuda-mps-control.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-application-profiles-410.93-key-documentation
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-ml.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-tls.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-glvkspirv.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libGLX_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/html/nvidia-ml.html
/usr/src/NVIDIA-Linux-x86_64-410.93/html/nvidia-debugdump.html
/usr/src/NVIDIA-Linux-x86_64-410.93/html/nvidiasettings.html
/usr/src/NVIDIA-Linux-x86_64-410.93/html/nvidia-smi.html
/usr/src/NVIDIA-Linux-x86_64-410.93/html/nvidia-persistenced.html
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-smi.1.gz
/usr/src/NVIDIA-Linux-x86_64-410.93/libEGL_nvidia.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-settings.png
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-cuda-mps-control
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-xconfig
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-ifr.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-fatbinaryloader.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/10_nvidia.json
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-cfg.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia_icd.i586.json
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-encode.so.410.93
/usr/src/NVIDIA-Linux-x86_
64-410.93/tls/libnvidia-tls.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-glcore.so.410.93
/usr/src/NVIDIA-Linux-x86_64-410.93/libnvidia-egl-wayland.so.1.1.0
/usr/src/NVIDIA-Linux-x86_64-410.93/nvidia-drm-outputclass.conf
/usr/src/kernel-modules/nvidia-340.107-default
/usr/src/kernel-modules/nvidia-uvm-340.107-default
/usr/src/kernel-modules/nvidia-uvm-340.107-default/rm/nvidia.mod.c
/usr/src/kernel-modules/nvidia-uvm-340.107-default/rm/.nvidia.ko.cmd
/usr/src/kernel-modules/nvidia-uvm-340.107-default/rm/.nvidia.mod.o.cmd
/usr/src/kernel-modules/nvidia-uvm-340.107-default/rm/.tmp_versions/nvidia.mod
/usr/src/kernel-modules/nvidia-uvm-340.107-default/rm/.nvidia.o.cmd
/usr/src/kernel-modules/nvidia-uvm-340.107-default/rm/nvidia.mod.o
/usr/src/kernel-modules/nvidia-uvm-340.107-default/rm/nvidia.ko
/usr/src/kernel-modules/nvidia-uvm-340.107-default/rm/nvidia.o
/usr/src/kernel-modules/nvidia-390.87-default
/usr/src/linux-4.12.14-lp150.12.28-obj/x86_64/default/include/config/net/vendor/nvidia.h
/usr/lib/libnvidia-opencl.so.1
/usr/lib/libnvidia-opencl.so.410.93
/usr/lib/libnvidia-ml.so.1
/usr/lib/libnvidia-glvkspirv.so
/usr/lib/nvidia
/usr/lib/libnvidia-compiler.so.410.93
/usr/lib/vdpau/libvdpau_nvidia.so.410.93
/usr/lib/libnvidia-ptxjitcompiler.so.410.93
/usr/lib/udev/rules.d/99-bumblebee-nvidia-dev.rules
/usr/lib/libnvidia-ml.so.410.93
/usr/lib/libnvidia-tls.so.410.93
/usr/lib/libnvidia-glvkspirv.so.410.93
/usr/lib/libnvidia-ptxjitcompiler.so.1
/usr/lib/libnvidia-fatbinaryloader.so.410.93
/usr/lib/libnvidia-glcore.so.410.93
/etc/zypp/repos.d/download.nvidia.com-leap.repo
/etc/modprobe.d/50-nvidia-default.conf.rpmsave
/etc/bumblebee/xorg.conf.nvidia
/etc/apparmor.d/abstractions/nvidia
/etc/OpenCL/vendors/nvidia.icd
/lib/firmware/nvidia
/lib/firmware/LICENCE.nvidia
/lib/modules/4.12.14-lp150.11-default/kernel/drivers/net/ethernet/nvidia
/lib/modules/4.12.14-lp150.12.28-default/kernel/drivers/net/ethernet/nvidia
/lib/modules/4.12.14-lp150.12.28-default/updates/nvidia-modeset.ko
/lib/modules/4.12.14-lp150.12.28-default/updates/nvidia-drm.ko
/lib/modules/4.12.14-lp150.12.28-default/updates/nvidia-uvm.ko
/lib/modules/4.12.14-lp150.12.28-default/updates/nvidia.ko
/var/cache/zypp/solv/download.nvidia.com-leap
/var/cache/zypp/packages/download.nvidia.com-leap
/var/cache/zypp/raw/download.nvidia.com-leap
/var/lib/dkms/nvidia
/var/lib/dkms/nvidia/410.93/4.12.14-lp150.12.28-default/x86_64/module/nvidia-modeset.ko
/var/lib/dkms/nvidia/410.93/4.12.14-lp150.12.28-default/x86_64/module/nvidia-drm.ko
/var/lib/dkms/nvidia/410.93/4.12.14-lp150.12.28-default/x86_64/module/nvidia-uvm.ko
/var/lib/dkms/nvidia/410.93/4.12.14-lp150.12.28-default/x86_64/module/nvidia.ko

Am wichtigsten sind die Einträge für die Module (Endung “.ko”) unter “lib/modules/EURE_AKTUELLE_KERNEL_VERSION”. Einige dieser Module sind für CUDA und die Nutzung der Grafikkarte als schnelles Rechenwerkzeug relevant.

Ihr seht an meinem Output u.a. auch, dass der Treiber die Version 410.93 hat. Der funktioniert für Geforce-Karten (ab Geforce 400). Diese Version wird sich mit künftigen Updates aus dem Bumblebee-Repository ändern. Ihr müsst deshalb vorab prüfen, ob eure Nvidia-Karte dafür geeignet ist (z.B. über die Nvidia-Web-Seite und die dortigen Treiber-Beschreibungen). Eine Beschreibung für den Fall, dass ihr einen älteren Treiber benötigt, erspare ich mir. Ich habe im Moment leider keine Zeit für entsprechende Tests.

In jedem Fall probieren wir nun aus, ob sich das nvidia-Modul laden lässt: Das Kommando “modprobe nvidia” sollte sich anstandslos durchführen lassen.

Erneutes mkinitrd

Nach einem Reboot wird die nvidia-Karte womöglich angeschaltet und das nvidia-Modul bereits geladen (Check Kommando “lsmod | grep “*nvidia*\|video”). Es gibt nun unterschiedliche Möglichkeiten, ein Booten ohne nvidia-Modul zu ermöglichen. Die
einfachste Möglichkeit besteht in folgenden Schritten:

Wir entladen folgende Module gem. ihrer Abhängigkeiten mittels “rmmod”:

nvidia_drm, nvidia_uvm, nvidia_modeset, nvidia

Wir führen dann testweise folgendes Kommandos durch, um die nvidia-Karte abzuschalten:

tee /proc/acpi/bbswitch >>>OFF

Auf meinem Laptop ändert sich dabei die Farbe des Power-Buttons von Rot auf Blau; vielleicht gibt es bei euch eine ähnliche Reaktion/anzeige.

Im Anschluss führen wir erneut “mkinitrd” durch.

Tests 3D-Beschleunigung

Nach einem Reboot sollte das nvidia-Kernel-Modul nicht geladen sein. Wir loggen uns in unseren KDE-Desktop ein. An einem Terminal-Fenster probieren wir “modprobe nvidia; lsmod | grep “nvidia\|video”. Wer will, kann das Modul dann auch wieder entladen.

An einem Terminalfenster setzen wir nun das Kommando “optirun glxspheres” ab. Wir sollten in einem sich öffnenden Fenster eine schnelle Abfolge der konzentrische Kreise aus 3D-Elementen bekommen. Die Rate sollte > 200 Mpixel/sec und > 200 Frames/sec ausweisen. Bei mir sind es annähernd 250 frames/sec (GT 645M).

Links

wiki.archlinux.org/index.php/kernel _mode _setting
https://fedoraproject.org/wiki/Features/KernelModesetting
https://de.wikipedia.org/wiki/Mode-Setting
https://doc.opensuse.org/documentation/ leap/startup/html/book.opensuse.startup/cha.boot_parameters.html
https://forums.opensuse.org/showthread.php/508996-Install-with-UEFI-Bios-and-no-KMS-support