Category Archives: Opensuse

Opensuse Leap, dolphin and abysmal slow data transfer between two external USB 3 disks

Posted on by

Today I had to copy a set of multiple directories, each with around 600 sub-directories and a total of 420,000 singular files, between two external USB-3 hard disks. Both the USB controllers of the mainboard and the controllers of the external disks were USB-3 capable. My PC has 64 GB RAM. Which in this case was of no help. I got the problem on an Opensuse Leap 15.3-system, but you may find it on other Linux-OS, too.

The problem

First I started to copy with the help of KDE’s Dolphin. The transfer rates were really bad – something about 5 MB/s. Would have had to wait for hours.

Then I tried with just a standard cp-command. The rates rose to something like 20 MB/s to 25 MB/s. But this was not a reasonable transfer rate either.

A workaround

A search on the Internet then gave me the following two commands

mytux:~ # echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes
mytux:~ # echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes

Note that setting values on these system parameters automatically sets the standard values of 

mytux:~ # cat /proc/sys/vm/dirty_ratio
20
mytux:~ # cat /proc/sys/vm/dirty_background_ratio
10

to zero! And vice versa. You either have to use dirty_bytes and dirty_background_bytes OR the related ratio-parameters!

The effect of the above settings was remarkable: The total transfer rate (i.e. reading from one disk and writing to another; both on the same controller) rose to 130 MB/s. Still far away from theoretical rates. But something I could live with.
It would be worth to experiment more with the size of the parameters, but I was satisfied regarding my specific problem.

Conclusion

An efficient data transfer on Linux systems between two external USB-3 disks may require system parameter settings by the user.
I admit: The whole thing left me baffled. For discussions and hints please refer to the Internet. It seems that the whole performance reduction is worst for users who have a lot of RAM. And the problem exists since 2014. Unbelievable!

Links

https://gist.github.com/ 2E0PGS/ f63544f8abe69acc5caaa54f56efe52f
https://www.suse.com/ support/kb/doc/? id=000017857
https://archived.forum.manjaro.org/ t/decrease-dirty-bytes-for-more-reliable-usb-transfer/62513
https://unix.stackexchange.com/ questions/ 567698/slow-transfer-write-speed-to-usb-3-flash-what-are-all-possible-solutions
https://unix.stackexchange.com/ questions/ 23003/ slow-performance-when-copying-files-to-and-from-usb-devices
https://lonesysadmin.net/ 2013/12/22/ better-linux-disk-caching-performance-vm-dirty_ratio/
https://forum.manjaro.org/t/the-pernicious-usb-stick-stall-problem/52297
https://stackoverflow.com/ questions/ 6834929/linux-virtual-memory-parameters

 

Problems with upgrading Cyrus based IMAP servers to Opensuse Leap 15.3

Posted on by

I use 2 Mail servers whose IMAP components are based on Cyrus. The mail servers are major parts of KVM/Qemu based VMs. The OS of these VMs over more than a decade was some edition of Opensuse Linux (after SLES got too expensive for a Freelancer). Authorization to access the IMAP-servers is granted by a separate LDAP server via an SASL interface. All clients access the mail servers via a TLS connection. The communication to the LDAP system uses TLS, too. TLS connections are handled by SSSD which in turn uses PAM.

The configuration of all involved clients and servers is a bit tricky – and all components have special settings to interact smoothly. So I was always happy when upgrade processes of the servers respected my settings and things went smoothly. This was not always the case, but at least the main components survived the upgrade processes. But NOT this time.

The Leap 15.3 repositories do not contain Cyrus packages any longer! And I became aware of this when it was too late. Also the SLES update repositories available after the upgrade did not contain any Cyrus packages. After the upgrade the IMAP components of my mail servers were annihilated. Not funny at all!

Fortunately, I had backuped my VMs – and could restore them to bridge the time when I had to solve the problem. Afterward I spent some hours to try to reconstruct a running Cyrus configuration on the upgraded Leap 15.3 version of the mail server VM.

I got a suitable version of a Cyrus package which works with Leap 15.3 versions from the following repository :
download.opensuse.org / repositories / server: / mail / 15.3/.

However, while the installation after some changes of the configuration file worked well locally, I could not get access to it from external clients. In Kmail I got the message that the server did not support any security mechanisms. But STARTTLS should have worked! I checked the SSSD configuration, checked the imapd.config, nsswitch, ldap.config and certificate references. All OK.

I found the solution after having read some of my own old blog posts. The Leap upgrade had brutally deleted my carefully crafted PAM files “imap” and “smtp” in “/etc/pam.d/”. This has happened before. See:

Mail-server-upgrade to Opensuse Leap 15 – and some hours with authentication trouble …

So: Keep backpus of your PAM configuration if you have some complicated TLS-interactions between your Opensuse machines!

And start acquiring knowledge on Dovecot and the migration from Cyrus to Dovecot. Who knows when Cyrus disappears from all SuSE Repositories. And be prepared for problems with Cyrus and Leap 15.4, too.

I find it also frustrating that “https://doc.opensuse.org/release-notes/x86_64/openSUSE/Leap/15.3/” does not explicitly state that the package “cyrus_imapd” was removed. The information refers to changes in “cyrus-sasl” – but this is a different package. Which ironically still exists (though modified) …

But I am too old to explode just because of the lack of important information …

Replacing unstable Blender 2.82 on Leap 15.3 with flatpak or snap based Blender 3.1

Posted on by

I wanted to work a bit more with my S-curve project in Blender. See e.g.

Blender – complexity inside spherical and concave cylindrical mirrors – IV – reflective images of a Blender variant of Mr Kapoor’s S-curve

At my present location I have to use my old but beloved laptop. As a new start in the project I just wanted to make a little movie showing dynamic changes in the reflections of some moving spheres in front of the S-curve’s metallic surface. As I had already created a few Blender movies some years ago I expected this to be an easy job. But I ran into severe trouble. A major reason was the fact that Opensuse does not provide a reasonably usable RPM of a current Blender versions for Leap 15.3.

I had upgraded the laptop to Leap 15.3 in December, 2022. The laptop has an Optimus system. Of course, I prefer the Nvidia card when creating scenes with Blender. I had had some minor problems with prime-select before. But after recent updates I did not get the Nvidia card running at all – neither with Bumblebee nor with Opensuse’s prime-select. I had to invest an hour to solve this problem. After a complete uninstall of Bumblebee and bbswitch and a de- and re-installation of the Nvidia drivers and suse-prime I got prime-select working in the end.

Just to find that Blender version 2.82 provided by the Leap 15.3 OSS repository was not really stable in a Leap 15.3 environment.

Blender 2.82 unstable on Opensuse Leap 15.3

I tried to create a movie with 100 frames (400×200) in Matroska format with H.264 encoding. Rendering of the complex S-curve reflections with a strongly reduced number of light ray samples still took considerable time with the Cycles renderer on 2 out of 4 real (and 8 hyperthreaded) CPU cores. It happened that the movie – after rendering an hour – was not baked correctly. In some runs Blender crashed after having rendered around 40 frames.

Therefore I changed my policy to rendering just a collection of PNG images. The risk of wasting something like 8 hours of CPU time for a real animation was just too big. But, a new disaster happened when I wanted to open a “video editing”- project in Blender to stitch the images together to create the final movie. Blender just crashed. Actually, Blender crashed on Leap 15.3 whenever I tried to open any project type other than “General”. Not funny … I had not experienced something like this on Leap 15.2.

Unfortunately, and in contrast to Tumbleweed there is no official and recent version of Blender available for Leap 15.3. So much about stability and reliability of Leap. What to do?

Canonical’s snap service or Flatpak to the rescue

I had read about SNAP of Canonical before – but never tried it. Canonical’s strange side steps during the evolution of Linux have always been disturbing for me. Regarding “snap” I also had a newspaper article in mind about a controversy with Linux Mint – with claims that snap has closed proprietary components and is transmitting telemetry data to Canonical. Working with root rights …

Therefore, a flatpack installation had to be considered, too…. Risks are small on my old laptop. I just tried – despite concerns and doubts regarding snap.

Blender 3.1 via “snap“

I just followed the instructions for Opensuse on https://snapcraft.io/docs/installing-snap-on-opensuse to install “snap” on my Leap 15.3 system. As root: 

zypper addrepo --refresh     https://download.opensuse.org/repositories/system:/snappy/openSUSE_Leap_15.3     snappy
zypper install snapd
systemctl enable --now snapd
systemctl enable --now snapd.apparmor
rcsnapd status
# install blender
snap install blender --channel=3.1/stable --classic

Another description can be found here:
https://www.linuxcapable.com/how-to-install-snap-snap-store-snapcraft-on-opensuse-leap-15/

The run-time environment of snap is handled by a daemon “snapd”. We find the majority of the installed files in four directories

/snap           -- 1.1 GB thereof 686 MB for Blender
/usr/lib/snap   -- 48 MB
/var/snap       -- almost nothing 
/var/lib/snapd  -- 360 MB

In the first directory you also find the installed applications. In my case “Blender 3.1.2”.
The folder “/snap” contains around 1.1 GB. Blender is part of it and consumes 686 MB.

The startup of Blender via 

/snap/bin/blender &

takes about 2 secs. Loading my S-curve test file took even longer. Otherwise snap’s Blender version 3.1.2 seemed to work perfectly. All the bugs I had seen with 2.82 were and are gone. And: I could not really notice a difference in performance when working with 3D objects in the Blender viewport. For a render test case I found 17.72 secs per frame on average. Memory release after leaving Blender seemed to be OK.

Blender 3.1 via Flatpak

Then I tried a Blender installation with flatpak. The installation is almost as simple as for snap. See https://www.flatpak.org/ setup/ openSUSE. Which sums up more or less to issuing the following commands : 

zypper install flatpak
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
flatpak install flathub org.blender.Blender

The required packages of the Opensuse repository were

Regarding the installation I was surprised to find that flatpak requires much more space than snap. The majority of flatpack files is located in the following folder: 

/var/lib/flatpak  -- 2.3 GB, thereof around 628 MByte for Blender

So, in comparison to snap, this makes a remarkable huge difference regarding the required files to provide a working runtime environment! I understand that for just one application many files have to be provided for a stable run-time environment on KDE – but in comparison to the snap installation it appears bloated.

We start flatpak’s Blender via 

flatpak run org.blender.Blender &

The startup time was comparable to the snap installation – no difference felt. The test render time per frame was 18.95 secs. So that was 1.2 secs or 6.5 % slower than snap. Reason unclear. The difference after repeated trials was sometimes a bit smaller (4%), but sometimes also bigger (8.5%) -depending on the start order of the blender applications. But on average snap’s Blender was always a bit faster. I regard the difference is not really as problematic. But it is interesting. Memory release worked also perfectly with flatpak’s Blender installation.

Conclusion

Before you start struggling with the Blender 2.82 binary Opensuse provides, it is really worth trying a flatpak-based or a snap-based installation of Blender 3.1.2. On my system both worked perfectly.

Before you make a decision whether to go with flatpak or snap take into account the critical points which were discussed on the Internet regarding proprietary aspects of snap. Personally, I am going to use flatpak for Blender.
Otherwise, my opinion is that distributors like Opensuse should provide an important application as Blender in its present version as a binary and resolve dependencies as soon as possible. For me a flatpack installation is just a compromise. And it cost me a lot of valuable SSD space.

Links

https://www.linux-community.de/ausgaben/ linuxuser/2018/02/dreikampf/
https://hackaday.com/2020/06/24/whats-the-deal-with-snap-packages/
https://www.makeuseof.com/snap-vs-appimage-vs-flatpak/

Ceterum censeo: The worst living fascist and war criminal today who must be isolated, denazified and imprisoned is the Putler.