Opensuse Leap, dolphin and abysmal slow data transfer between two external USB 3 disks

Today I had to copy a set of multiple directories, each with around 600 sub-directories and a total of 420,000 singular files, between two external USB-3 hard disks. Both the USB controllers of the mainboard and the controllers of the external disks were USB-3 capable. My PC has 64 GB RAM. Which in this case was of no help. I got the problem on an Opensuse Leap 15.3-system, but you may find it on other Linux-OS, too.

The problem

First I started to copy with the help of KDE’s Dolphin. The transfer rates were really bad – something about 5 MB/s. Would have had to wait for hours.

Then I tried with just a standard cp-command. The rates rose to something like 20 MB/s to 25 MB/s. But this was not a reasonable transfer rate either.

A workaround

A search on the Internet then gave me the following two commands

mytux:~ # echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes
mytux:~ # echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes

Note that setting values on these system parameters automatically sets the standard values of

mytux:~ # cat /proc/sys/vm/dirty_ratio
20
mytux:~ # cat /proc/sys/vm/dirty_background_ratio
10 

to zero! And vice versa. You either have to use dirty_bytes and dirty_background_bytes OR the related ratio-parameters!

The effect of the above settings was remarkable: The total transfer rate (i.e. reading from one disk and writing to another; both on the same controller) rose to 130 MB/s. Still far away from theoretical rates. But something I could live with.
It would be worth to experiment more with the size of the parameters, but I was satisfied regarding my specific problem.

Conclusion

An efficient data transfer on Linux systems between two external USB-3 disks may require system parameter settings by the user.
I admit: The whole thing left me baffled. For discussions and hints please refer to the Internet. It seems that the whole performance reduction is worst for users who have a lot of RAM. And the problem exists since 2014. Unbelievable!

Links

https://gist.github.com/ 2E0PGS/ f63544f8abe69acc5caaa54f56efe52f
https://www.suse.com/ support/kb/doc/? id=000017857
https://archived.forum.manjaro.org/ t/decrease-dirty-bytes-for-more-reliable-usb-transfer/62513
https://unix.stackexchange.com/ questions/ 567698/slow-transfer-write-speed-to-usb-3-flash-what-are-all-possible-solutions
https://unix.stackexchange.com/ questions/ 23003/ slow-performance-when-copying-files-to-and-from-usb-devices
https://lonesysadmin.net/ 2013/12/22/ better-linux-disk-caching-performance-vm-dirty_ratio/
https://forum.manjaro.org/t/the-pernicious-usb-stick-stall-problem/52297
https://stackoverflow.com/ questions/ 6834929/linux-virtual-memory-parameters

 

Nupro X3000 RC – a solid high quality supplement to your Linux Audio

A friend asked me what sound equipment I use on my Linux machine. She wanted to to buy some new decent speakers. I had to make a similar decision a year ago. Coming to a conclusion back then became a more difficult process than I had expected.

I admit that I am a total amateur regarding sound equipment. I have not changed my sound cards (Asus Sonar D2X, Creative X-Fi Titanium, Onboard High Definition GM206) for a long, long time. And I do not hear as well as in my younger years. But during Corona and home office times I became really discontent with my old Creative speakers. One cannot all the time wear headphones. So some new speakers for my Linux workstation became a topic on my private agenda.

Questions ahead of a decision for some speakersfor your PC

When I seriously started thinking about some investment the following questions came up:

A surround system? Active or passive boxes? Suitable for a shelf or standing on the floor? Do you want to use the speakers later also in other contexts than just as a background equipment in your working room? What is appropriate for your room size? Connections cable (copper, optical?) based or WiFi or Bluetooth based? In my age when hearing capabilities are reduced: Will high end properties make a difference at all? And the most limiting factor: budget.

Taking all these factors into account will certainly lead to very personal decisions. So, when I make an explicit recommendation here – take it with caution and a grain of salt.

Guidelines to choosing speakers for a non-professional PC environment

Here are the personal guidelines which I followed – after I had read reviews, listened to Teufel and Edifier speakers at friends and listened to a relative expensive Logitech surround system at my nephew. You may have other references, other budgets and hear much better and more differentiated than I do. So relax if you come to other conclusions.

And do not forget: I am talking about sound equipment on a PC for background music enjoyment in a working room – not for professional objectives and High End specialists.

  • Recommendation 1: If you are interested in sound quality and are a music enthusiast – forget about surround systems. Quantity (many speakers) almost always enforces quality compromises, which you are going to hear in the end. Better invest your money into a 2.0 or 2.1 system which fits the (probably) limited size of your working room.
  • Recommendation 2: If your room size is up to 30 square meters, invest into relatively small speakers – but of studio quality. They will give you a much more pronounced and positioned sound than surround systems. Regarding money think of speakers which you later can supplement with a sub-woofer – e.g. in case you want to move the speakers to a larger room sometime in the future.
  • Recommendation 3: Regarding bass: I am a heavy metal friend – sometimes. I have my phases and periods regarding music … Sometimes I like Jazz, only. Bass in the named two cases has a different meaning to me – but in any case I do not like resonances of my speakers. The stereo speakers alone should already provide a solid, broad and resonance free bass fundament – without a sub-woofer. A sub-woofer can deliver an extra feeling in the case of metal – but for Jazz and classical music I would not consider a sub-woofer as really relevant. So go for some solid speakers with the option of adding a sub-woofer in the future.
  • Recommendation 4: Do not underestimate the effect (or limitations) of the DAC in your sound card! At a certain quality level of your future speakers you are probably going to hear differences. So – if you are lucky and can invest into expensive speakers rethink your sound card equipment, too.
  • Recommendation 5: Do not underestimate the effect of the boxes’ positions in the room. Also in small rooms you will experience bass line effects around 100 Hz or so if you place your boxes in the room’s corners. This leads to the point that you may want some equalizer option to optimize the bass base a bit. Well, Linux or at least most music applications for Linux supply you with equalizers; but it is a nice option to be able to do something at the (active) boxes themselves to get a basic “direction” into your sound environment. And here we would also like to have the option of defining some “presets”.
  • Recommendation 6: Active boxes or amplifier? A very difficult question! In a PC and mobile environment I would tend to active speakers, but … The amplifier technique today is so good that at least in my case my hearing deficits are certainly more important.
  • Recommendation 7: Wifi? personally, I would say: Yes, you should have this option. But if so: Go for a 5 GHz band. And check whether your router offers you the option to define the precise band it should work on or whether the router automatically adapts the precise channel to avoid disturbances with other sources.
  • Personal opinion some people certainly would like to crucify me for: Teufel speakers seem to be a bit overestimated. Personally I do not think that the quality-price relation is convincing. After having heard to a standing speaker pair I think that the balance between bass and mid-range frequency sound is strange. Very vague in a way.

Nupro X3000 speakers as a solid option for a reasonable price

Taking all these aspects into account I ended up with a decision for (active) Nupro X3000 RC speakers from the producer “Nubert electronic GmbH“.

So far, I have not regretted this decision for a second. These boxes did not disappoint me – neither with Classical music, Jazz nor Heavy Metal.

Though admittedly, if you want to feel bass and drumming these boxes improve their performance in larger rooms certainly a bit when combined with a sub-woofer (which I personally use at a second sound card). But this happens at rare occasions …

Ease of setup?

The setup of the active boxes is very simple; the explanations on the accompanying leaflets are fully sufficient. You define everything by a 4 direction control button on one of the speakers. The button and a small display are hidden behind magnetically attached front panels.

Basically, you just have to define a master and a slave speaker in the first setup round and choose a connection to your sound source – here to the output connectors of a PC soundcard. In the end I used the “aux” entry and still live with an analog cable based connection between the sound card and the main box plus a digital coax cable between the boxes. (Due to the speakers’ distance I had to buy an additional coax cable. It disappears behind a shelf).

But a WiFi connection between the speakers works very well, too. I could see no major conflict with the 5 GHz channels occupied by the WLAN routers in my surroundings.

The basic connection options to your PC and sound card are manifold: The USB-interface of the Nupro sound processor appears as an USB sound card on your PC; this
“sound card” is well supported on my Opensuse and KDE based Linux systems. You just have to chose the SPDIF stereo variant of the two options offered in the KDE/Phonon sound settings.

Besides an USB cable the connection cables delivered with the speakers include an optical cable with TOSLink adapters, a SPDIF cable and analog cables with cinch connectors. And eventually there also is the option of a Bluetooth connection – if your PC has such a device.

In the end I personally heard no major difference between analog and digital signal handling. Neither with USB nor the optical connection to my old ASUS Xonar D2X sound card or the optical connection to the X-FI Titanium nor the onboard GM206 High Definition soundcard. The TI-Burr-Brown DAC of the Asus card still seems to be relatively good – at least for my ears.

I also have an additional X-FI Titanium card from Creative in my PC. I like the sound of the Asus card better with my Sennheiser headphones. Regarding the Nupro X3000 I was actually in doubt: For some music I find the sound slightly crispier with the X-Fi. However, whether this is a sign of quality is questionable. I change the sound card from time to time, just for fun – and still have no real preference.

Regarding distances the analog cable option for the connection to your PC’s sound card may be the most reasonable solution – as the optical, SPDIF coax and USB cables coming with the speakers are of limited length.

There is even a possibility to realize a pure Wifi connection from your PC to the X3000 RC speakers. Such a solution, however, requires a special transceiver (135 €) from the producer Nubert; see below. I have no tested this type of connection, yet.

They speakers offer you some basic options regarding the sound balance. A very positive feature is the integrated 5 band equalizer. As said above this allows for a basic adjustment of the sound signature. Not unimportant in my age. In addition the handheld remote control device allows for a change of the relative basic balance between bass and treble.

You can also define a lower cut-off frequency for the bass and the transition frequency to a sub-woofer. Furthermore you can set 6dB a gain of certain analog input channels.

Disappointments ?

Something which disappointed me was the Bluetooth connection of the X3000 RC to my old Samsung smartphone – here I got periodic dropouts. I have not clarified this problem up to now. I do not exclude problems with the Bluetooth and the VLC player on my phone. In reviews I have not read about any such dropouts – but you have been warned. I recently tried a Bluetooth connection from my laptop, too. This one worked flawless. So, I do not know …

Another major disappointment was and is Nubert’s “X-Remote App”. In my case it simply does not work on my Android 6 device. It gets stopped by Android just after granting permission to determine the geo-location. Which by the way is something I do not like in general. I got in contact with the Nubert company recently. They affirmed that they do not collect data, but that it is Google which enforces the explicit accept for geo-location when building up Wifi connections. Had to be expected, we know this stupid problem already from the mess with the German Corona App on Android. BBG again – Big Brother Google … No further comments required.

I had no real need for the App so far. After the basic setup of all the speaker’s internal settings (e.g. the equalizer) I can control the most needed adjustments via the handheld remote control accompanying the speakers. The “room calibration” feature of the App would have been nice – but it requires buying an additional piece of microphone equipment from Nubert for Android smartphones.

Sound quality

Do not expect a solid sound quality review from me. I have neither equipment nor objective, trained ears for such a review. I can only describe an impression – very much in analogy to wine – a sort of personal sound “taste and feeling”
after having heard a lot of music on the speakers. Do I like them with different kinds of music, vocals and instruments?

In a nightlong session I have also compared the Nupro X3000 capabilities with my old Elac 4π (4 Pi) speakers in the living room. They are controlled by NAD pre- and end-amplifiers plus a NAD CD player. I did the comparison with music pieces of very different styles. I really was astonished how good the the small Nupro 3000x speakers could follow the 4π (4 Pi) Elac speakers and fill the room with sound and a solid bass base! Well, of course the Elacs do a better job with the bass at some point, but no wonder regarding their dimensions. Still, this first impression of the Nupro speakers was very convincing.

Then I moved the Elacs and Nupros boxes into my smaller working room – well, the Nupro X3000 at once felt much more adequate. They positioned different sound origins in the stereo sound cloud much more precisely – which is no wonder either. And they filled the whole room with music easily.

A hint: As the speakers work with a bass reflex opening at their backside you should not position the boxes directly at at wall – but leave some space.

Meanwhile, I have listened to a broad spectrum of music on these speakers – ranging from Eberhard Weber, Jan Gabarek, Kjetil Bjørnstad (with an without vocals), Laurie Anderson to compositions of Steve Reich, Rihm, Arvo Pärt and to recent recordings of classical music as of the Danish String Quartet or Sol Gabetta. Intermixed with stuff from Riverside, Korn, Linkin Park, Amorphis, Insomnium, Dark Tranquility, In Flames and Rammstein. As well as a lot of classical symphony and opera recordings. And – as a very welcome side effect – I have re-detected the wonders in the songs of Tom Waits.

You know what: All of it was pure joy – taking into account the sometimes strange intentional distorted mix you find in some heavy metal pieces.

In my opinion the balance between bass, mid-range and treble of the X3000 RC speakers is very good. You (almost) never loose the resolution of instruments covering different frequency regions. Some critics in the audio press was directed to problems in the mid-range frequency area. Personally, I cannot confirm this. If there is some problem, I would bet it appears in larger rooms. But this is not the target environment of these speakers. In my working room the mid range appears very present – both with vocals and classical instruments. But, probably I do not know what high end sound really is … 🙂

I could not hear any bass resonances so far – with standard settings. But when you place the speakers close to a wall or corner you may want to reduce the low bass (< 100 Hz) a bit.

Summary: I very seldom use my Sennheiser headphones these days. I really do like the sound of these speakers.

Are there weaknesses? Well, the X3000 speakers have a little weakness at very low volume in my opinion – the relative weight of mid-range vs. bass changes to bass. May have to do with reflections in the room (or my hearing). But the advantage is that I have so far not felt any need for setting the loudness option to on.

Future options?

Now, I come to a point which makes the Nupro boxes also an investment into some future wireless audio infrastructure: For 135€ you get the NuConnect trX Wireless transceiver (https://www.nubert.de/nuconnect-trx/p4210/). This little brick allows eg. for multi-room wireless solutions, but also for a transmission of digital signals from your PC or other sources to the active speakers.

Alternatively, you could also think about a combination of the trX Transceiver with the “NuControl 2 pre-amplifier” or (a cheaper) AmpX amplifier – both interesting products of Nubert. The latter amplifier uses in my understanding the same amplifying bricks as the active speakers, but now combined and supplemented with other electronics and thus turned into a full amplifier. The critics of this 700 € amplifier
are surprisingly good (see: https://www.nubert.de/nuconnect-ampx/p3646/?category=225).

So, the speakers mark an entrance into a much broader eco-system. In my case a completely digitized audio center on a Linux workstation combined with the trX transceiver, the X3000 speakers, the AmpX and other already existing audio equipment in different rooms appears on the horizon.

Sound support on my Linux system

Working with two soundcards
As I have two sound cards available I kept the three front speakers and the subwoofer box of my old Creative speaker set. The front speakers are placed on my working table – the subwoofer on the floor. This allows for astonishing surround feelings even with stereo sound. A little contribution of these desktop speakers to the louder sound coming from the X3000 in the background and you “swim in an extended audio space”. Interesting for some kinds of music. Here the Pulseaudio mixer (pavucontol) on a Linux system is of advantage to balance sound contributions between the different channels of the active sound cards accurately and al gusto.

Regarding the Linux sound support in general
As a Linux user I have made my peace with Pulseaudio, pavucontrol, the Ladspa equalizer and KDE’s Phonon over the years. It is sometimes still a mess to reproduce working settings for multiple multi-channel sound cards after system upgrades – but once PA and Phonon do work as expected, they do their work well.

The last time when strange things happened was when I upgraded to Opensuse Leap 15.2. Reason: Substantial changes to the Phonon user interface combined with a loss of differentiated setting options. As a result I had to manipulate the directives in the PA configuration files locally in my home directory and below /etc/pulse to get everything right again. The loss or hiding of options is a sickness that has spread itself over central KDE applications during the last years …. I always make a backup of my personal PA settings in my home directory and central Alsa and PA settings, now.

A major topic always is to find working settings which direct all sound output of any application through the Ladspa equalizer and then its output to multiple sound cards. On a KDE desktop such settings have to be consistent with Phonon settings – or the system will forget and overwrite your preferences with the next system start. Then you know that you have to manually change entries in the configuration files …

Be careful with your new speakers when experimenting and switching to new sound configurations – e.g. from analog to digital signals or changes of the the sound card or moving from PA to pure Alsa. The resulting sound and, in some cases, also distortions may be louder than you expect! Always turn the volume of your external speakers to a minimum ahead of such experiments – and also reduce the volume of sound sources to a very low level.

During the last three to four years I have used the PA mixer “pavucontrol” to control the relative volumes of sound sources (i.e. applications) and the audio channels of the different sound cards on my system. But be careful with your settings here, too. In the past Pulseaudio did some strange things with audio signals from the system – e.g. turning them suddenly to 100%. I have not experienced such things in the past 3 years, but Nupro X boxes are too expensive to risk any accidental damage.

The 15-band PA Ladspa equalizer helps to define some basic sound presets with very slight adjustments – the Nupro speakers basically do not need any significant changes from a flat frequency curve of the equalizer.

Note that changes of the equalizer’s settings may be accompanied by a general volume reduction on pavucontrol and a loss of relative channel weights there. Saving (and loosing) presets of the equalizer is no fun either. Some mess will probably always remain with PA … You just need to invest some time into balanced
presets – and then do not touch the central equalizer again.

The good thing is that you can change the direction of the output of applications to a sound sink directly with pavucontrol. So, you can configure the sound output of music applications to run through an equalizer or not. Again – be careful with the impact of such changes on the volume.

My favorite player still is Clementine at 48.000 or 96.000 Hz sampling rate. It offers its own equalizer. If you want to fiddle with an equalizer than use this one.

Sound extraction from CD recordings I do with K3B to “lossless” Ogg Vorbis or Flac encoding.

Conclusion

The active Nupro X 3000 RC speakers are worth the money you have to pay for them. They suit any Linux workstation well. The connection options to sound sources are manifold. Basic analog cable connections work, of course. An USB connection was directly supported on my Opensuse Linux. Optical and SPDIF coax connections to respective output connectors of sound cards work well, too. The possibility to create a full Wifi based solution with some extra (135€) equipment from Nubert is an additional goody.

The setup and the configuration of a speaker pair were very simple. You get an included 5 band equalizer in each speaker, which allows for basic room and position adjustment.

The general sound quality is in my opinion and for my ears excellent. The speakers easily fill small and even rooms up to 40 square meters with sound and provide a solid bass. The balance between bass, mid range and treble fits my ears. Single instruments in complicated arrangements are well distinguished. The positioning of sources in the stereo range is very good.

Links

https://www.igorslab.de/en/welcher-passt-besser-nubert-nupro-x-3000-rc-oder-nupro-x-4000-rc-und-die-qual-der-wahl-2/4/
https://www.lite-magazin.de/2018/11/aktivlautsprecher-nubert-nupro-x-3000-kompakte-komplettloesung-auf-audiophilem-niveau/
https://www.technic3d.com/ article/ audio/ lautsprecher/2087-test-aktive-kompaktbox-nubert-nupro-x-3000-rc/1.htm

Enforcing specific command arguments for a selected user with sudo

In one of my last articles I needed to enforce the execution of a command with certain arguments – specific for the present user. I.e., I wanted to take away the freedom of the user to set arbitrary command argument(s) as he/she liked.

This had to be done in addition to another set of rules – namely a bunch of iptables filter rules which also depended on the UID. So the command had to be run with the UID of the user him/herself and not with the UID of root.

The solution came with sudo. This may appear a bit surprising to some readers. The reason is that sudo normally is used to allow selected users to run commands with another UID than the one they have themselves. I call “the other user” the “effective user” below. In a sudo context the effective user corresponds to a SUDO_UID variable in addition to the UID environment variable. The predominant example for invoking the sudo-mechanism certainly is to allow users to run a command as root. But it can be extended to any other user (made harmless by taking away hsi/her login shell or being especially privileged due to membership in a special group).

In my case I needed to enforce command execution with an effective user being identical to the original user him/herself – but with special arguments. In such a situation you have to take take the permission to execute/read the original command completely away from the user. Otherwise he/she could use it with any argument. But sudo requires that the defined effective user is able to read and execute the command. This seemingly contradictory situation can be solved by invoking a special user-group.

Maybe the recipe described below helps some readers to enforce command execution with specific arguments in other contexts.

A simple scenario

Let us assume you have developed a program “myprog” which accesses special web-service that you have installed on some web-servers in your Intranet. Let us further assume that some specific users shall be restricted to access the service on a defined server only – and there only with certain arguments. Such parameters may reduce or give rights to access certain data the service could in principle provide. All this is regulated by 2 arguments to the myprog-program: a FQDN for the host and a “level”. “level 0” allows free access to a very basic service version. “level 1” invokes the program with personalized options and requires a login. But your people have started to play around with the Login. So, you want a group of users to issue the command with a certain “host” and “level 0” only. Let us assume that user “mark” is one of those users who should invoke the command only in the form

mark@mytux:~>myprog -h myserv.anraconc.de -L 0

How can we achieve this with sudo?

Restricting command use to specific arguments

Below I discuss modifications of the “/etc/sudoer” file. This is risky in very many ways – not only regarding security.

Disclaimer: I take no responsibility whatever for the consequences of the sudo approach described below and its application to your computers. The sudoer rules have to be tested carefully before the are used in a production environment and their setup must be supervised by an expert.

I assume that you have installed your program at the path “/usr/bin/myprog” with standard rights

-rwxr-xr-x 1 root root 334336 17. Mai 2020  /usr/bin/myprog

Then one can follow the following recipe (as root) to get “mark” under control:

  • Step 1: Create a special group for the command in question, e.g. “mygroup”. Ensure that mark does not become a member of this group.
  • Step 2: Change the ownership and access rights of “/
    usr/bin/myprog” according to
    • chown root.mygroup /usr/bin/myprog
    • chmod 750 /usr/bin/myprog
  • Step 3: Add some lines to “/etc/sudoers” (with visudo):
     
    ....
    Defaults env_reset
    Defaults env_keep = "LANG LC_ADDRESS LC_CTYPE LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_ATIME LC_ALL LANGUAGE LINGUAS XDG_SESSION_COOKIE"
    
    Defaults:mark env_keep += "DISPLAY"
    
    #Defaults targetpw   # ask for the password of the effective target user e.g. root
    #ALL   ALL=(ALL) ALL
    
    mark ALL=(mark:mygroup) /usr/bin/myprog -h mysrv.anraconc.de -L 0
    ....
    

Explanation:
Due to step 2 user “mark” cannot read, change or execute the command directly any longer. The rest depends a bit on ensuring the “mark” never becomes a member of group “mygroup”. (But other users which you trust may become members.)

Regarding the sudoer rules I assumed that you reset the environment of a sudo user as a default. Keeping up the “DISPLAY” variable helps to get around some access problems with the present X11-screen of “mark”. I also assumed that you use the sudo-mechanism in a way which requests that the user enters his/her password. The last line allows “mark” on all hosts/terminals to execute “/usr/bin/myprog” as him/herself, but with the GID of group “mygroup” and exactly with the options “-h mysrv.anraconc.de -L 0”.

Note that sudo compares the command including arguments as one string!

User “mark” must run the command myprog from now on in the form:

sudo -u mark -g mygroup  myprog -h myserv.anraconc.de -L 0

and enter his password. Any deviation will be blocked by the sudoer mechanism.

Some practical hints

After carefully evaluating security implications you can make life easier for “mark” in two ways:

  • Let him execute the command (with the defined arguments) without providing a password. Use the NOPASSWD attribute; see the man pages for the sudoers file.
  • Write a script which encapsulates the described sudo command with the options.

By such measures, you may save “mark” some typing time.

Another point is to keep the file permissions up in the future. This may become a problem if and when you apply the described mechanism to some standard Linux commands which are installed and updated by some package administration tool of your distribution. You have to carefully check that the installation routines do not overwrite the permission settings! A handwritten systemd service or a cron job may help you with this task.

With some reading or experience it should be easy to extend the described recipe to groups of users and to other commands.

There are multiple ways to allow other users to execute the command freely if this should be required. The sudoer file knows about a logical NOT operator (!); this helps to add a sudoer rule for all users but NOT “mark”. Another simple approach would be to add all users but “mark” to the group “mygroup”.

Conclusion

The sudoer-mechanism is a mighty Linux tool. We can not only allow users to execute commands as another user, but also with the permissions of another group. AND we can enforce the usage of commands with predefined arguments for selected users or user groups.

As fiddling with the sudoer mechanism is always a bit risky: Please, write me a mail if you find some major mistake or security problem of my approach.