Tag Archives: two external USB disks

Opensuse Leap, dolphin and abysmal slow data transfer between two external USB 3 disks

Posted on by

Today I had to copy a set of multiple directories, each with around 600 sub-directories and a total of 420,000 singular files, between two external USB-3 hard disks. Both the USB controllers of the mainboard and the controllers of the external disks were USB-3 capable. My PC has 64 GB RAM. Which in this case was of no help. I got the problem on an Opensuse Leap 15.3-system, but you may find it on other Linux-OS, too.

The problem

First I started to copy with the help of KDE’s Dolphin. The transfer rates were really bad – something about 5 MB/s. Would have had to wait for hours.

Then I tried with just a standard cp-command. The rates rose to something like 20 MB/s to 25 MB/s. But this was not a reasonable transfer rate either.

A workaround

A search on the Internet then gave me the following two commands

mytux:~ # echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes
mytux:~ # echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes

Note that setting values on these system parameters automatically sets the standard values of 

mytux:~ # cat /proc/sys/vm/dirty_ratio
20
mytux:~ # cat /proc/sys/vm/dirty_background_ratio
10

to zero! And vice versa. You either have to use dirty_bytes and dirty_background_bytes OR the related ratio-parameters!

The effect of the above settings was remarkable: The total transfer rate (i.e. reading from one disk and writing to another; both on the same controller) rose to 130 MB/s. Still far away from theoretical rates. But something I could live with.
It would be worth to experiment more with the size of the parameters, but I was satisfied regarding my specific problem.

Conclusion

An efficient data transfer on Linux systems between two external USB-3 disks may require system parameter settings by the user.
I admit: The whole thing left me baffled. For discussions and hints please refer to the Internet. It seems that the whole performance reduction is worst for users who have a lot of RAM. And the problem exists since 2014. Unbelievable!

Links

https://gist.github.com/ 2E0PGS/ f63544f8abe69acc5caaa54f56efe52f
https://www.suse.com/ support/kb/doc/? id=000017857
https://archived.forum.manjaro.org/ t/decrease-dirty-bytes-for-more-reliable-usb-transfer/62513
https://unix.stackexchange.com/ questions/ 567698/slow-transfer-write-speed-to-usb-3-flash-what-are-all-possible-solutions
https://unix.stackexchange.com/ questions/ 23003/ slow-performance-when-copying-files-to-and-from-usb-devices
https://lonesysadmin.net/ 2013/12/22/ better-linux-disk-caching-performance-vm-dirty_ratio/
https://forum.manjaro.org/t/the-pernicious-usb-stick-stall-problem/52297
https://stackoverflow.com/ questions/ 6834929/linux-virtual-memory-parameters