USB transfer speeds slow

[theprofa325]

Code: Select all

System:    Kernel: 5.10.0-27-amd64 [5.10.205-2] x86_64 bits: 64 compiler: gcc v: 10.2.1 
           parameters: BOOT_IMAGE=/boot/vmlinuz-5.10.0-27-amd64 root=UUID=<filter> ro quiet splash 
           Desktop: Xfce 4.18.1 tk: Gtk 3.24.24 info: xfce4-panel wm: xfwm 4.18.0 vt: 7 
           dm: LightDM 1.26.0 Distro: MX-21.3_x64 Wildflower October 20  2021 
           base: Debian GNU/Linux 11 (bullseye) 
Machine:   Type: Laptop System: SAMSUNG product: 940X3G/930X3G v: P05ACJ serial: <filter> Chassis: 
           type: 9 serial: <filter> 
           Mobo: SAMSUNG model: NP940X3G-K04US v: SAMSUNG_SW_REVISION_1234567890ABCD 
           serial: <filter> UEFI: American Megatrends v: P05ACJ.128.140819.dg date: 08/19/2014 
Battery:   ID-1: BAT1 charge: 38.2 Wh (79.7%) condition: 47.9/55.5 Wh (86.3%) volts: 7.8 min: 7.6 
           model: SAMSUNG Electronics SR Real Battery type: Li-ion serial: <filter> 
           status: Discharging cycles: 1522 
CPU:       Info: Dual Core model: Intel Core i7-4500U bits: 64 type: MT MCP arch: Haswell 
           family: 6 model-id: 45 (69) stepping: 1 microcode: 26 cache: L2: 4 MiB 
           flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 19155 
           Speed: 898 MHz min/max: 800/3000 MHz Core speeds (MHz): 1: 898 2: 898 3: 898 4: 898 
           Vulnerabilities: Type: gather_data_sampling status: Not affected 
           Type: itlb_multihit status: KVM: VMX disabled 
           Type: l1tf mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable 
           Type: mds mitigation: Clear CPU buffers; SMT vulnerable 
           Type: meltdown mitigation: PTI 
           Type: mmio_stale_data status: Unknown: No mitigations 
           Type: retbleed status: Not affected 
           Type: spec_rstack_overflow status: Not affected 
           Type: spec_store_bypass 
           mitigation: Speculative Store Bypass disabled via prctl and seccomp 
           Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer sanitization 
           Type: spectre_v2 mitigation: Retpolines, IBPB: conditional, IBRS_FW, STIBP: 
           conditional, RSB filling, PBRSB-eIBRS: Not affected 
           Type: srbds mitigation: Microcode 
           Type: tsx_async_abort status: Not affected 
Graphics:  Device-1: Intel Haswell-ULT Integrated Graphics vendor: Samsung Co driver: i915 
           v: kernel bus-ID: 00:02.0 chip-ID: 8086:0a16 class-ID: 0300 
           Device-2: Silicon Motion WebCam SC-10HDP12B24N type: USB driver: uvcvideo bus-ID: 1-5:3 
           chip-ID: 2232:1049 class-ID: 0e02 serial: <filter> 
           Display: x11 server: X.Org 1.20.11 compositor: xfwm4 v: 4.18.0 driver: 
           loaded: modesetting unloaded: fbdev,vesa display-ID: :0.0 screens: 1 
           Screen-1: 0 s-res: 3200x1800 s-dpi: 96 s-size: 847x476mm (33.3x18.7") 
           s-diag: 972mm (38.3") 
           Monitor-1: eDP-1 res: 3200x1800 hz: 60 dpi: 277 size: 293x165mm (11.5x6.5") 
           diag: 336mm (13.2") 
           OpenGL: renderer: Mesa DRI Intel HD Graphics 4400 (HSW GT2) v: 4.5 Mesa 20.3.5 
           compat-v: 3.0 direct render: Yes 
Audio:     Device-1: Intel Haswell-ULT HD Audio vendor: Samsung Co driver: snd_hda_intel v: kernel 
           bus-ID: 00:03.0 chip-ID: 8086:0a0c class-ID: 0403 
           Device-2: Intel 8 Series HD Audio vendor: Samsung Co driver: snd_hda_intel v: kernel 
           bus-ID: 00:1b.0 chip-ID: 8086:9c20 class-ID: 0403 
           Sound Server-1: ALSA v: k5.10.0-27-amd64 running: yes 
           Sound Server-2: JACK v: 1.9.21 running: no 
           Sound Server-3: PulseAudio v: 14.2 running: yes 
Network:   Device-1: Intel Wireless 7260 driver: iwlwifi v: kernel modules: wl port: f040 
           bus-ID: 02:00.0 chip-ID: 8086:08b1 class-ID: 0280 
           IF: wlan0 state: up mac: <filter> 
           Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet vendor: Samsung Co 
           driver: r8169 v: kernel port: e000 bus-ID: 03:00.0 chip-ID: 10ec:8168 class-ID: 0200 
           IF: eth0 state: down mac: <filter> 
           IF-ID-1: tun0 state: unknown speed: 10 Mbps duplex: full mac: N/A 
Bluetooth: Device-1: Intel Bluetooth wireless interface type: USB driver: btusb v: 0.8 
           bus-ID: 1-4:2 chip-ID: 8087:07dc class-ID: e001 
           Report: hciconfig ID: hci0 rfk-id: 21 state: up address: <filter> bt-v: 2.1 lmp-v: 4.0 
           sub-v: 500 hci-v: 4.0 rev: 500 
           Info: acl-mtu: 1021:5 sco-mtu: 96:5 link-policy: rswitch hold sniff 
           link-mode: slave accept service-classes: rendering, capturing, object transfer, audio 
Drives:    Local Storage: total: 469.47 GiB used: 237.38 GiB (50.6%) 
           SMART Message: Unable to run smartctl. Root privileges required. 
           ID-1: /dev/sda maj-min: 8:0 vendor: Samsung model: MZNTE256HMHP-000 size: 238.47 GiB 
           block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s type: SSD serial: <filter> 
           rev: 1K0Q scheme: GPT 
           ID-2: /dev/sdb maj-min: 8:16 type: USB vendor: Kingston model: DT microDuo 3C 
           size: 231 GiB block-size: physical: 512 B logical: 512 B type: N/A serial: <filter> 
           rev: PMAP scheme: MBR 
           SMART Message: Unknown USB bridge. Flash drive/Unsupported enclosure? 
Partition: ID-1: / raw-size: 70.31 GiB size: 68.66 GiB (97.65%) used: 49.37 GiB (71.9%) fs: ext4 
           dev: /dev/sda3 maj-min: 8:3 
           ID-2: /boot/efi raw-size: 1024 MiB size: 1022 MiB (99.80%) used: 420 KiB (0.0%) 
           fs: vfat dev: /dev/sda1 maj-min: 8:1 
           ID-3: /home raw-size: 151.16 GiB size: 147.73 GiB (97.73%) used: 126.12 GiB (85.4%) 
           fs: ext4 dev: /dev/sda4 maj-min: 8:4 
Swap:      Kernel: swappiness: 15 (default 60) cache-pressure: 100 (default) 
           ID-1: swap-1 type: partition size: 16 GiB used: 185.2 MiB (1.1%) priority: -2 
           dev: /dev/sda2 maj-min: 8:2 
Sensors:   System Temperatures: cpu: 48.0 C mobo: 48.0 C 
           Fan Speeds (RPM): N/A 
Repos:     Packages: note: see --pkg apt: 2609 lib: 1316 flatpak: 0 
           No active apt repos in: /etc/apt/sources.list 
           Active apt repos in: /etc/apt/sources.list.d/brave-browser-release.list 
           1: deb [signed-by=/usr/share/keyrings/brave-browser-archive-keyring.gpg] https://brave-browser-apt-release.s3.brave.com/ stable main
           Active apt repos in: /etc/apt/sources.list.d/debian-stable-updates.list 
           1: deb http://deb.debian.org/debian/ bullseye-updates main contrib non-free
           Active apt repos in: /etc/apt/sources.list.d/debian.list 
           1: deb http://deb.debian.org/debian/ bullseye main contrib non-free
           2: deb http://security.debian.org/debian-security/ bullseye-security main contrib non-free
           Active apt repos in: /etc/apt/sources.list.d/mx.list 
           1: deb http://mxrepo.com/mx/repo/ bullseye non-free main
Info:      Processes: 259 Uptime: 11d 17h 29m wakeups: 9 Memory: 7.68 GiB used: 3 GiB (39.1%) 
           Init: SysVinit v: 2.96 runlevel: 5 default: 5 tool: systemctl Compilers: gcc: 10.2.1 
           alt: 10 Client: shell wrapper v: 5.1.4-release inxi: 3.3.06 
Boot Mode: UEFI

I just took ONE WEEK 24/7 to rsync 65 GB to the Kingston flash drive given in above system info, plugged into a blue "S/S" usb port (avg tfr rate ~ 0.5 MB/s).

Then I tried the echoes to /proc/sys/vm/dirty... suggested above. It increased the speed to 13.5 MB/s for rsync on a single .deb file. Unfortunately when I did the same rsync transfer with the default values (dirty_bytes=20 000 000 dirty_background_bytes = 0) it went at 7 MBPS. The multiple file transfer is apparently 10x slower than a single file.

13.5 MBPS still sucks. Claimed transfer rate for this USB3.2 Gen 1 drive is 5 GbPS ~ 600 MBPS. It is a factor 40 slow. Later in the week I will try the same drive on a Windoze machine at work and report the result here.

Does anyone know whether Kingston flash drives are slow? I don't do this often enough to spend a lot of time and money trying different brand name drives and some earlier posts suggest that's a waste of time anyhow.

ANy other suggestions welcome.

[j2mcgreg]

@theprofa325

Don't tack on to an old topic. Instead start a new one.

[Eadwine Rose]

Topic split off, so fixed now, but don't dig up old topics next time.

[TTwrs]

You're not the only one to such results...
https://www.amazon.com/Kingston-DTDUO3CG3/dp/B0B1W53H1W
https://www.amazon.com/gp/customer-revi ... D2HFSTIYHO

As a possible explanation, see my comment here...
viewtopic.php?p=600275#p600275

You might will be better off with one of these instead (fast, fast, fast)...
https://www.amazon.com/AXE-MEMORY-Super ... B09TTXS151

Detail: LABEL=AXE-DRIVE TYPE=exfat
Vendor: AXE | Model: USB_DISK_3.2 gen 2 -- Portable SSD
Drive: 465.8GB | Partition: 466GB | Free: 466GB | Blocksize: 256KB
Test filesize: 1GB | Loops: 1 | Rlimit: 120s (4k:60s) | Q1T1
R/W tests are 1.0MB-Sequential and 512kB, 128kB, 4.0kB Random
1.0MB Read: 369 MiB/s (387 MB/s) Write: 579 MiB/s (607 MB/s)
512kB Read: 328 MiB/s (344 MB/s) Write: 540 MiB/s (566 MB/s)
128kB Read: 214 MiB/s (225 MB/s) Write: 472 MiB/s (495 MB/s)
4.0kB Read: 24.0 MiB/s (25.2 MB/s) Write: 90.7 MiB/s (95.1 MB/s)
Test date: 2023-06-22 06:30:59 | fio version: 3.25 | Total test-time: 76sec.

You could try CrystalDiskMark (Windows) or KDiskMark (Linux) for testing.

[Mauser]

You are not alone on this one because I and many others are experiencing this when it comes to Linux. As for if this happening on Windows or not I can't comment on it since I haven't used Windows since October 2015 with a very few, very rare occasion. I gave up on finding a solution to this issue and have accepted very slow USB transfer speeds as "it is what it is."

[figueroa]

It's probably the USB flash drive itself. All makers stumble from time to time, some more than others. I've had satisfactory results with the Samsung Bar Plus flash drives, all sizes. They are currently still available and decently priced. You never get the advertised speed on a sustained basis.

I always test new flash memory for counterfeit with f3 (f3write, f3read, f3probe). See https://fight-flash-fraud.readthedocs.i ... index.html

[theprofa325]

Thanks, folks for all the help. I have tried 4 USB devices now, with results that really surprised me but seems to put the topic to rest.
Using rsync -v to transfer a single 1.4 GB file. Using the dirty_... values suggested in old thread.
Kingston DT microDuo 3C 13.5 MBPS
Generic freeby from oscilloscope company: 12.3 MBPS
Generic "NXT" branded stick: 33.4 GBPS
Seagate 3eeap6-500 external spinning hard drive with USB-A to micro-B cable: 102 MBPS

What I get from this-
* apparently usb sticks do differ significantly in speed ON LINUX. I'll try this again on Windoze this week.
* even the fastest device does not approach 5 BbPS ~ 600 MBPS.

[m_pav]

I disagree with that, a slow USB device is slow, period, irrespective of what OS is running the computer. Linux just makes it VERY clear that writing to the drive is very slow and most people running Windows or Apple simply do not know the difference so whatever it takes is whatever it takes and they just drink it up without any critique. FAT32 is a terrible file system, and it's the slowest of all the major types in use today.

I dealt with this so many times when I operated an IT shop that it was not funny. Small businesses using Sandisk Cruzer Flash drives and other el-cheapo chinese junk drives, esp those promo swivel drives with supplier logos on the steel band, were consistently getting into trouble with failed writes to these horrid devices.

When you fail to do your research and trust the advertised major spec, you'll swallow anything that's dished up to you. So many flash drives are labelled as USB3.2 but in reality, they barely make USB2.0 speeds which have a theoretical maximum of 480mbps (60MBps), of which you're incredibly lucky if you get 28MBps write speeds and 40MBps read. The better USB Flash drives might make 65MBps at a burst, but they tail of very quickly to 42 or less, that is in all known OS's.

The key thing to respect here is the drive storage format type has a massive part to play in it's performance. With the hardware being identical and capable, as I will show soon, you simply can not expect a Windows formatted drive to perform at its best when connected to a Linux Host. Apple knows this, so they refused to allow NTFS support in their OS, smart move. Linux Supports NTFS because we're all inclusive, but it's not native to Linux and it will never perform as well as it would when connected to its native host OS. To compare like for like, it is necessary to use format types that are native to the OS when doing speed comparisons, keeping in mind the ability of the drives internal firmware to accept the format type in use, else the test is a complete failure before it's even started. That is why it is imperative that only quality flash drives and external media are used as these will have a more complete firmware load and superior data handling capabilities. Anything else provides a false economy.

I have seen "expert" Linux reviewers publish articles claiming 20 minutes to make a Live USB using MX Live USB Maker. I no longer have the acutal data, but from memory, I get between 2.5 to 5 minutes on most half decent USB Flash drives, but I get a consistent 11 seconds to make a MX23.2 Xfce Live USB on my Netac 128Gb USB-SSD Flash drive. Keep in mind this includes erasing the drive no less than 2x, assigning and creating 2 partitions that combined consume only 15% of the drive space, unpacking the ISO and writing it to the drive, marking the drive as bootable, then writing the logfile to my host machine. This is using my daily driver which only has USB3.0 ports, I'd expect greater speeds from a machine running USB3.2 ports.

I proved the 11 seconds by ripping the drive out the USB port as soon as the process completed, sticking it into another machine and booting it with the option to do the checksum tests, all passed and the boot completed without error. On these drives, of which I have 3, I create a data partition using the ExFat File System. It's not the fastest, but it performs relatively equally well wherever I use it, and that's everywhere. There is only a minor difference between all 3 major OS's in its overall speeds, almost indecipherable.

Having said all that, I suggest you try for a week or two one of the Liquorix Kernels. Your laptop, like mine is a little exotic, in that not that many were made as compared to consumer level machines. The more exotic machines tend to have a slightly higher percentage of issues with Linux. With the stock Debian kernel, mine has sleep/wake issues tied to the TPM chip that strangely enough affect the USB ports, they suddenly slow to a crawl. The Liquorix kernels don't put a stop to it, but I get 3-4 x longer power-on times and sleep/wake events before the USB degradation happens and my graphics capabilities are significantly improved.

[theprofa325]

New data- it is a combination of the drive and the file system.
Bought a new 256 GB Kingston Data Traveler Max Type A based on an internet speed test page that I can no longer find. They claimed over 900 MB/s.
Copying a single 2.2 GB file onto the as-delivered vfat formatted with rsync -v gave 318 MB/sec.
Reformatting to ext4 using MX formatting tool gave 683 MB/s.

[DukeComposed]

New data- it is a combination of the drive and the file system.
Bought a new 256 GB Kingston Data Traveler Max Type A based on an internet speed test page that I can no longer find. They claimed over 900 MB/s.
Copying a single 2.2 GB file onto the as-delivered vfat formatted with rsync -v gave 318 MB/sec.
Reformatting to ext4 using MX formatting tool gave 683 MB/s.

It's been a long time since I've worried about file system performance comparisons, but apparently it's a known issue that FAT32 is weak at copying many small files. This hearsay detail doesn't match your rsync test, but I think it's indicative that the choice of file system can have an impact on your drive's throughput.