USB transfer speeds slow/ data needs to be written message

[JayM]

Conclusion: Linux stinks for file transfers.
Hoping someone will find and fix the USB issues Linux (ALL) have.
Haven't found a version of Linux yet that can hold a candle to Windows transfer speeds.

That's funny, it seems to work okay for everybody else.

As you haven't posted your Quick System Info in this thread please do so. You can find instructions by clicking the link in my signature.

[crazysquirrel]

System: Host: <filter> Kernel: 4.19.0-12-amd64 x86_64 bits: 64 compiler: gcc v: 8.3.0
parameters: BOOT_IMAGE=/vmlinuz-4.19.0-12-amd64
root=UUID=<filter> ro fromhd=UUID=XXXXXXXX splash
Desktop: Xfce 4.14.2 tk: Gtk 3.24.5 info: xfce4-panel wm: xfwm4 dm: LightDM 1.26.0
Distro: MX-19.2_x64 patito feo May 31 2020 base: Debian GNU/Linux 10 (buster)
Machine: Type: Desktop Mobo: ASUSTeK model: P7H55-M PRO v: Rev 1.xx serial: <filter>
BIOS: American Megatrends v: 1709 date: 01/04/2011
CPU: Topology: Dual Core model: Intel Core i5 660 bits: 64 type: MT MCP arch: Nehalem
family: 6 model-id: 25 (37) stepping: 5 microcode: 7 L2 cache: 4096 KiB
flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 26748
Speed: 3477 MHz min/max: 1200/3334 MHz boost: enabled Core speeds (MHz): 1: 3567
2: 3477 3: 3513 4: 3457
Vulnerabilities: Type: itlb_multihit status: KVM: Split huge pages
Type: l1tf mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable
Type: mds
status: Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable
Type: meltdown mitigation: PTI
Type: spec_store_bypass
mitigation: Speculative Store Bypass disabled via prctl and seccomp
Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer sanitization
Type: spectre_v2 mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW,
STIBP: conditional, RSB filling
Type: srbds status: Not affected
Type: tsx_async_abort status: Not affected
Drives: Local Storage: total: 3.70 TiB used: 165.06 GiB (4.4%)
ID-1: /dev/nvme0n1 vendor: Western Digital model: WDS500G3X0C-00SJG0 size: 465.76 GiB
block size: physical: 512 B logical: 512 B speed: 31.6 Gb/s lanes: 4 serial: <filter>
rev: 102000WD scheme: MBR (This is where my linux install is but had to put /boot on another hard drive since Mobo won't boot off NvMe drive)
ID-2: /dev/sda vendor: Seagate model: ST2000LX001-1RG174 size: 1.82 TiB block size:
physical: 4096 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter>
rev: SDM1 scheme: MBR
ID-3: /dev/sdb vendor: Hitachi model: HTS543232A7A384 size: 298.09 GiB block size:
physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter>
rev: A60W scheme: MBR
ID-4: /dev/sdc vendor: Seagate model: ST3250318AS size: 232.88 GiB block size:
physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 7200 rpm serial: <filter>
rev: CC46 scheme: MBR
ID-5: /dev/sdd vendor: Seagate model: ST9500325AS size: 465.76 GiB block size:
physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter>
rev: SDM1 scheme: MBR
ID-6: /dev/sde vendor: Hitachi model: HTS545050B9A300 size: 465.76 GiB block size:
physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter>
rev: C60F scheme: MBR
Partition: ID-1: / raw size: 32.00 GiB size: 31.25 GiB (97.65%) used: 9.02 GiB (28.9%) fs: ext4
dev: /dev/nvme0n1p1
ID-2: /home raw size: 433.76 GiB size: 425.95 GiB (98.20%) used: 155.97 GiB (36.6%)
fs: ext4 dev: /dev/nvme0n1p2
ID-3: swap-1 size: 8.00 GiB used: 64.0 MiB (0.8%) fs: swap
swappiness: 15 (default 60) cache pressure: 100 (default) dev: /dev/sda3
Sensors: System Temperatures: cpu: 48.0 C mobo: N/A
Fan Speeds (RPM): N/A
Info: Processes: 250 Uptime: 2h 46m Memory: 7.59 GiB used: 2.67 GiB (35.2%) Init: SysVinit
v: 2.93 runlevel: 5 default: 5 Compilers: gcc: 8.3.0 alt: 8 Shell: quick-system-in
running in: quick-system-in inxi: 3.0.36

Report copied to system clipboard

Press any key to close

Edited out a few things that have no bearing on transfer speeds.

I have a dozen flash drives (different brands, different vendors) and 6 external drives - ALL of them have very slow write speeds.
Note: I just reformatted and reinstalled with the latest (19.2 xfce) version yesterday.
Still no speed increase.

Every computer I ever put linux (any version, any vendor) on has had this problem with gawd awful SLOW USB/external transfer speeds (compared to XP and Win7 for example).

Is there a way to turn off write caching for USB and not reset on reboot?

First 60 MB roughly loads very fast (which is tolerable) then slows down to a crawl (1-5 mb/s) for hours (which is INTOLERABLE).
The more/bigger the files the slower it gets. Transferring files one at a time just is not practicable.
I did notice that Linux Mint (current version, cinnamon) will queue files as added. This seems to help slightly but not by much.

Flash drives formatted to FAT32, NTFS,eXFAT, EXT4 at various times. Makes no discernible difference.
External drives formatted to NTFS via Windows.

I even try transferring files by using a live version of linux. No difference.

Side note: I do not suggest installing fmovies app via browser. You can not get rid of it without R/R your setup hence the main reason to R/R my setup)

Going bald pulling my hair out trying to fix this problem.

[crazysquirrel]

Here is an interesting article:

https://help.marklogic.com/Knowledgebas ... ataordered

[JmaCWQ]

I've got into the habit nowdays of using Grsync for most copy operations.
It seems to do that job at the maximum speed possible on Linux and when it's finished the operation there's no waiting for something to complete copying from a buffer or anything, things usually unmount straight away.

I've seen variations of speed at times, sometimes USB 3 devices would slow to a crawl for a while, then speed up again for no explainable reason.
Discovered eventually it was something to do with the port/s I was using on my particular machine, it has USB 3 ports front and rear and the front ones occasionally don't seem to work correctly.

For testing read/write speeds on USB devices I use 'f3', can't remember if it's installed by default or not.
It seems to be fairly accurate with regards to read/write speeds as the speeds are usually pretty much the same when using Grsync on the same devices.
In Terminal 'f3write /mountpoint/diskname', once that's finished 'f3read /mountpoint/diskname'.
It gives a fairly accurate idea of the device's read/write speeds and in some instances explains why it seems to take so long for copying to complete as that's limited to the slow write speed of the device itself.

Here's one I just did on a 4GB cheap Fleabay USB stick:

Code: Select all

  $ f3write /media/home/DISK_IMG/
Free space: 3.68 GB
Creating file 1.h2w ... OK!                        
Creating file 2.h2w ... OK!                        
Creating file 3.h2w ... OK!                        
Creating file 4.h2w ... OK!                           
Free space: 0.00 Byte
Average writing speed: 2.82 MB/s


 $ f3read /media/home/DISK_IMG/
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
Validating file 2.h2w ... 2097152/        0/      0/      0
Validating file 3.h2w ... 2097152/        0/      0/      0
Validating file 4.h2w ... 1426576/        0/      0/      0

  Data OK: 3.68 GB (7718032 sectors)
Data LOST: 0.00 Byte (0 sectors)
	       Corrupted: 0.00 Byte (0 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 29.91 MB/s 

[crazysquirrel]

Average writing speed: 2.82 MB/s
That is HORRIBLE for a write speed.

USB 2.0 should be about 15-20 times that and USB 3.x 30 times or so.

Unless grsync has a GUI I am unable to use it.
Command line gives me a brain cramp. Raises my blood pressure. Ever since I wiped out 10 years of data by using command line.

Also, does grsync need a blank flash drive or ?

A good test for any flash or external drive is to copy an ISO of star wars, 4.3 GB. Watch burst speed and such. Copy the normal way not the grsync method. Then do it again with grsync.
Next test is to copy 100 GB of something like a folder full of pictures or movies.

I copy sometimes a TB of data via USB. Depends on if I get a new external drive as a present :-)

[wdscharff]

Sometimes I wonder why others have such problems. Before I used Linux, or even when I still used win7/linux in dual boot, I had a few, small, cheap USB sticks. Since I only use Linux (from the beginning of 2015) only USB3 and of course I pay attention to the data transfer rate, which is mostly directly related to the price. Then there are USB 3 sticks which are nominally hardly faster than USB 2 sticks.
for snapshots, i use several Sandisk Extreme USB 3.0. according to mx life usb maker, writing a 21 gb snapshot took 3:50, which is ~90mb/sec,. reading, using Thunar to copy the content of the stick to an internal SSD, was about 240MB/sec (current value of this morning)
My SanDisk Ultrafit USB 3.0 for example is a good 1/3 slower in read&write (which was about the price difference at that time).
The values were similar on all my PCs, the two older ones (core i5 & core i7 from 2011/2012) were limited to 120mb/sec for USB 3, the data bus did not give more.

But the slowest is my external USB3.0 WD hard drive WD Elements 25A1, which oscillates at best up to 50mb/sec for large files and collapses at lower USB2 speed of 1-5mb/sec for small files. Mind you, this is the same USB3 port where my Seagate Barracuda writes values between (worst case) 60mb/sec and 200mb/sec.
Can I see /home, /video, /pictures, /music every time I do a total backup of my /home (350gb) , the rest is just a tragedy.
But it was nice and cheap, no 100€ for 4TB was that cheap in 2017.
But now it's not the USB interface, but the hard disk. Taken out of the external case and in the PC SATA port it was not faster.

[JmaCWQ]

I believe it mostly comes down to the write speed of the destination device, slow write speed to device = slow total transfer time.
Many small files, like browser profile or lots of images or something usually slow quite considerably the speeds.

Writing to a USB 2 harddrive I get usually around 25 - 30 MB/s, to a USB 3 harddrive, depending on what machine I use, I see anywhere from around 70 - 120 MB/s.
Any USB 2 sticks I've owned have never achieved anywhere near 30 MB/s write speeds that I can recall, haven't used a USB 3 capable stick, only drives.
Different machines (all with MX) seem to transfer data to the same USB 3 device at different write speeds, no idea why, USB 2 seems mostly consistent across machines.
I watched it when performing that f3 testing I pasted above, most of the time it showed write speed of about 5 MB/s, but for a few minutes it dropped to 300 to 400 KB/s, dropping the average.

Grsync is the GUI version of Rsync, available in the MX Package Installer, Stable Repo tab.
No Grsync doesn't need an empty device to copy to, only requirement is having enough room on the destination device.
It can pause/resume operations, even after a shutdown/restart or whatever it can resume from where it left off.

I used f3 to test the speeds from the command line and I'm pretty sure I read somewhere there's a GUI available for it nowdays.
And even from the command line it's a very basic couple of commands to test write and read speeds.
f3write /media/home/DISK_IMG/ was the first one to test write speed, which is simply f3write followed by the mount point (/media/home) and device name (/DISK-IMG/).
f3read /media/home/DISK_IMG/ is the one to test the reading speed, same deal, f3read followed by mount point/device name.
It really is that simple.

f3write fills the device with files to see if it can all be written to, and at what speed.
f3read then checks if the read data is the same as the written data, checks it for corruption etc. etc..
I believe it was initially created to test if the capacity of a device was as it's advertised, I'm sure we've all heard stories of buying a USB stick cheap somewhere that's supposed to be 'X' number of GB when in fact it's only 'Y' number of GB and a scam by the seller, f3 = Fight Flash Fraud.
The files it writes to whatever device to test need to be deleted manually after testing is finished, not sure about the GUI version.

Why I use Grsync for transferring & f3 for testing is so I don't have to sit there and watch them while the process is going on.

[crazysquirrel]

Ok will check into grsync.

I have flash drives as cheap as $5 and others up to over $100. I even have a metal USB 3.0 flash drive.

External drives from $60 to well over $200

On large file transfers, my burst speed is fine. But then it slows down to a crawl. Somewhere around 60 MB of the transfer is when it slows way down.

Currently using a 5TB WD Black external WITH the 'Y' adapter to get it more power. One side of the 'Y' Adapter is even plugged in externally to a USB power supply (other end has to go into the computer).

Best sustained speed was right about 30MB/S then it slows down.
Typical speed between 1MB/S to 10 MB/S.

One time estimate on the transfer said a whopping 58 HOURS for 50GB of data.

Oh and having all kinds of problems with right click copy/paste too. Going to try a different clipboard manager to see if that fixes the problem.
Seems clipboard has a mental problem and keeps forgetting what I right clicked and copied on. It forgets somewhere between the copy and the paste.
Have to do that multiple times to get it to take.

I have 4 different computers and ALL show the same problem with slow file transfers.
Even my very fastest one still has slow speeds so it must not be my computers.

I have 5 copies of files that I need to condense and eliminate 4 of the copies.
Takes a lot of time too going one by one to ensure I don't lose anything.
And that takes a long time to transfer back and forth.
About 5TB of data in all.
Once I get it all done I can make a single backup copy from one drive to another.
But slow transfer speeds will make this take months to a year to get it done.
Going to try the grsync tonight.

[JayM]

I think it's time for you to post your Quick System Info. See the link in my signature for detailed instructions.

[crazysquirrel]

Trying grsync right now.
Lowest was 0.62kbps
Highest was 29.83 mbps
Typical is around 13,bps
says I need 733 more minutes to copy 232GB of data.