Page 1 of 1

USB transfer speeds slow/ data needs to be written message

Posted: Thu Jan 03, 2019 8:27 pm
by ronjnk
I have a weird problem with USB thumb drives. This proiblem occurs regardless of which thumb drive I utilize. This is a relatively new MX 18 install within the last couple weeks and this is my first attempt to write to them since installation. I had no problem transferring data that was on the USB drives to my computer but writing is brutally slow. I should mention that I have had no problem reading and writing to my WD 500Gig hard drive. That's fast both ways. Read/write.

It took about 5 minutes to transfer 1 gig of data to my thumb drive. It is formatted FAT32. So I reformatted to NTFS. Transfer speed increased for the 1 gig transfer to under 15 seconds. Then I unmount and it says "There is data that needs to be written to the device before it can be removed. That process takes 5 minutes or more to complete. I have no idea what it is doing but the black message box associated with it sits prominently on the screen blocking some of the screen until it completes the unmount.

I tried to transfer 7 gigs of data to the same USB drive. It starts out transferring fast but after a few seconds, I can see it start to slow down and the calculated minutes rise. It starts at over 125 MB/s and quickly drops to 12 MB/s. The first gig transfers fast and then takes roughly 5 minutes to transfer the last 6 gigs. Then I try to unmount but it sits there for many minutes with the "writing data to the device" message. I've never seen this problem on any windows or linux I've used before. But this is a new computer. I suspect it is a settings or formatting issue. Format is msdos and NTFS or FAT32. I've used my windows laptop to reformat as well as Gparted. I've used different USB devices and different USB slots on the computer. The behavior is the same.

Why does data transfer fast initially and then bog down to turtle speed? Fat 32 is even slower which is why I went with NTFS format. And why when I try to unmount is it writing data for a long time? Is it related to this bug? https://bugs.launchpad.net/archlinux/+s ... ug/1762595 Any idea how to fix this? Thanks! Ron

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Jan 03, 2019 9:16 pm
by BitJam
Most of these problems are related to a minor bug in Linux. The default buffer size is gianormous. So writing at first is fast because it is mostly writing to RAM until the buffer fills. Then it takes a long time to do the rest of the writing because after the buffer is full, the speed is limited by the speed of the device.

I always run the "sync" command before I unmount and remove a usb stick. This makes sure all the writing gets done to the actual device.

To fix the buffer problem, I suggest you run the command:

Code: Select all

sudo sysctl vm.dirty_bytes=20000000
You could also add this to your /etc/rc.local script without the "sudo".

Our live-usb-maker programs do this and it actually increases the transfer rate by about 10%. We originally did it so the progress bar wasn't fooled by the huge Linux buffer. Using the default buffer size, the bar would say the whole thing was done in 10 or 20 seconds then you would have to wait about a minute for it to actually finish.

If you reduce the Linux buffer size then the writing should happen at a more constant speed and you will seldom have that big wait at the end for the writing to complete. You can usually just wait a second after the write command is done and you are good to go without running "sync".

There is a related problem with some usb sticks. They can have their own internal RAM buffer. So the first writes are very fast because they go to RAM but then when that RAM buffer gets full, the writing is limited by the speed of the flash memory which is much slower. Often these usb sticks with built in RAM buffers post really great specs even though their flash memory is slow.

I have to system monitors running (conky and gkrellm) and they each show me the disk activity in a rolling graph. That helps me understand what is going on. For usb sticks with RAM buffers you can see the early fast writing and then see it slow down. When it suddenly drops to near zero (assuming nothing else big is going on) then it is safe to unmount the drive.

HTH

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Jan 03, 2019 9:29 pm
by timkb4cq
A lot does depend on the flash drive. Most write far slower than they read While there are USB-3.0 flash drives that will write at over 200MB/s if you just go to the store and pick up a cheap Sandisk or Kingston you'll often get one that will only write at 4 - 10 MB/s with sequential writing like transferring an ISO being faster than writing a bunch of files.
4MB/s = 240MB/min so over 4 minutes to transfer 1GB and half an hour for 7GB.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Jan 03, 2019 10:37 pm
by Eadwine Rose
I have seen this a few times with my sdcard.. basically I need to close all windows that had anything to do with the transfer before doing the unmount and then things happen properly and normally.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Fri Jan 04, 2019 8:35 am
by ronjnk
Good Morning Folks,

Thank you very much for the feedback.

BitJam...Right on. Your explanation of what is happening makes a great deal of sense. I was so baffled as to the initial speed and then slowing down. It was only filling up a buffer for the most part. And then when trying to unmount, I received that lengthy message that it needed to write more data and that was simply the system still writing data to the USB drive. So my assessment and worry that I had something messed up was wrong.

The suggested command you gave me worked like a charm.

Code: Select all

sudo sysctl vm.dirty_bytes=20000000
I ran your command and then transferred 7.7 gigs to USB, when finished, I hit unmount and it was immediately unmounted. Done deal. No message.

I have a couple of questions. Is that command permanent now in my system? In other words, I don't need to open terminal and input that code every time I want to make a large file transfer to USB? I did not use and am not familiar with the sync command. I don't think it is necessary in my case since the behavior seems great now, but for future reference and to perhaps help others that browse this thread, could you explain further how to use the sync command and what it is actually doing? Also, you wrote
You could also add this to your /etc/rc.local script without the "sudo".
I'm uncertain how to properly do that and I'm wondering if I really even need to do that unless the above command is only temporary and disappears once I restart the computer. Thank you BitJam!

timkb4cq... You were right! The speeds I saw were typical. I proved it by transferring the same data from a windows computer and the speeds were the same. What really threw me off was the initial speed which was only filling a buffer ( I had no idea) and then the "writing data" message which lasted so long. Although I tried different USB devices and drives, they are all usb 2.0. Thanks for your input!

Eadwine Rose...I actually thought I had another problem. When I saved to USB, unmounted and then reinserted my USB, many times when I right clicked on the drive, various commands like rename, copy, delete were grayed out and unavailable. Then I realized I had several Thunar file system windows open. It didn't like that. As soon as I closed any open file system windows, all the commands were available. So good advice as a general rule to close any file system windows. Thanks for your input!

Have a great day! Ron

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Sep 24, 2020 10:24 pm
by crazysquirrel
After reading the fixes, I wonder why MX doesn't have them preset on initial install.

I am having slow speeds too.
WD Black External 5TB USB 3/0 connected to ASUS x401A USB port with included cable. Laptop has a 2TB hybrid hard drive, 8GB Ram.

Transfer speed started out at a paltry 60MB/S then dropped to 50 and I expect it to drop to near zero by morning like it did a few days ago.
60MB/S is like 1/10th the speed it should be getting.

I tried all this a few days ago and suddenly the computer wouldn't dual boot back to win7 (Dual booting Win7 and MX 19.2 X64) And MX took 5 min to boot.

Had to R/R the whole drive and start all over without getting any updates for either.
I pay by the GB for data and cannot afford to experiment.
I installed win7 first and did some basic scan disking.
No problems found so far.
I did not do a thorough (surface) scan disk as that would take days to do.

When reinstalling MX 19.2, I did check the box for checking for bad blocks. None found.

Oh and cpu temps SOARED to an average 150F !

Only thing plugged in is the external drive and have the screen turned off to reduce heat.

Side note: ASUS sent me an email stating that the cpu on all their laptops are soldered onto the motherboard.

I also discovered that most external hard drives now are not 'harvestable' due to the USB connector soldered onto the hard drive.

Seems ANYTHING to force you to spend more money.

I would need step by step EXACT steps on what to do (assume nothing please) on how to change my files to speed things up.
I do not have a swap file - had one the first time and none this time.

Exact meaning:
Left click on (whatever)
Scroll till you see (whatever) and left click on that.
If I need to choose what to open something with I need that info too.
What to type and where to put it.
Step by step. Help manual does not have sufficient information. Like to NEVER find the screensaver...

For me, MX is like visiting a foreign country (USA vs UK for example - similar but not the same).

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Sep 24, 2020 10:30 pm
by JayM
According to the specs the Asus x401a has a USB2.0 port and a USB3.0 port. Make sure you're using the right one.

Please start new threads for your other issues and include your Quick System Info.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Sep 24, 2020 10:37 pm
by crazysquirrel
Yes I am. The USB 2 port is where you plug in external power. USB port opposite corner and blue. External plugged into the 3.0 port.

Oh and the transfer is transferring FROM the external drive.
Similar experience transferring to the drive last week.

Program reports a whopping 5 hours to transfer. By morning I expect it to say 1822 hours with 2/3 more to go yet like it did last time.
No kidding either.
It takes a lot longer than what is reported.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Sep 24, 2020 10:40 pm
by JayM
Sounds like you either have a bad laptop or a bad external drive or USB cable. There's not much that MX Linux can do about hardware issues. And if your system is getting as hot as you say it is that's probably the cause of the slowdown due to thermal throttling of the CPU. Is the fan working? It may be clogged with dust. Search the web for info about how to clean it.

Edit: 150F is only 65C which isn't all that hot and is within normal range for most CPUs. I don't know about yours as you haven't supplied your Quick System Info when asking for help (as per the forum rules) so I don't know what your CPU is.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Sep 24, 2020 10:54 pm
by crazysquirrel
HD and cable are brand new (less than 2 weeks old).
I do feel heat coming out of the side.
We did clean it about a year ago and even then it was not dirty.
I will have to wait till tomorrow when it is done to take it apart and check.
PS it is now down to 17MB/S and dropping.
200GB transferred out of 1.5 TB.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Fri Sep 25, 2020 5:49 am
by tony37
crazysquirrel wrote: Thu Sep 24, 2020 10:54 pm HD and cable are brand new (less than 2 weeks old).
It may happen to new hard drives that they are, well, rubbish. Like recently here: viewtopic.php?p=599328#p599328

Re: USB transfer speeds slow/ data needs to be written message

Posted: Fri Sep 25, 2020 9:05 am
by paul1149
crazysquirrel wrote: Thu Sep 24, 2020 10:54 pm HD and cable are brand new (less than 2 weeks old).
Can you try the drive in another unit? Or try the transfer over LAN with the drive attached either to the router or another unit?

Re: USB transfer speeds slow/ data needs to be written message

Posted: Fri Sep 25, 2020 10:17 am
by crazysquirrel
Update:
CPU temps @ 104F
Transfer speed 187K
Estimated time to complete transfer from external to internal - 1114 HOURS.
309GB/1.1TB

If there was a hardware problem, one would think that it would be a constant issue.
But it starts out at 60MB/S then drops overnight to 187K only 1/3 the way through.

Room temperature is 71F

Blood pressure getting sky high...

Re: USB transfer speeds slow/ data needs to be written message

Posted: Fri Sep 25, 2020 7:00 pm
by crazysquirrel
Another update:
I tried transferring files in MZ19-2 KDE on a desktop and got the same problem - slow transfers and on the KDE one, locked up the file browser.
18.3 (non KDE) does not do this. FTN on the 18.3 to be certain.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Fri Sep 25, 2020 8:05 pm
by tascoast
I'd be tempted to create a fresh Live USB of the latest MX and confirm that two different machines struggle with the drive causing problems. That might suggest the drive itself is the issue, new as it may be.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Sat Sep 26, 2020 11:49 am
by TTwrs
With most flashdrive there can be a huge difference between transferring large files and small files, especially when it comes to writes.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Sat Sep 26, 2020 6:34 pm
by m_pav
I think it best that you look into your hardware, there are multiple issues that could be affecting your transfer speeds.
Please ensure your external drive is connected and open gsmartcontrol
Identify the external drive and double click it to open and view its attributes. You will see tabs across the top. If any of those tabs have even a hint of red in the lettering, the drive you selected has a problem that needs investigating. The brighter the red, the worse the issue and you'll have to dig into each tab highlighted with red lettering to see what is being reported. Also, there is no need to run any of the self tests, these are meaningless for this objective.
Repeat and do the same for your hybrid drive.

I highly recommend you educate yourself on how to read each tab, especially the Attributes tab. QSI, do not post back in this thread.
That is step 1. Do not proceed any further within this thread if you have any red lettering displayed in gsmartcontrol. If either of your drives is flagged with red lettering and you want further assistance, please start a new thread and include your QSI as required when asking for assistance. A screen shot would help us to help you with the best advice moving forward.

Step 2 is to see if either drive has any minor inconsistencies, or dirty flags on the storage media. It only takes 1 or 2 files that are in particularly bad shape to cause the exact behaviour you have recorded. If either has inconsistencies, fix them first using their native OS to do the fixing. Fixing a drive partition where the drive has red highlighting in gsmartcontrol can result in permanent data loss, do so at your own risk or let us guide you through it using some of the best tools the industry has which are built into MX Linux.

Step 3 examine the content you are trying to copy. Does the content consist of thousands of small files or are you trying to transfer genuinely larger files, like 200MB+ in size. For example, doing a file copy on 22,000 small files that collectively use 2.2GB of drive space will take a monumentally longer time to transfer than 11 x 200MB files.

Step 4 change the file system. If the external drive is using FAT32, expect all file copy processes to be slow. FAT32 is a very old technology, expect it to behave as an old technology. FAT32 is not a native Linux file system. NTFS has a number of versions, some faster than others, but again, not native to Linux, therefore, as with FAT32, it will be slower than a native Linux file system. In my experience, FAT32 is slower than NTFS and NTFS is about 20-25% slower than ext4.

Step 5 Compare your USB ports output current with that of the external drive. USB2.0 port standards are 5v@ 500ma, USB 3.0 port standards are 5v@ 900ma. If you are connecting an external drive that draws 550ma into a USB2.0 port, it will be in a state of browning and cause read/write failures. A USB 3.0 external drive with a separate USB Power lead plugged into a USB 2.0 port will be expecting a power delivery of up to 900ma, so a USB 2.0 port will not usually provide sufficient current. I have seen all too often faulty brand new twin head USB cables where one head is for data and the other for power.

This is a big post and I've covered a fair bit of stuff but I'm personally leaning towards Step 2 as being the ultimate remedy and hoping for your sake, it's not Step 1.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Sat Sep 26, 2020 10:39 pm
by figueroa
BitJam wrote: Thu Jan 03, 2019 9:16 pm Most of these problems are related to a minor bug in Linux. The default buffer size is gianormous. So writing at first is fast because it is mostly writing to RAM until the buffer fills. Then it takes a long time to do the rest of the writing because after the buffer is full, the speed is limited by the speed of the device.
I thought it was a little strong to call this a bug. I think of it as a tunable parameter, along with a big handful of others related to vm cache. I've been happy enough to leave my vm settings at default and it's sufficient to know what's going on. The USB drive won't actually unmount until data is written. Having lower settings won't make the device write faster.

The user tuning his machine will do well to do an internet search on sysctl vm.dirty_bytes or just sysctl vm.dirty and read through some of those pages.

Notably: https://www.suse.com/support/kb/doc/?id=000017857 is very informative as is https://lonesysadmin.net/2013/12/22/bet ... rty_ratio/. You can set permanent boot-time changes to default settings in /etc/sysctl.conf

Re: USB transfer speeds slow/ data needs to be written message

Posted: Mon Sep 28, 2020 10:06 am
by crazysquirrel
Transfer was completed last night.
1.1TB transferred, 691,000 files.
External drive formatted to EXT4, internal partition formatted (by Windows) NTFS.
External Drive WB Black, 5TB (actually about 4.5TB they lie)
Connection from external to computer was USB 3.0 using supplied cable to the only USB 3 port on that X401a laptop.
Internal drive is a Hybrid drive (Seagate I think), 2TB.

Conclusion: Linux stinks for file transfers.
Hoping someone will find and fix the USB issues Linux (ALL) have.
Haven't found a version of Linux yet that can hold a candle to Windows transfer speeds.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Sat Oct 03, 2020 8:42 pm
by crazysquirrel
Another update:

I found a problem.

Seems the transfer is going to sleep for far too long.

Task Manager reports sleep 20 as being the only thing noticeable that is causing the problem.
When first starting a transfer, it zooms pretty fast, then 20 minutes later the transfer slows then apparently stops.

20 DAYS is far too long. So is 20 hours or 20 minutes. Even 20 seconds is too long.
I have been waiting over 3 hours to transfer three 1.5Gb files to a flash drive. Still far from done.

I do not know how to cancel the sleep function so I can resume data transfer to a USB drive.
I tried Killing sleep in task manager to no avail. Keeps coming right back the moment it is stopped or killed.

https://www.cyberciti.biz/faq/linux-uni ... scripting/

Re: USB transfer speeds slow/ data needs to be written message

Posted: Sat Oct 03, 2020 8:59 pm
by JayM
crazysquirrel wrote: Mon Sep 28, 2020 10:06 am Conclusion: Linux stinks for file transfers.
Hoping someone will find and fix the USB issues Linux (ALL) have.
Haven't found a version of Linux yet that can hold a candle to Windows transfer speeds.
That's funny, it seems to work okay for everybody else.

As you haven't posted your Quick System Info in this thread please do so. You can find instructions by clicking the link in my signature.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Sun Oct 25, 2020 7:09 pm
by crazysquirrel
System: Host: <filter> Kernel: 4.19.0-12-amd64 x86_64 bits: 64 compiler: gcc v: 8.3.0
parameters: BOOT_IMAGE=/vmlinuz-4.19.0-12-amd64
root=UUID=<filter> ro fromhd=UUID=XXXXXXXX splash
Desktop: Xfce 4.14.2 tk: Gtk 3.24.5 info: xfce4-panel wm: xfwm4 dm: LightDM 1.26.0
Distro: MX-19.2_x64 patito feo May 31 2020 base: Debian GNU/Linux 10 (buster)
Machine: Type: Desktop Mobo: ASUSTeK model: P7H55-M PRO v: Rev 1.xx serial: <filter>
BIOS: American Megatrends v: 1709 date: 01/04/2011
CPU: Topology: Dual Core model: Intel Core i5 660 bits: 64 type: MT MCP arch: Nehalem
family: 6 model-id: 25 (37) stepping: 5 microcode: 7 L2 cache: 4096 KiB
flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 26748
Speed: 3477 MHz min/max: 1200/3334 MHz boost: enabled Core speeds (MHz): 1: 3567
2: 3477 3: 3513 4: 3457
Vulnerabilities: Type: itlb_multihit status: KVM: Split huge pages
Type: l1tf mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable
Type: mds
status: Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable
Type: meltdown mitigation: PTI
Type: spec_store_bypass
mitigation: Speculative Store Bypass disabled via prctl and seccomp
Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer sanitization
Type: spectre_v2 mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW,
STIBP: conditional, RSB filling
Type: srbds status: Not affected
Type: tsx_async_abort status: Not affected
Drives: Local Storage: total: 3.70 TiB used: 165.06 GiB (4.4%)
ID-1: /dev/nvme0n1 vendor: Western Digital model: WDS500G3X0C-00SJG0 size: 465.76 GiB
block size: physical: 512 B logical: 512 B speed: 31.6 Gb/s lanes: 4 serial: <filter>
rev: 102000WD scheme: MBR
(This is where my linux install is but had to put /boot on another hard drive since Mobo won't boot off NvMe drive)
ID-2: /dev/sda vendor: Seagate model: ST2000LX001-1RG174 size: 1.82 TiB block size:
physical: 4096 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter>
rev: SDM1 scheme: MBR
ID-3: /dev/sdb vendor: Hitachi model: HTS543232A7A384 size: 298.09 GiB block size:
physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter>
rev: A60W scheme: MBR
ID-4: /dev/sdc vendor: Seagate model: ST3250318AS size: 232.88 GiB block size:
physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 7200 rpm serial: <filter>
rev: CC46 scheme: MBR
ID-5: /dev/sdd vendor: Seagate model: ST9500325AS size: 465.76 GiB block size:
physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter>
rev: SDM1 scheme: MBR
ID-6: /dev/sde vendor: Hitachi model: HTS545050B9A300 size: 465.76 GiB block size:
physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter>
rev: C60F scheme: MBR
Partition: ID-1: / raw size: 32.00 GiB size: 31.25 GiB (97.65%) used: 9.02 GiB (28.9%) fs: ext4
dev: /dev/nvme0n1p1
ID-2: /home raw size: 433.76 GiB size: 425.95 GiB (98.20%) used: 155.97 GiB (36.6%)
fs: ext4 dev: /dev/nvme0n1p2
ID-3: swap-1 size: 8.00 GiB used: 64.0 MiB (0.8%) fs: swap
swappiness: 15 (default 60) cache pressure: 100 (default) dev: /dev/sda3
Sensors: System Temperatures: cpu: 48.0 C mobo: N/A
Fan Speeds (RPM): N/A
Info: Processes: 250 Uptime: 2h 46m Memory: 7.59 GiB used: 2.67 GiB (35.2%) Init: SysVinit
v: 2.93 runlevel: 5 default: 5 Compilers: gcc: 8.3.0 alt: 8 Shell: quick-system-in
running in: quick-system-in inxi: 3.0.36

Report copied to system clipboard

Press any key to close

Edited out a few things that have no bearing on transfer speeds.

I have a dozen flash drives (different brands, different vendors) and 6 external drives - ALL of them have very slow write speeds.
Note: I just reformatted and reinstalled with the latest (19.2 xfce) version yesterday.
Still no speed increase.

Every computer I ever put linux (any version, any vendor) on has had this problem with gawd awful SLOW USB/external transfer speeds (compared to XP and Win7 for example).

Is there a way to turn off write caching for USB and not reset on reboot?

First 60 MB roughly loads very fast (which is tolerable) then slows down to a crawl (1-5 mb/s) for hours (which is INTOLERABLE).
The more/bigger the files the slower it gets. Transferring files one at a time just is not practicable.
I did notice that Linux Mint (current version, cinnamon) will queue files as added. This seems to help slightly but not by much.

Flash drives formatted to FAT32, NTFS,eXFAT, EXT4 at various times. Makes no discernible difference.
External drives formatted to NTFS via Windows.

I even try transferring files by using a live version of linux. No difference.

Side note: I do not suggest installing fmovies app via browser. You can not get rid of it without R/R your setup hence the main reason to R/R my setup)

Going bald pulling my hair out trying to fix this problem.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Sun Oct 25, 2020 7:23 pm
by crazysquirrel

Re: USB transfer speeds slow/ data needs to be written message

Posted: Mon Oct 26, 2020 2:22 am
by JmaCWQ
I've got into the habit nowdays of using Grsync for most copy operations.
It seems to do that job at the maximum speed possible on Linux and when it's finished the operation there's no waiting for something to complete copying from a buffer or anything, things usually unmount straight away.

I've seen variations of speed at times, sometimes USB 3 devices would slow to a crawl for a while, then speed up again for no explainable reason.
Discovered eventually it was something to do with the port/s I was using on my particular machine, it has USB 3 ports front and rear and the front ones occasionally don't seem to work correctly.

For testing read/write speeds on USB devices I use 'f3', can't remember if it's installed by default or not.
It seems to be fairly accurate with regards to read/write speeds as the speeds are usually pretty much the same when using Grsync on the same devices.
In Terminal 'f3write /mountpoint/diskname', once that's finished 'f3read /mountpoint/diskname'.
It gives a fairly accurate idea of the device's read/write speeds and in some instances explains why it seems to take so long for copying to complete as that's limited to the slow write speed of the device itself.

Here's one I just did on a 4GB cheap Fleabay USB stick:

Code: Select all

  $ f3write /media/home/DISK_IMG/
Free space: 3.68 GB
Creating file 1.h2w ... OK!                        
Creating file 2.h2w ... OK!                        
Creating file 3.h2w ... OK!                        
Creating file 4.h2w ... OK!                           
Free space: 0.00 Byte
Average writing speed: 2.82 MB/s


 $ f3read /media/home/DISK_IMG/
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
Validating file 2.h2w ... 2097152/        0/      0/      0
Validating file 3.h2w ... 2097152/        0/      0/      0
Validating file 4.h2w ... 1426576/        0/      0/      0

  Data OK: 3.68 GB (7718032 sectors)
Data LOST: 0.00 Byte (0 sectors)
	       Corrupted: 0.00 Byte (0 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 29.91 MB/s 

Re: USB transfer speeds slow/ data needs to be written message

Posted: Mon Oct 26, 2020 8:24 pm
by crazysquirrel
Average writing speed: 2.82 MB/s
That is HORRIBLE for a write speed.

USB 2.0 should be about 15-20 times that and USB 3.x 30 times or so.

Unless grsync has a GUI I am unable to use it.
Command line gives me a brain cramp. Raises my blood pressure. Ever since I wiped out 10 years of data by using command line.

Also, does grsync need a blank flash drive or ?

A good test for any flash or external drive is to copy an ISO of star wars, 4.3 GB. Watch burst speed and such. Copy the normal way not the grsync method. Then do it again with grsync.
Next test is to copy 100 GB of something like a folder full of pictures or movies.

I copy sometimes a TB of data via USB. Depends on if I get a new external drive as a present :-)

Re: USB transfer speeds slow/ data needs to be written message

Posted: Tue Oct 27, 2020 1:36 am
by wdscharff
Sometimes I wonder why others have such problems. Before I used Linux, or even when I still used win7/linux in dual boot, I had a few, small, cheap USB sticks. Since I only use Linux (from the beginning of 2015) only USB3 and of course I pay attention to the data transfer rate, which is mostly directly related to the price. Then there are USB 3 sticks which are nominally hardly faster than USB 2 sticks.
for snapshots, i use several Sandisk Extreme USB 3.0. according to mx life usb maker, writing a 21 gb snapshot took 3:50, which is ~90mb/sec,. reading, using Thunar to copy the content of the stick to an internal SSD, was about 240MB/sec (current value of this morning)
My SanDisk Ultrafit USB 3.0 for example is a good 1/3 slower in read&write (which was about the price difference at that time).
The values were similar on all my PCs, the two older ones (core i5 & core i7 from 2011/2012) were limited to 120mb/sec for USB 3, the data bus did not give more.

But the slowest is my external USB3.0 WD hard drive WD Elements 25A1, which oscillates at best up to 50mb/sec for large files and collapses at lower USB2 speed of 1-5mb/sec for small files. Mind you, this is the same USB3 port where my Seagate Barracuda writes values between (worst case) 60mb/sec and 200mb/sec.
Can I see /home, /video, /pictures, /music every time I do a total backup of my /home (350gb) , the rest is just a tragedy.
But it was nice and cheap, no 100€ for 4TB was that cheap in 2017.
But now it's not the USB interface, but the hard disk. Taken out of the external case and in the PC SATA port it was not faster.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Tue Oct 27, 2020 2:06 am
by JmaCWQ
I believe it mostly comes down to the write speed of the destination device, slow write speed to device = slow total transfer time.
Many small files, like browser profile or lots of images or something usually slow quite considerably the speeds.

Writing to a USB 2 harddrive I get usually around 25 - 30 MB/s, to a USB 3 harddrive, depending on what machine I use, I see anywhere from around 70 - 120 MB/s.
Any USB 2 sticks I've owned have never achieved anywhere near 30 MB/s write speeds that I can recall, haven't used a USB 3 capable stick, only drives.
Different machines (all with MX) seem to transfer data to the same USB 3 device at different write speeds, no idea why, USB 2 seems mostly consistent across machines.
I watched it when performing that f3 testing I pasted above, most of the time it showed write speed of about 5 MB/s, but for a few minutes it dropped to 300 to 400 KB/s, dropping the average.

Grsync is the GUI version of Rsync, available in the MX Package Installer, Stable Repo tab.
No Grsync doesn't need an empty device to copy to, only requirement is having enough room on the destination device.
It can pause/resume operations, even after a shutdown/restart or whatever it can resume from where it left off.

I used f3 to test the speeds from the command line and I'm pretty sure I read somewhere there's a GUI available for it nowdays.
And even from the command line it's a very basic couple of commands to test write and read speeds.
f3write /media/home/DISK_IMG/ was the first one to test write speed, which is simply f3write followed by the mount point (/media/home) and device name (/DISK-IMG/).
f3read /media/home/DISK_IMG/ is the one to test the reading speed, same deal, f3read followed by mount point/device name.
It really is that simple.

f3write fills the device with files to see if it can all be written to, and at what speed.
f3read then checks if the read data is the same as the written data, checks it for corruption etc. etc..
I believe it was initially created to test if the capacity of a device was as it's advertised, I'm sure we've all heard stories of buying a USB stick cheap somewhere that's supposed to be 'X' number of GB when in fact it's only 'Y' number of GB and a scam by the seller, f3 = Fight Flash Fraud.
The files it writes to whatever device to test need to be deleted manually after testing is finished, not sure about the GUI version.

Why I use Grsync for transferring & f3 for testing is so I don't have to sit there and watch them while the process is going on.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Tue Oct 27, 2020 10:20 pm
by crazysquirrel
Ok will check into grsync.

I have flash drives as cheap as $5 and others up to over $100. I even have a metal USB 3.0 flash drive.

External drives from $60 to well over $200

On large file transfers, my burst speed is fine. But then it slows down to a crawl. Somewhere around 60 MB of the transfer is when it slows way down.

Currently using a 5TB WD Black external WITH the 'Y' adapter to get it more power. One side of the 'Y' Adapter is even plugged in externally to a USB power supply (other end has to go into the computer).

Best sustained speed was right about 30MB/S then it slows down.
Typical speed between 1MB/S to 10 MB/S.

One time estimate on the transfer said a whopping 58 HOURS for 50GB of data.

Oh and having all kinds of problems with right click copy/paste too. Going to try a different clipboard manager to see if that fixes the problem.
Seems clipboard has a mental problem and keeps forgetting what I right clicked and copied on. It forgets somewhere between the copy and the paste.
Have to do that multiple times to get it to take.

I have 4 different computers and ALL show the same problem with slow file transfers.
Even my very fastest one still has slow speeds so it must not be my computers.

I have 5 copies of files that I need to condense and eliminate 4 of the copies.
Takes a lot of time too going one by one to ensure I don't lose anything.
And that takes a long time to transfer back and forth.
About 5TB of data in all.
Once I get it all done I can make a single backup copy from one drive to another.
But slow transfer speeds will make this take months to a year to get it done.
Going to try the grsync tonight.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Tue Oct 27, 2020 10:29 pm
by JayM
I think it's time for you to post your Quick System Info. See the link in my signature for detailed instructions.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Tue Oct 27, 2020 11:49 pm
by crazysquirrel
Trying grsync right now.
Lowest was 0.62kbps
Highest was 29.83 mbps
Typical is around 13,bps
says I need 733 more minutes to copy 232GB of data.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Tue Oct 27, 2020 11:56 pm
by crazysquirrel
JayM wrote: Tue Oct 27, 2020 10:29 pm I think it's time for you to post your Quick System Info. See the link in my signature for detailed instructions.
I already did post it above.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Wed Oct 28, 2020 12:07 am
by JayM
You didn't follow the instructions and that's not your complete Quick System Info. Please edit your post above, delete its existing contents, run Quick System Info, then right-click paste in the post (not copy-paste.)

Re: USB transfer speeds slow/ data needs to be written message

Posted: Wed Oct 28, 2020 6:56 pm
by crazysquirrel
{Start Button}>Mouse over Mx Tools>Quick System Info

Shows the same things I posted.

I am not sure what you want me to do. You said not to copy/paste then tell me to copy it to the post.
Getting a bit frustrated.

Exactly what information specifically are you looking for?
Seems to me all anyone needs is CPU, Motherboard, and Memory.
What else is there?

Edit: I clicked on your name and the link on how to obtain QSI.
It reported the same information I posted before.
I edited out irrelevant information like Network which is not necessary and edited out chip ID's where I saw them, similar unrelated things to the problem.
Those are not necessary.
I avoid providing certain unnecessary things because the internet is what it is.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Wed Oct 28, 2020 7:33 pm
by JayM
crazysquirrel wrote: Wed Oct 28, 2020 6:56 pm {Start Button}>Mouse over Mx Tools>Quick System Info

Shows the same things I posted.

I am not sure what you want me to do. You said not to copy/paste then tell me to copy it to the post.
Getting a bit frustrated.

Exactly what information specifically are you looking for?
Seems to me all anyone needs is CPU, Motherboard, and Memory.
What else is there?

Edit: I clicked on your name and the link on how to obtain QSI.
It reported the same information I posted before.
I edited out irrelevant information like Network which is not necessary and edited out chip ID's where I saw them, similar unrelated things to the problem.
Those are not necessary.
I avoid providing certain unnecessary things because the internet is what it is.
I didn't ask for the full QSI just to be snoopy, I asked for a reason. If you're so sure that the information that you redacted (then posted as plain text in violation of the forum rules, please read them) is unrelated to the problem you must be wiser than I am. Please post the solution after you figure this out and fix it.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Wed Oct 28, 2020 8:33 pm
by crazysquirrel
Here is an excerpt of a portion of the grsync I tried:

cd+++++++++ Win7 iso backup/
>f+++++++++ Win7 iso backup/DVD1.iso
4,159,715,328 100% 19.98MB/s 0:03:18 (xfr#1, to-chk=10/12)
>f+++++++++ Win7 iso backup/DVD2.iso
3,878,873,088 100% 19.89MB/s 0:03:05 (xfr#2, to-chk=9/12)
>f+++++++++ Win7 iso backup/DVD3.iso
3,877,251,072 100% 19.84MB/s 0:03:06 (xfr#3, to-chk=8/12)
>f+++++++++ Win7 iso backup/DVD4.iso
1,277,546,496 100% 19.72MB/s 0:01:01 (xfr#4, to-chk=7/12)
Those speeds are quite slow.

I have no solution. That is why I came here seeking advice.

Please note: if forum rules are too long or complicated then people will shy away from posting.
Developers lose the ability to glean information on potential problem areas.

I do not intend on violating the rules. Been a long while since I read them and post infrequently.
Mods are always free to edit or delete my posts as they see fit. I don't mind at all if they need to.

Some previous information on USB I obtained a few years ago:
One person who was the developer of the USB 'drivers' so to speak for Linux had some issues with certain people and left the development team.
I was hoping that the person's work would have been completed by now.

Most of this may be a moot point anyway as tech is transitioning away from USB to something faster. USB C is just an interim. I suspect a port running on something like PCIE with 16 lanes might come about. It works for NvMe quite well (31.6GB/S speed) with perhaps 4 parallel serial connections to a storage device.

For me, I need it to work every time and as I expect it to work.

When one loses irreplaceable pictures and things due to problems, it does tend to make one unhappy. Recovery software doesn't work well enough to recover the majority of things.
And I have lost quite a bit due to hardware issues and attempting to use command line.

This is my frustration - trying to backup my things without losing anything AND not spend an inordinate amount of time doing it.

My total storage is about 14TB spread out among several drives.
Goal is to condense everything (eliminate the extra copies) onto one drive then back up that drive.

A consistent and fast transfer speed is needed.
Thank you for your suggestions and consideration.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Oct 29, 2020 9:05 am
by JmaCWQ
I think the forum rules part is referring to the pasting of the system info to forum threads.
It's preferred that is posted between code tags, for example [code] relevant text pasted here [/code]
With MX when you run the QSI from MX tools, it auto saves the results to the clipboard including the code tags so when posting it to the forum a right click > Paste is all that's required as the code tags are already saved to the clipboard.
Any unique serials are filtered.
I find it does make threads easier to read.

I have a brand new Toshiba Canvio 1TB USB 3 drive (ext4) that when writing to it from empty to nearly full capacity, for the first third or so it shows reasonable speeds 100+ MB/s, then it slows to around 24 MB/s for most of the remainder, takes hours, and only the last 20 or 30 GB shows speeds of over 100 again, why it does this I have no idea, tried different ports on the computer, tried copying & pasting with Thunar, tried using the cp command in Terminal and Grsync and the results were similar, fast for the first few hundred GB then slows to a crawl for the majority of the rest of the process.
I tried using a USB 3 hub, different USB leads, different computers etc. etc. & nothing seems to make any difference.
At present I believe it's the drive itself, though I haven't tested it properly for speed or badblocks or anything as yet.

They do seem rather slow those speeds you posted, especially if it's via a true USB 3 port to a USB 3 destination device.
I think I would be looking at properly testing the write speeds of the destination device/s.

I hear ya when it comes to data loss, I think we've all been there at one time or another, I have similar amounts of storage scattered all over the place, one thing I did learn the hard way years ago was one backup can sometimes not be enough, now I always keep two separate backups of everything I cannot afford to lose.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Oct 29, 2020 11:41 pm
by crazysquirrel
A person here has the same problem with speed that I do.
Starts out really fast then slows to a crawl.


https://gist.github.com/2E0PGS/f63544f8 ... 4f56efe52f
If your running a x64 bit Ubuntu or other Linux and find USB transfers hang at the end apply this fix:

echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes

I suggest you edit your /etc/rc.local file to make this change persistant across reboots.

sudo nano /etc/rc.local

Go to the bottom of the file and leave a space then paste in those two lines.

Save the file with ctrl + x then press y.

To revert the changes enter this in console and remove the lines in /etc/rc.local

echo 0 > /proc/sys/vm/dirty_background_bytes
echo 0 > /proc/sys/vm/dirty_bytes"
Seems to work but I did not put any of that into any file. Just a terminal.
Speeds are now right at 30MB/S roughly when transferring large files (1.4GB to 4GB roughly).

Going to take me 14 hours to transfer 1.4TB of data :-(

Another site suggested fiddling with the USB stuff in a folder to alter the caching.
I won't fool with things I don't know anything about.
R/R takes a lot of time and costs me over $1.25/GB of data to do.
Don't think I am allowed to sit in McD's for 14 hours to get data....
(exaggeration of course).

Still another site suggested using modprobe twice in a row.

Is this better?

Code: Select all

System:    Host: <filter> Kernel: 4.19.0-12-amd64 x86_64 bits: 64 compiler: gcc v: 8.3.0 
           parameters: BOOT_IMAGE=/vmlinuz-4.19.0-12-amd64 
           root=UUID=<filter> ro fromhd=UUID=F269-E939 splash 
           Desktop: Xfce 4.14.2 tk: Gtk 3.24.5 info: xfce4-panel wm: xfwm4 dm: LightDM 1.26.0 
           Distro: MX-19.2_x64 patito feo May 31  2020 base: Debian GNU/Linux 10 (buster) 

Machine:   Type: Desktop Mobo: ASUSTeK model: P7H55-M PRO v: Rev 1.xx serial: <filter> 
           BIOS: American Megatrends v: 1709 date: 01/04/2011 

CPU:       Topology: Dual Core model: Intel Core i5 660 bits: 64 type: MT MCP arch: Nehalem 
           family: 6 model-id: 25 (37) stepping: 5 microcode: 7 L2 cache: 4096 KiB 
           flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 26751 
           Speed: 2615 MHz min/max: 1200/3334 MHz boost: enabled Core speeds (MHz): 1: 1929 
           2: 1790 3: 2199 4: 2372 
           Vulnerabilities: Type: itlb_multihit status: KVM: Split huge pages 
           Type: l1tf mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable 
           Type: mds 
           status: Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable 
           Type: meltdown mitigation: PTI 
           Type: spec_store_bypass 
           mitigation: Speculative Store Bypass disabled via prctl and seccomp 
           Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer sanitization 
           Type: spectre_v2 mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, 
           STIBP: conditional, RSB filling 
           Type: srbds status: Not affected 
           Type: tsx_async_abort status: Not affected 

Graphics:  Device-1: Intel Core Processor Integrated Graphics vendor: ASUSTeK driver: i915 
           v: kernel bus ID: 00:02.0 chip ID: XXXX-XXXX 
           Display: x11 server: X.Org 1.20.4 driver: intel resolution: 1680x1050~60Hz 
           OpenGL: renderer: Mesa DRI Intel Ironlake Desktop v: 2.1 Mesa 18.3.6 
           direct render: Yes 

Audio:     Device-1: Intel 5 Series/3400 Series High Definition Audio vendor: ASUSTeK 
           driver: snd_hda_intel v: kernel bus ID: 00:1b.0 chip ID: XXXX-XXXX
           Sound Server: ALSA v: k4.19.0-12-amd64 

Drives:    Local Storage: total: 10.07 TiB used: 4.13 TiB (41.0%) 
           
           ID-1: /dev/nvme0n1 vendor: Western Digital model: WDS500G3X0C-00SJG0 size: 465.76 GiB 
           block size: physical: 512 B logical: 512 B speed: 31.6 Gb/s lanes: 4 serial: <filter> 
           rev: 102000WD scheme: MBR 
           
           ID-2: /dev/sda vendor: Seagate model: ST2000LX001-1RG174 size: 1.82 TiB block size: 
           physical: 4096 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter> 
           rev: SDM1 scheme: MBR 
          
           ID-3: /dev/sdb vendor: Hitachi model: HTS543232A7A384 size: 298.09 GiB block size: 
           physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter> 
           rev: A60W scheme: MBR 
          
           ID-4: /dev/sdc vendor: Seagate model: ST3250318AS size: 232.88 GiB block size: 
           physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 7200 rpm serial: <filter> 
           rev: CC46 scheme: MBR 
          
           ID-5: /dev/sdd vendor: Seagate model: ST9500325AS size: 465.76 GiB block size: 
           physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter> 
           rev: SDM1 scheme: MBR 
          
           ID-6: /dev/sde vendor: Hitachi model: HTS545050B9A300 size: 465.76 GiB block size: 
           physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter> 
           rev: C60F scheme: MBR 
           
           ID-7: /dev/sdf type: USB vendor: Samsung model: HD204UI size: 1.82 TiB block size: 
           physical: 512 B logical: 512 B serial: <filter> scheme: MBR 
          
           ID-8: /dev/sdg type: USB vendor: Western Digital model: WD Game Drive size: 4.55 TiB 
           block size: physical: 4096 B logical: 512 B serial: <filter> rev: 5002 scheme: GPT 

Partition: ID-1: / raw size: 32.00 GiB size: 31.25 GiB (97.65%) used: 9.43 GiB (30.2%) fs: ext4 
           dev: /dev/nvme0n1p1 
           
           ID-2: /boot raw size: 2.00 GiB size: 1.91 GiB (95.30%) used: 160.5 MiB (8.2%) 
           fs: ext4 dev: /dev/sda2 
          
           ID-3: /home raw size: 433.76 GiB size: 425.95 GiB (98.20%) used: 157.91 GiB (37.1%) 
           fs: ext4 dev: /dev/nvme0n1p2 
          
           ID-4: swap-1 size: 8.00 GiB used: 1.0 MiB (0.0%) fs: swap swappiness: 15 (default 60) 
           cache pressure: 100 (default) dev: /dev/sda4 

Info:      Processes: 241 Uptime: 6h 04m Memory: 7.59 GiB used: 1.84 GiB (24.3%) Init: SysVinit 
           v: 2.93 runlevel: 5 default: 5 Compilers: gcc: 8.3.0 alt: 8 Shell: quick-system-in 
           running in: quick-system-in inxi: 3.0.36 
Edit:
Here is another possible solution:

"for those who dont have /etc/rc.local can use this:
sudo sysctl -w vm.dirty_bytes=50331648
sudo sysctl -w vm.dirty_background_bytes=16777216"

Another one said to disable swapfile.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Tue Jan 26, 2021 5:14 pm
by gimbleguy
"USB transfer speeds slow/ data"

I second this.

I have

SanDisk usb3.1 Extreme Pro
Samsung T5 SSD USB

File transfers are half the speed compared to a Windows 10 environment.

Linux USB file transfers are abysmal :(

Hope they can fix this soon

Re: USB transfer speeds slow/ data needs to be written message

Posted: Fri Jan 29, 2021 6:49 pm
by gimbleguy
https://youtu.be/lSnPhpxgQIg


SanDisk Extreme Pro USB stick
Something to think about...

Re: USB transfer speeds slow/ data needs to be written message

Posted: Fri Jan 29, 2021 7:54 pm
by BitJam
gimbleguy wrote: Fri Jan 29, 2021 6:49 pm https://youtu.be/lSnPhpxgQIg

SanDisk Extreme Pro USB stick
Something to think about...
The speed comparison is meaningless if you don't drop caches in between the two copies. Data cached from the first copy operation is all stored in RAM (assuming you have enough free RAM). For the 2nd copy the data is written from RAM to the usb device without having to do any reads. Here is how you can drop caches on the command line:

Code: Select all

echo 3 | sudo tee /proc/sys/vm/drop_caches
If you do this before each copy then the test should be fair.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Fri Jan 29, 2021 8:14 pm
by gimbleguy
But what about automation? Why do we need to do that for? We are at the age of AI, but it feels like we are still in the 80s with linux.ugh