USB transfer speeds slow/ data needs to be written message

[crazysquirrel]

I think it's time for you to post your Quick System Info. See the link in my signature for detailed instructions.

I already did post it above.

[JayM]

You didn't follow the instructions and that's not your complete Quick System Info. Please edit your post above, delete its existing contents, run Quick System Info, then right-click paste in the post (not copy-paste.)

[crazysquirrel]

{Start Button}>Mouse over Mx Tools>Quick System Info

Shows the same things I posted.

I am not sure what you want me to do. You said not to copy/paste then tell me to copy it to the post.
Getting a bit frustrated.

Exactly what information specifically are you looking for?
Seems to me all anyone needs is CPU, Motherboard, and Memory.
What else is there?

Edit: I clicked on your name and the link on how to obtain QSI.
It reported the same information I posted before.
I edited out irrelevant information like Network which is not necessary and edited out chip ID's where I saw them, similar unrelated things to the problem.
Those are not necessary.
I avoid providing certain unnecessary things because the internet is what it is.

[JayM]

{Start Button}>Mouse over Mx Tools>Quick System Info

Shows the same things I posted.

I am not sure what you want me to do. You said not to copy/paste then tell me to copy it to the post.
Getting a bit frustrated.

Exactly what information specifically are you looking for?
Seems to me all anyone needs is CPU, Motherboard, and Memory.
What else is there?

Edit: I clicked on your name and the link on how to obtain QSI.
It reported the same information I posted before.
I edited out irrelevant information like Network which is not necessary and edited out chip ID's where I saw them, similar unrelated things to the problem.
Those are not necessary.
I avoid providing certain unnecessary things because the internet is what it is.

I didn't ask for the full QSI just to be snoopy, I asked for a reason. If you're so sure that the information that you redacted (then posted as plain text in violation of the forum rules, please read them) is unrelated to the problem you must be wiser than I am. Please post the solution after you figure this out and fix it.

[crazysquirrel]

Here is an excerpt of a portion of the grsync I tried:

cd+++++++++ Win7 iso backup/
>f+++++++++ Win7 iso backup/DVD1.iso
4,159,715,328 100% 19.98MB/s 0:03:18 (xfr#1, to-chk=10/12)
>f+++++++++ Win7 iso backup/DVD2.iso
3,878,873,088 100% 19.89MB/s 0:03:05 (xfr#2, to-chk=9/12)
>f+++++++++ Win7 iso backup/DVD3.iso
3,877,251,072 100% 19.84MB/s 0:03:06 (xfr#3, to-chk=8/12)
>f+++++++++ Win7 iso backup/DVD4.iso
1,277,546,496 100% 19.72MB/s 0:01:01 (xfr#4, to-chk=7/12)
Those speeds are quite slow.

I have no solution. That is why I came here seeking advice.

Please note: if forum rules are too long or complicated then people will shy away from posting.
Developers lose the ability to glean information on potential problem areas.

I do not intend on violating the rules. Been a long while since I read them and post infrequently.
Mods are always free to edit or delete my posts as they see fit. I don't mind at all if they need to.

Some previous information on USB I obtained a few years ago:
One person who was the developer of the USB 'drivers' so to speak for Linux had some issues with certain people and left the development team.
I was hoping that the person's work would have been completed by now.

Most of this may be a moot point anyway as tech is transitioning away from USB to something faster. USB C is just an interim. I suspect a port running on something like PCIE with 16 lanes might come about. It works for NvMe quite well (31.6GB/S speed) with perhaps 4 parallel serial connections to a storage device.

For me, I need it to work every time and as I expect it to work.

When one loses irreplaceable pictures and things due to problems, it does tend to make one unhappy. Recovery software doesn't work well enough to recover the majority of things.
And I have lost quite a bit due to hardware issues and attempting to use command line.

This is my frustration - trying to backup my things without losing anything AND not spend an inordinate amount of time doing it.

My total storage is about 14TB spread out among several drives.
Goal is to condense everything (eliminate the extra copies) onto one drive then back up that drive.

A consistent and fast transfer speed is needed.
Thank you for your suggestions and consideration.

[JmaCWQ]

I think the forum rules part is referring to the pasting of the system info to forum threads.
It's preferred that is posted between code tags, for example [code] relevant text pasted here [/code]
With MX when you run the QSI from MX tools, it auto saves the results to the clipboard including the code tags so when posting it to the forum a right click > Paste is all that's required as the code tags are already saved to the clipboard.
Any unique serials are filtered.
I find it does make threads easier to read.

I have a brand new Toshiba Canvio 1TB USB 3 drive (ext4) that when writing to it from empty to nearly full capacity, for the first third or so it shows reasonable speeds 100+ MB/s, then it slows to around 24 MB/s for most of the remainder, takes hours, and only the last 20 or 30 GB shows speeds of over 100 again, why it does this I have no idea, tried different ports on the computer, tried copying & pasting with Thunar, tried using the cp command in Terminal and Grsync and the results were similar, fast for the first few hundred GB then slows to a crawl for the majority of the rest of the process.
I tried using a USB 3 hub, different USB leads, different computers etc. etc. & nothing seems to make any difference.
At present I believe it's the drive itself, though I haven't tested it properly for speed or badblocks or anything as yet.

They do seem rather slow those speeds you posted, especially if it's via a true USB 3 port to a USB 3 destination device.
I think I would be looking at properly testing the write speeds of the destination device/s.

I hear ya when it comes to data loss, I think we've all been there at one time or another, I have similar amounts of storage scattered all over the place, one thing I did learn the hard way years ago was one backup can sometimes not be enough, now I always keep two separate backups of everything I cannot afford to lose.

[crazysquirrel]

A person here has the same problem with speed that I do.
Starts out really fast then slows to a crawl.


https://gist.github.com/2E0PGS/f63544f8 ... 4f56efe52f

If your running a x64 bit Ubuntu or other Linux and find USB transfers hang at the end apply this fix:

echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes

I suggest you edit your /etc/rc.local file to make this change persistant across reboots.

sudo nano /etc/rc.local

Go to the bottom of the file and leave a space then paste in those two lines.

Save the file with ctrl + x then press y.

To revert the changes enter this in console and remove the lines in /etc/rc.local

echo 0 > /proc/sys/vm/dirty_background_bytes
echo 0 > /proc/sys/vm/dirty_bytes"

Seems to work but I did not put any of that into any file. Just a terminal.
Speeds are now right at 30MB/S roughly when transferring large files (1.4GB to 4GB roughly).

Going to take me 14 hours to transfer 1.4TB of data :-(

Another site suggested fiddling with the USB stuff in a folder to alter the caching.
I won't fool with things I don't know anything about.
R/R takes a lot of time and costs me over $1.25/GB of data to do.
Don't think I am allowed to sit in McD's for 14 hours to get data....
(exaggeration of course).

Still another site suggested using modprobe twice in a row.

Is this better?

Code: Select all

System:    Host: <filter> Kernel: 4.19.0-12-amd64 x86_64 bits: 64 compiler: gcc v: 8.3.0 
           parameters: BOOT_IMAGE=/vmlinuz-4.19.0-12-amd64 
           root=UUID=<filter> ro fromhd=UUID=F269-E939 splash 
           Desktop: Xfce 4.14.2 tk: Gtk 3.24.5 info: xfce4-panel wm: xfwm4 dm: LightDM 1.26.0 
           Distro: MX-19.2_x64 patito feo May 31  2020 base: Debian GNU/Linux 10 (buster) 

Machine:   Type: Desktop Mobo: ASUSTeK model: P7H55-M PRO v: Rev 1.xx serial: <filter> 
           BIOS: American Megatrends v: 1709 date: 01/04/2011 

CPU:       Topology: Dual Core model: Intel Core i5 660 bits: 64 type: MT MCP arch: Nehalem 
           family: 6 model-id: 25 (37) stepping: 5 microcode: 7 L2 cache: 4096 KiB 
           flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 26751 
           Speed: 2615 MHz min/max: 1200/3334 MHz boost: enabled Core speeds (MHz): 1: 1929 
           2: 1790 3: 2199 4: 2372 
           Vulnerabilities: Type: itlb_multihit status: KVM: Split huge pages 
           Type: l1tf mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable 
           Type: mds 
           status: Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable 
           Type: meltdown mitigation: PTI 
           Type: spec_store_bypass 
           mitigation: Speculative Store Bypass disabled via prctl and seccomp 
           Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer sanitization 
           Type: spectre_v2 mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, 
           STIBP: conditional, RSB filling 
           Type: srbds status: Not affected 
           Type: tsx_async_abort status: Not affected 

Graphics:  Device-1: Intel Core Processor Integrated Graphics vendor: ASUSTeK driver: i915 
           v: kernel bus ID: 00:02.0 chip ID: XXXX-XXXX 
           Display: x11 server: X.Org 1.20.4 driver: intel resolution: 1680x1050~60Hz 
           OpenGL: renderer: Mesa DRI Intel Ironlake Desktop v: 2.1 Mesa 18.3.6 
           direct render: Yes 

Audio:     Device-1: Intel 5 Series/3400 Series High Definition Audio vendor: ASUSTeK 
           driver: snd_hda_intel v: kernel bus ID: 00:1b.0 chip ID: XXXX-XXXX
           Sound Server: ALSA v: k4.19.0-12-amd64 

Drives:    Local Storage: total: 10.07 TiB used: 4.13 TiB (41.0%) 
           
           ID-1: /dev/nvme0n1 vendor: Western Digital model: WDS500G3X0C-00SJG0 size: 465.76 GiB 
           block size: physical: 512 B logical: 512 B speed: 31.6 Gb/s lanes: 4 serial: <filter> 
           rev: 102000WD scheme: MBR 
           
           ID-2: /dev/sda vendor: Seagate model: ST2000LX001-1RG174 size: 1.82 TiB block size: 
           physical: 4096 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter> 
           rev: SDM1 scheme: MBR 
          
           ID-3: /dev/sdb vendor: Hitachi model: HTS543232A7A384 size: 298.09 GiB block size: 
           physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter> 
           rev: A60W scheme: MBR 
          
           ID-4: /dev/sdc vendor: Seagate model: ST3250318AS size: 232.88 GiB block size: 
           physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 7200 rpm serial: <filter> 
           rev: CC46 scheme: MBR 
          
           ID-5: /dev/sdd vendor: Seagate model: ST9500325AS size: 465.76 GiB block size: 
           physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter> 
           rev: SDM1 scheme: MBR 
          
           ID-6: /dev/sde vendor: Hitachi model: HTS545050B9A300 size: 465.76 GiB block size: 
           physical: 512 B logical: 512 B speed: 3.0 Gb/s rotation: 5400 rpm serial: <filter> 
           rev: C60F scheme: MBR 
           
           ID-7: /dev/sdf type: USB vendor: Samsung model: HD204UI size: 1.82 TiB block size: 
           physical: 512 B logical: 512 B serial: <filter> scheme: MBR 
          
           ID-8: /dev/sdg type: USB vendor: Western Digital model: WD Game Drive size: 4.55 TiB 
           block size: physical: 4096 B logical: 512 B serial: <filter> rev: 5002 scheme: GPT 

Partition: ID-1: / raw size: 32.00 GiB size: 31.25 GiB (97.65%) used: 9.43 GiB (30.2%) fs: ext4 
           dev: /dev/nvme0n1p1 
           
           ID-2: /boot raw size: 2.00 GiB size: 1.91 GiB (95.30%) used: 160.5 MiB (8.2%) 
           fs: ext4 dev: /dev/sda2 
          
           ID-3: /home raw size: 433.76 GiB size: 425.95 GiB (98.20%) used: 157.91 GiB (37.1%) 
           fs: ext4 dev: /dev/nvme0n1p2 
          
           ID-4: swap-1 size: 8.00 GiB used: 1.0 MiB (0.0%) fs: swap swappiness: 15 (default 60) 
           cache pressure: 100 (default) dev: /dev/sda4 

Info:      Processes: 241 Uptime: 6h 04m Memory: 7.59 GiB used: 1.84 GiB (24.3%) Init: SysVinit 
           v: 2.93 runlevel: 5 default: 5 Compilers: gcc: 8.3.0 alt: 8 Shell: quick-system-in 
           running in: quick-system-in inxi: 3.0.36 

Edit:
Here is another possible solution:

"for those who dont have /etc/rc.local can use this:
sudo sysctl -w vm.dirty_bytes=50331648
sudo sysctl -w vm.dirty_background_bytes=16777216"

Another one said to disable swapfile.

[gimbleguy]

"USB transfer speeds slow/ data"

I second this.

I have

SanDisk usb3.1 Extreme Pro
Samsung T5 SSD USB

File transfers are half the speed compared to a Windows 10 environment.

Linux USB file transfers are abysmal :(

Hope they can fix this soon

[gimbleguy]

https://youtu.be/lSnPhpxgQIg


SanDisk Extreme Pro USB stick
Something to think about...

[BitJam]

https://youtu.be/lSnPhpxgQIg

SanDisk Extreme Pro USB stick
Something to think about...

The speed comparison is meaningless if you don't drop caches in between the two copies. Data cached from the first copy operation is all stored in RAM (assuming you have enough free RAM). For the 2nd copy the data is written from RAM to the usb device without having to do any reads. Here is how you can drop caches on the command line:

Code: Select all

echo 3 | sudo tee /proc/sys/vm/drop_caches

If you do this before each copy then the test should be fair.