USB transfer speeds slow/ data needs to be written message

[tony37]

HD and cable are brand new (less than 2 weeks old).

It may happen to new hard drives that they are, well, rubbish. Like recently here: viewtopic.php?p=599328#p599328

[paul1149]

HD and cable are brand new (less than 2 weeks old).

Can you try the drive in another unit? Or try the transfer over LAN with the drive attached either to the router or another unit?

[crazysquirrel]

Update:
CPU temps @ 104F
Transfer speed 187K
Estimated time to complete transfer from external to internal - 1114 HOURS.
309GB/1.1TB

If there was a hardware problem, one would think that it would be a constant issue.
But it starts out at 60MB/S then drops overnight to 187K only 1/3 the way through.

Room temperature is 71F

Blood pressure getting sky high...

[crazysquirrel]

Another update:
I tried transferring files in MZ19-2 KDE on a desktop and got the same problem - slow transfers and on the KDE one, locked up the file browser.
18.3 (non KDE) does not do this. FTN on the 18.3 to be certain.

[tascoast]

I'd be tempted to create a fresh Live USB of the latest MX and confirm that two different machines struggle with the drive causing problems. That might suggest the drive itself is the issue, new as it may be.

[TTwrs]

With most flashdrive there can be a huge difference between transferring large files and small files, especially when it comes to writes.

[m_pav]

I think it best that you look into your hardware, there are multiple issues that could be affecting your transfer speeds.
Please ensure your external drive is connected and open gsmartcontrol
Identify the external drive and double click it to open and view its attributes. You will see tabs across the top. If any of those tabs have even a hint of red in the lettering, the drive you selected has a problem that needs investigating. The brighter the red, the worse the issue and you'll have to dig into each tab highlighted with red lettering to see what is being reported. Also, there is no need to run any of the self tests, these are meaningless for this objective.
Repeat and do the same for your hybrid drive.

I highly recommend you educate yourself on how to read each tab, especially the Attributes tab. QSI, do not post back in this thread.
That is step 1. Do not proceed any further within this thread if you have any red lettering displayed in gsmartcontrol. If either of your drives is flagged with red lettering and you want further assistance, please start a new thread and include your QSI as required when asking for assistance. A screen shot would help us to help you with the best advice moving forward.

Step 2 is to see if either drive has any minor inconsistencies, or dirty flags on the storage media. It only takes 1 or 2 files that are in particularly bad shape to cause the exact behaviour you have recorded. If either has inconsistencies, fix them first using their native OS to do the fixing. Fixing a drive partition where the drive has red highlighting in gsmartcontrol can result in permanent data loss, do so at your own risk or let us guide you through it using some of the best tools the industry has which are built into MX Linux.

Step 3 examine the content you are trying to copy. Does the content consist of thousands of small files or are you trying to transfer genuinely larger files, like 200MB+ in size. For example, doing a file copy on 22,000 small files that collectively use 2.2GB of drive space will take a monumentally longer time to transfer than 11 x 200MB files.

Step 4 change the file system. If the external drive is using FAT32, expect all file copy processes to be slow. FAT32 is a very old technology, expect it to behave as an old technology. FAT32 is not a native Linux file system. NTFS has a number of versions, some faster than others, but again, not native to Linux, therefore, as with FAT32, it will be slower than a native Linux file system. In my experience, FAT32 is slower than NTFS and NTFS is about 20-25% slower than ext4.

Step 5 Compare your USB ports output current with that of the external drive. USB2.0 port standards are 5v@ 500ma, USB 3.0 port standards are 5v@ 900ma. If you are connecting an external drive that draws 550ma into a USB2.0 port, it will be in a state of browning and cause read/write failures. A USB 3.0 external drive with a separate USB Power lead plugged into a USB 2.0 port will be expecting a power delivery of up to 900ma, so a USB 2.0 port will not usually provide sufficient current. I have seen all too often faulty brand new twin head USB cables where one head is for data and the other for power.

This is a big post and I've covered a fair bit of stuff but I'm personally leaning towards Step 2 as being the ultimate remedy and hoping for your sake, it's not Step 1.

[figueroa]

Most of these problems are related to a minor bug in Linux. The default buffer size is gianormous. So writing at first is fast because it is mostly writing to RAM until the buffer fills. Then it takes a long time to do the rest of the writing because after the buffer is full, the speed is limited by the speed of the device.

I thought it was a little strong to call this a bug. I think of it as a tunable parameter, along with a big handful of others related to vm cache. I've been happy enough to leave my vm settings at default and it's sufficient to know what's going on. The USB drive won't actually unmount until data is written. Having lower settings won't make the device write faster.

The user tuning his machine will do well to do an internet search on sysctl vm.dirty_bytes or just sysctl vm.dirty and read through some of those pages.

Notably: https://www.suse.com/support/kb/doc/?id=000017857 is very informative as is https://lonesysadmin.net/2013/12/22/bet ... rty_ratio/. You can set permanent boot-time changes to default settings in /etc/sysctl.conf

[crazysquirrel]

Transfer was completed last night.
1.1TB transferred, 691,000 files.
External drive formatted to EXT4, internal partition formatted (by Windows) NTFS.
External Drive WB Black, 5TB (actually about 4.5TB they lie)
Connection from external to computer was USB 3.0 using supplied cable to the only USB 3 port on that X401a laptop.
Internal drive is a Hybrid drive (Seagate I think), 2TB.

Conclusion: Linux stinks for file transfers.
Hoping someone will find and fix the USB issues Linux (ALL) have.
Haven't found a version of Linux yet that can hold a candle to Windows transfer speeds.

[crazysquirrel]

Another update:

I found a problem.

Seems the transfer is going to sleep for far too long.

Task Manager reports sleep 20 as being the only thing noticeable that is causing the problem.
When first starting a transfer, it zooms pretty fast, then 20 minutes later the transfer slows then apparently stops.

20 DAYS is far too long. So is 20 hours or 20 minutes. Even 20 seconds is too long.
I have been waiting over 3 hours to transfer three 1.5Gb files to a flash drive. Still far from done.

I do not know how to cancel the sleep function so I can resume data transfer to a USB drive.
I tried Killing sleep in task manager to no avail. Keeps coming right back the moment it is stopped or killed.

https://www.cyberciti.biz/faq/linux-uni ... scripting/