To know about the fastest way to cp files to USB Drive

[chandra1961]

Can I please know about the fastest method to copy and paste large files to USB flash drive in MX Linux?

[gimcrack]

The right hardware.......everything I use now is USB 3.x compatible if possible.

There is an area of memory set aside as a cache. When you are copying from one device to another each input record is read into the cache and then written from cache to the output device. If the input device is faster than the output device the copy runs at the input device speed but writes from cache to the output device at the output device speed until the cache fills up. Then the copy cannot read an input record until cache space becomes available for another input record. So at that point the reported copy speed drops to the speed of the output device.

Some copy programs report the end of the copy when the last input record is read into the cache. In fact the copy does not end until the cache is completely written to the output device. So is important that you issue a umount command before removing any devices. The umount command makes sure that there are no outstanding records still waiting to be written from the cache before unmounting the device.

As to device speed there are many cheap USB devices that run at less than the rated speed so two supposedly equal devices may run at different speeds.

[Eadwine Rose]

And then there is the computer.

With all help requests, please share your full Quick System Info, found in the menu under Favorites. Press the Copy for forum button, then click paste in a reply. If need be, you can do this with the LiveUSB, thanks.

[MikeR]

An informative start could be https://www.baeldung.com/linux/file-system-caching
... Lots of stuff about cache on the web.
However the biggest factor would probably be your hardware: Where are you copying from,
what are you copying through (USB 2, 3.1, 3.2...) and your copy destination
(Two apparently identical USB devices can show differing results)
Also I would think that direct copy (cp) would probably be faster than copy-&-paste...
Sorry if this does not help, but details of HW and use case are lacking

[BitJam]

Reducing the buffer size can drastically increase usb writing speeds. I think MX may do this by default already, but in case it doesn't, you can run:

Code: Select all

sudo sysctl -w vm.dirty_bytes=20000000

You can find the current value with:

Code: Select all

sysctl vm.dirty_bytes 

[m_pav]

Having considered all the earlier comments and including the removable media type you're copying data to, the File System on the external device will have an impact of performance too. Using a non-native Unix format type like the Windows FAT32, ExFat or NTFS will result in slower writes than a native Unix type File System like ext4, when copying data to USB from a host machine running a Linux Desktop system.

Adding to that, one must bear in mind the innate abilities of the drive in use. Not all USB storage media are created equal and not all that claim USB3 or USB3.x will have the ability to move data at a speed that reflects the standards.

Large file transfers to SSD will gradually slow down as the drive fills up because the SSD controller tries to allocate data first to the least used pages. A Spinning platter drive will simply find sectors marked as available, but a higher volume of fragmentation will result in increasingly slower writes too.

All indicated drive read/write speeds are typically quoted on a new drive that has not seen much use, is relatively empty and typically formatted with NTFS for the larger storage units and ExFat for the smaller to median Flash type media. These rated speeds will be the maximum speeds the drive is capable of reaching, and usually only for a reasonably short burst. One can always expect all drives to reach a point where they slow due to file system type limitations and their reduced ability to function when space becomes limited.

Another issue that needs to be considered with large file transfers is the type of data in transit and my experience over many years seems to show large media files are consistently slower than most other types.

I can transfer a 25GB Windows backup image I made for a customer in roughly 3.5 - 4 minutes to an external drive through USB3.1, but a 26GB MP4 takes closer to 7 minutes on the same drive. It's not the additional 1GB that is slowing the transfer, it's the content type.

I know this is not an answer to your question, but I hope it helps with understanding.

[figueroa]

You can compress the source files into a tar archive on-the-fly which means you actually transfer less data, and you are actually transferring just one large file. Here are some examples.

Transfer group of files as a compressed tar archive on the destination flash drive:

Code: Select all

tar cpvf /run/media/USERNAME/SP256/databackup.tar.zst -I "zstd -3 -T0" --acls --xattrs --numeric-owner --no-wildcards-match-slash -X /home/USERNAME/bin/exclude.files /mnt/backup/data/*

In it's simplest form where destination is a gzip compressed tar archive:

Code: Select all

tar czpvf /run/media/USERNAME/SP256/databackup.tgz /mnt/backup/data/*

Transfer group of files while encrypting and not compressing the destination:

Code: Select all

tar cvf - /mnt/backup/data/* | gpg -c --batch --yes --passphrase-file /scratch/bin/.passrc --compress-algo none -o /run/media/USERNAME/SP256/databackup.tar.gpg

Useful manual entries for reference: "man gpg" and "man tar"