Why does this dip occur?

Why does this dip occur?

copying from internal HDD to SD Card

all files are about the same size. Flacs.

same HDD to same SD card.

1 file of 10gb

now the dip is at the start

dip is back

Buffers generally.

HDD buffer?

sliding windows
there is some redundancy error or drop of data, the system backs off and uses smaller pieces of data till everything is okay, then starts using larger pieces of data again

Buffers and shitty filesystems, coping same amount of small files takes longer than one big file.

I don't entirely understand but can that be avoided somehow?

>coping same amount of small files takes longer than one big file.
I know. but see: and

It's a HDD, you have a spinner disk, it's fragmented by design.

>can that be avoided somehow
you wouldn't really want to avoid it, it's a data integrity thing
it might be cool to see a solid bar but when that solid bar has randomly dropped data because of buffers overflowing, you're not getting all your data

so the solution is just HDD's with bigger buffers?

Do you really know?
Seems like you dont

look again. the dip occurs while copying 1 file of 10gb

it also occurs when copying hundreds of files that are all the same fucking size.

no

Three possible reasons come to mind.

- for small files, their file data has to be written to the file system. As in, the OS has to tell the file system to create file x at location y with attributes {z}, etc etc. Then the filesystem calls back the OS that it is created, you can proceed giving data, etc. Because of this, even a 1 byte file will require a minimum amount of time to create that is longer than the read speed of the drive / size of the file. For example 1000x one kbyte files will take way longer to copy than one 1000kbyte file.

- Due to fragmentation, the HDD cannot transfer at its maximum speed. This is exacerbated by the fact that on HDDs, data speed is different depending on where the data is physically located (inner part of the platter or outer part). Combine the two, and you got speed drops.

- SSDs are affected by the above but in a different way, the given flash cell may be partially written and needs to be read, cleared, rewritten, which takes longer than writing to an empty cell. The OS handles all of this transparently, you'll only see some speed drops from it. Due to fragmentation, this can occur more often over time. Consequently, defragging can indeed improve speeds on an SSD (not even trolling here - I tried it before). However I still don't recommend defragging, since if you are on Windows you'll lose shadow volume copies that way which are extremely useful for accidentally deleted/damaged files.

So for those reasons, you'll see dips in file copy speeds, you won't get max speed constantly all the time.

oh wait, you are using an SD card.

Then you also have to include more possible reasons, such as

- whatever filesystem the card uses (for my usb drives, FAT32 seems insane slow for some reason, ntfs and exfat are orders of magnitude faster, but not all OS support them, I keep the drive in fat32 for self booting).

- whatever SD card reader you use, has a connection that might sometimes drop, assuming it is a USB one.

- the controller for the card reader might be low quality (usb3 controllers sometimes can be a real bitch)

- the sd card might just suck, or be low quality trash advertised as super hq usb3 giga speed.

additionally, sometimes antivirus will chew on the files for a little bit when copying too. Not much, but when you are processing hundreds of files, it can add up fast.

also if you use a sd card that is already slow as shit, then the flash cell rewrites I mentioned above might be orders of magnitude slower.

thanks user. very informative.

I'm using the SD card reader in my high end Dell monitor. It uses USB 2 to connect to my pc. Using MTP USB3.0 to my USB-C phone had an ETA that was very long so I canceled and put the micro-SD card in an adapter and in my SD card reader.

Yeah, I tend to remove the sd card from my camera and connect it directly to a card reader too, because it is five times faster that way, than if I connected the camera to usb.

If you want to make sure it is a fragmentation problem, make a backup of everything on the drive, format it (full format, not quick), then copy everything back. At least that one large MKV should have a flat speed curve, unless the hdd you copy from is already shot to shit.

Buffer/cache fill and clean.
Your SD is shit/worn so speeds are unstable.