Issues copying ~10TB with rsync

[esoral]

I'm having an issue copying data to my drivepool and it's really starting to annoy the crap out of me.

 

I have about 10TB of stuff on a ZFS pool that I'm trying to move to my drivepool.  I'm going about this by mounting my share which is my drivepool on my linux pc where my ZFS pool is, then using rsync to copy everything over through the mounted share.

 

Here are the options I'm using in rsync:

rsync -av --no-links --no-compress --progress --stats --partial /zfs-pool /drivepool-share

While watching it everything seems to copy fine, I'm typically copying at about 110-120MB/sec which would make sense given my gigabit home network.  While watching the drivepool UI I see similar write speeds.  

 

Then out of left field rsync will start showing impossible copy speeds for my network, like 300+ MB/sec.  I look at the drivepool UI and there will be very little disk activity.  I browse the folders in my drivepool and as I suspected nothing is being written, despite what rsync --progress is reporting at the moment.

 

Every file will continue like this, rsync will tell me it's copying everything at impossible speeds, nothing will actually be copied then rsync will complete like it's finished.   If I wait a few minutes and restart rsync it will pick up where it left off in the actual copy process and continue on actually copying the remaining files like I want for a random length of time before this happens all over again.

 

 

I thought maybe the SSD optimizer was causing this, and when my SSD was getting full and being dumped into the archive drives, rsync was bugging out.  I have disabled all balancing options and this still continues.  I sincerely doubt it's an issue with my network connectivity so I'm at a loss here, it's taking me several days to copy these files when it should have only taken me the better part of a day.

 

Any suggestions?

[Spider99]

have you tried to just copy via a smb/samba share rather than mounting the folder

 

or pull the data from the zfs box to windows via a share

 

no need for rsync unless its over a wan

[gj80]

I'm not sure what rsync is doing during those "spikes", but I agree with Tim... I'd copy with Robocopy from the windows side (use the multithread flag - /mt) to pull things down. If you put the copy thread count high enough, you'll get drastically faster copy performance than you would with rsync (without chunking rsync out with multiple batches simultaneously).

 

I could see a motivation for using rsync if you were running it against an rsyncd daemon on the other end, because in that case it could do its own CRC validation from one side to the other (default behavior) for extra data validation on top of standard tcpip, but when you're running rsync against a plain file share, that doesn't occur.

 

A heads-up, though - be sure you check your ZFS pool for any file paths (a restriction linux doesn't have, but Windows does) that might extend past (or even come close to) 260 characters (the windows max path limit). Rsync against a Drivepool share will copy the long paths, but you will then be unable to access them using explorer, powershell, etc. I made this mistake, and had a hell of a time dealing with it retroactively - do yourself a favor and shorten them beforehand.

[Spider99]

if its windows 10 and i assume 2016 there is a new switch to allow files paths longer than 260 in explorer etc

[gj80]

There is, but I did some testing of that at work with a client who was all on Windows 10 and had issues with long paths... what Microsoft apparently did was that they added the flag, but they didn't bother to do anything whatsoever beyond that with core applications.

 

What that means is that things which had been recently-compiled did "just work" with long paths, but Explorer actually has explicit coding to disallow saving files in long paths (it throws up error messages, as I'm sure you've seen). Microsoft didn't bother to remove that......which is beyond moronic, but there you go.

 

So, I made a very long series of directories with powershell and had no problems there, but the moment I tried to do things in there with Explorer I started running into weird issues. I could go down past the 260 character limit, but not all the way to the bottom of the series of folders, and Explorer would intermittently throw up its "path too long - I'm not allowing this!" message when I tried to create/modify files. And of course, Explorer not working causes problems with many other things since it's relied upon for open/save dialogs, etc.

 

Thanks Microsoft.

[Spider99]

hmm - not good news - thought they had sorted it obviously not!!!

[Christopher (Drashna)]

The 260 path limit is due to the API in question, it's not an NTFS issue, nor a kernel issue.  In fact, StableBit Drivepool has no issues with paths longer than that (in fact, IIRC, the actual limit is 32k characters)

 

 

As for the odd behavior, could you make sure you're on the latest version of StableBit DrivePool?

http://dl.covecube.com/DrivePoolWindows/beta/download/StableBit.DrivePool_2.2.0.734_x64_BETA.exe

 

And if the issue persists, enable file system logging, and reproduce the issue. 

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection