Jump to content
Google Drive is Discontinued (FAQ added)

Hereisphilly

Members
 View Profile  See their activity

  • Posts

    7

  • Joined

  • Last visited

 Content Type 

Posts posted by Hereisphilly

    Extremely slow balance speeds

    in General

    Posted

    9 hours ago, gtaus said:

    I don't know if this will work for your situation, but I recently had to move lots of data (TB's of movie files) into my DrivePool. The standard Windows Explorer copy/move of bulk files is, like you said, pretty much a babysitting manual copy/move job. The first problem Windows Explorer hits in the bulk transfer will stop everything and wait for you to clear the error box before it will continue. I had about 10 hours of files to move to DrivePool, I set it up to move the files overnight, but it snagged a problem only 30 minutes into the transfer and stopped. Hardly any files were moved.

    Since that experience, I now use the free TeraCopy for my large bulk file transfers. TeraCopy has a number of advantages over Windows Explorer. First, and foremost for me, it queues up all the files in a transfer list. If TeraCopy hits a problem with a file transfer, it will mark that file in the queue, skip over it, and continue on with transferring all the other files in the list. In my overnight example, it transferred all the files except 1 file that was corrupt. In the morning, all I had to do was look at the one file that errored out on the transfer. Much, much better than Windows Explorer.

    Teracopy also has a checkbox for verifying files after a copy/move. Of course, this doubles your transfer time with a verification on each file, but for important files that you want verified it works great. I almost always use the verify command on files I transfer to archive backup drives. It takes a little longer, but I really don't care as it does it in the background and I don't have to babysit TeraCopy.

    How are you measuring the balancing transfer speeds of DrivePool? The read and write status on my DrivePool GUI does not register any disk speeds when DrivePool is balancing the pool in the background. I know it is working because I can see the lights flashing on my USB HDDs, showing activity, and balancing will list the % of completion. But I don't see anywhere the speed is measured for balancing in the background.

    In my case, I am not very concerned about the speed of background balancing as long as it gets done correctly. I use DrivePool mainly for my media center file storage, so I don't need really fast transfer speeds. Having said that, I added a 248GB SSD to the front end of my DrivePool and that is very fast. I set the SSD Optimizer to re-balance only when there is at least 100GB of data to move. Essentially, I now have a 100GB SSD cache for my files, and if/when I reach 100GB threshold, DrivePool re-balances the data off the SSD in the background and I don't even notice it.

    You did not mention any SSD on your DrivePool, so I would encourage you to look into the benefits of front loading your files to a SSD. It really works great for me. Obviously, once you transfer more data than your threshold (100GB in my case), the transfer speed will slow down to the read/write speed of your USB drive(s) because DrivePool will not attempt to overfill the SSD and will switch over to writing to the slower USB drives directly until the SSD flushes its cache. DrivePool works great, and it is even better with a SSD on the frontend.

    Anyway, using TeraCopy and a SSD on the frontend of my DrivePool has made everything easier and faster for me.

    I had looked at tera copy and was an option, but at the moment I have just let the disks run while I'm raising the ticket with drivepoool direct as I really want the underlying problem fixing.

    This affected array is my cold storage backup so gets fired up once a week to sync and then is powered off. I don't have any of the speed issues when copying to the pool, its just in the balance where the problem is.

    Disk speeds are shown in stablebit scanner and also can be seen in task manager.

     

    SSD as a cache wont really be necessary for me as my main array sees more intensive IO and I'm currently happy with those speeds anyway. A cache for this backup array is definitely not needed as i set the sync off and let it run until complete 

    Extremely slow balance speeds

    in General

    Posted

    4 minutes ago, Shane said:

    @Hereisphilly Hmm. Nothing obvious springs to mind, sorry. Maybe some kind of buffer or other limit is being hit in the USB drivers, since the problem only showed up when you added a third USB drive? When you're doing the direct copy via Windows and getting the normal expected speeds, are you doing so from one mybook to the other mybook? Do you get that same speed when direct copying from one of the mybook drives to the elements drive? Have you tested with both a single very large file and many very small files of comparable total size to see if that influences the result?

    @MattG Also, are any of the drives in your pool are SMR type drives?

    I'm not sure the problem arose when adding this drive, up until now I've never needed to balance as the 2 mybooks were bought at the same time last year and then initially synced with the main array fine. Weekly backups go through fine with them too.

    I've tried direct copying from the ntfs mounts of each mybook into the element direct mount both with loads of 3-10mb files, 10gb and then 30gb files, all of which give speeds circa 60 Mb/s for the small files, and the 130 for the larger, and that's whilst the balance is still happening. Turning that off increases the speed slightly even more, so I think I can rule out any USB3 buffering issues. 

    Both mybooks into the element are the same on performance

    Extremely slow balance speeds

    in General

    Posted

    6 hours ago, Shane said:

    @Hereisphilly could you give more details on the hardware (e.g. drive and enclosure models, board they're plugged into, how they're plugged in, etc), including the new disk you've added?

    @MattG You can delve into the advanced settings here: https://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings

    Don't know if it'll help in your case, but for whatever it's worth my personal settings for "I don't want DrivePool to do balancing/duplication/measuring/etc with background/idle/low priority, even if it impacts delivery" are:

    CoveFs_AsyncFindFilesLowPriorityThreads - override from null to 2

    CoveFs_MeasureBackgroundTaskPriority - override from null to false

    DrivePool_BackgroundTasksPreventSleep - override from null to true (don't use if you want your computer to go into standby at night or whenever)

    DrivePool_BackgroundTasksPriority - override from null to 0

    FileBalance_BackgroundIO - override from null to false

    FileDuplication_BackgroundIO - override from null to false

    ... and if you're like me and insist drivepool checks for itself that what it told Windows to copy between disks is actually what got copied between disks (because I once ran into a problem where a SATA controller was silently corrupting files and Windows didn't notice at all) though this does impact performance:

    DrivePool_VerifyAfterCopy - override from null to true (note that this seems to be independent of whether "Verify after copy" is ticked in the GUI under Cog -> Troubleshooting)

    YMMV.

    Shane

    Thanks for getting back to me, the pc is an I3 4150 on an Asrock H97M Pro4

    The disks in this array are 2 X 10Tb WD mybooks connected over usb 3 to the rear Io usb3 ports and then the new 10tb WD element that I connected up again on the rear Io.

    Swapping disks over to the front Io ports don't change anything in terms of speed. 

     

    This is my cold storage backup array that I can power down, my other main array on an internal pcie hba are all working fine

    Direct copy speeds via windows direct to the disks works absolutely fine and I get the seeds I would expect for a rust spinner over usb3

    I've made the recommended changes to the settings.json file, restarted my pc and the doesn't seem to be any change. Balance speeds in talks manager and scanner are around the 4-10 MB/s mark

     

     

     

    Extremely slow balance speeds

    in General

    Posted

    As above, I'm experiencing extremely slow balancing speeds of around 4 MB/s when I've added in another new disk to my pool

    Burst testing with scanner and directly copying the files in windows results in speeds of around 150 MB/s which is as I'd expect

    The pool is 3 x 10TB WD external drives connected by USB 3. I've tried clicking the forward arrow to increase the priority and ot doesn't so alot

     

    Any help would be appreciated as I have alot of data to move and dont fancy babysitting a manual copy

    Thanks

    Disks not spinning down, but settings.json is blank

    in General

    Posted

    Hi, im a long time avid user of drivepool. Recently added a bunch more disks to mypool and only now noticed that the disks dont seem to spin down ever. Checking with drivepowerstate shows all the disks will constantly stay spinning even when windows is saying to spindown after 10 mins

    This is including another hard drive that isn't even part of the pool. Stopping the drivepool service however results in all my disks spinning down immediately, so i've definitely found the culprit.

    Running version 2.2.2.934 with windows 10 1809

     

    I think i'm a victim of the bitlocker bug as described here https://community.covecube.com/index.php?/topic/4252-drives-not-spinning-down/

     

    The problem is when I open the settings.json file, it just comes up completely blank in notepad. Any thoughts on how I can access this, or have these settings been moved elsewhere?

     

    Thanks in advance 

     

     

×
×
  • Create New...