Best strategy to migrate 20TB to drivepool

[Spider99]

Hi

 

Just planning to move 20TB to a new drivepool setup

 

i have the option to add a SSD cache to the pool(s) and I am wondering if i should not enable the ssd cache until after the transfer is complete - as it appears if you do not have the settings quite right (as i'm new at this) you can fill up the ssd before they have a chance to move the data into the pool and hence complicate the issue. I obviously want this to run unattended as its probably going to take more than a day to complete.

 

From experience should i also disable duplication until its complete to speed things up?

 

A test copy(samba/smb) shows the transfer will happen at ~ 117/120 MB/s which is about the max you can expect for large files anyway

 

Anything else to be away of/consider?

 

Thanks

[Christopher (Drashna)]

Well, how are you moving the data? 

 

I ask, beacuse we have a "quick and dirty" method, to expidite this:

http://wiki.covecube.com/StableBit_DrivePool_Q4142489

 

As for the setting up the SSD Optimizer balancer plugin, you can do that at any time. 

And no, you won't risk filling the disks right away. Until the balancer is enabled, the pool's default placement strategy for new files is to place the files on the disk with the most (absolute) free space. Meaning that unless these are LARGE SSDs, they'll be the last to get data. 

 

However, I would move the data over, let it rebalance and reduplicate the data before configuring the SSD options.  THis is because the "SSD Optimizer" uses the balancing system, and if the pool is duplicating data, it won't balance data off of the SSD drives right away. 

 

That said .... even if the SSDs get full, the software will move onto the next available disk, (by default, after the SSDs are 75% full). 

 

 

 

Additionally, if you are using duplication, then you will want to use multiple SSDs for the SSD Optimizer. 

[Spider99]

Thanks

 

Quick and dirty would be nice but the data is on raid5 ext4 drives  so network for direct copy

 

but am thinking maybe copying the data to a usb drive formatted as ntfs and then do the quick and dirty method as usb3 is a bit faster than Ethernet

 

So basically copy first - duplicate/balance - then add ssd(s) to pool then turn on optimiser.

 

Whats the rule of thumb for ssd's added to a pool is there some size/ratio or number of drives to add?

[Christopher (Drashna)]

Ah, that would make it a bit more difficult then! 

 

As for ethernet vs USB3.... that depends on the drives. A lot of consumer drives will get max of 100-120MB/s for writes, and maybe less ....   But that depends a lot on the hardware, and drivers.  But either way ....

 

 

As for SSDs, you need a number of SSDs equal to the level of duplication/redundancy you want. So if you want 3x duplication (each file on 3 disks), then you'd need 3x SSDs.  That is because of the real time duplication feature. Meaning that files are written to all of the destinations in parallel, rather than duplicated later.  

 

As for size, the larger, the better.  Since files will be written to the SSDs first.... it may impact the size of files you can copy. 

Additionally, there is a setting for the SSDs (75% by default), and once the SSDs get this full, the software will write to the Archive drives instead.

http://dl.covecube.com/DrivePoolBalancingPlugins/SsdOptimizer/Notes.txt

[Spider99]

Well i copied all the data to five disks via USB3 and have now just used the quick and dirty method to add the data to the pool - works a treat

 

One thing though the i think http://wiki.covecube.com/StableBit_DrivePool_Q2299585Bneeds updating as i could not find any troubleshooting section to reset the options

 

As it turned out setting the ssd cache kicked things into life

 

Drivepool is digesting 18TB of data at the moment - not much happening but i have just given it the best part of 1M files to figure out - will leave it to get on with it and turn on duplication later

[Spider99]

Well not having any luck getting a balanced pool - drivepool just "hangs" at 0.7% for hours with no file movement - it did initially move files at upto 250MB/s but after a few minutes this stopped with no more disk activity.

 

I think its crashing because you cant shut down the pc and have to hard reset

 

on reboot the same thing happens

 

tried this 4-5 times

 

any ideas as to what to try/change?

[Christopher (Drashna)]

If you're having to do a hard reset, then something is definitely going wrong. 

 

What version of StableBit DrivePool are you using?

And what OS? 

 

If you're not already, could you upgrade to the latest beta version and see if that helps? 

http://dl.covecube.com/DrivePoolWindows/beta/download/

 

Specifically, the 2.2.0.711 version. 

 

 

If that doesn't help, could you enable logging and "allow" this to happen (eg, wait for it to happen)?

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

And could you trigger a crash dump, so it dumps the contents of memory to disk (and upload that)?

http://wiki.covecube.com/StableBit_DrivePool_System_Freeze

[Spider99]

Thanks Chris

 

Well a fun day of testing has i think got to the bottom of this - a corrupt BIOS which may have occurred during one of the hard reboots - having a dual bios motherboard helps along with spare RAM to prove the original ram was fine which was my initial suspicion.

 

The system is back up and booting properly now (cross fingers) have managed to get it to un-duplicate the files it had done which worked fine. I am now running the re- balance to see if it will actually work this time and move the data across all (or most) of the disks. Lots of CPU activity on Drivepool Service and System although minimal disk activity so far and balancing is not showing any % complete yet but i guess its working out what to move where.

 

Ah ha we have data movement but no activity on the pool (Is that correct???) % changed to a massive 0.1%

 

Well the data appears to be being moved - although no activity on the pool being shown - @ 200MB/s - 280MB/s

 

Now at @ 0.5% if it gets past 3.8% then we will have moved on as that was the max i have seen previously

 

Yes i'm on 711 as tried that as an option to help with the lockups

Win 10 Pro 64bit Anniversary Update

 

Tim

[Spider99]

Well we are back to Drivepool crashing @ 0.8%

 

I was watching its progress and there was a system pause (lockup for a few seconds) then the file copying came to a halt not long after and i left it for 20 minutes with no more disk activity and minimal cpu usage

 

The Drivepool service cannot be stopped or terminated and eventually the system has crashed again

 

Looking at the logs it has not written anything of note - no errors just stops working - last log update approx 19:50 crashed after that point

 

 

Sigh

T

[Spider99]

Well knock me down with a tribble its working

 

well at least its done a full balance with no reported errors 

 

Two things i changed this morning

1) cleared out the Drivepool>Service directory - so i would have fresh config and new error logs

2) Removed out of the pool the directory containing circa 500k of files in multiple directories

 

Restarted the service and it completed

 

Couple of things i noticed

a) the re balance was going significantly quicker in terms of % complete ~ 0.1% every couple of seconds - i only moved out about 1.5TB so the pool still has 17.3TB of data to balance

the data was being written at a slower pace ~180MB/s rather than the 250+ of last night - might be a red herring but odd

 

So i am going to put back some of the removed data to see if it chokes again and will report back

[Spider99]

Getting closer to the problem i think

 

I decided to re balance the pool with the Disk Space Equalised before copying back the directories i moved out of the pool (reverse quick and dirty)

 

This appears to be working fine

 

So i got bored and decided to push my luck a bit

 

So i picked a small directory of a 4k files and about 300MB and moved it to the pool - this worked fine and balancing continued with no lock ups

 

So i got bolder and moved a large directory of 16k files and 4GB - this failed to even start and DP died as well (reboot tried and failed - hard reset again) - it also left one disk at 100% utilisation according to task manager - dont know if this was the disk the files were potentially going to or the disk DP was writing to at the time of the crash as at that time was not using resource monitor to see what was going on in the background.

 

Nothing in the error log - empty

 

To me at the moment - its looking like a conflict between DP and a driver or DP chokes on a large numbers of files

 

Either way its a bit wierd that i am not getting error messages - so i could track back to something i might be able to change/repair etc

 

There are errors in the event log in windows but i think those are all to do with the corrupt bios but can export them if you think you need them 

Errors for DP/Scanner/VDS Provider/VDS and a few for the RocketRaid HBA

 

So Chris how would you like to proceed?

[Spider99]

Posted a support call https://stablebit.com/Contact/2940692

[Spider99]

To close this out for anybody in the future

 

1. RR HBA 2720SGL cards do not work together in Win 10 Pro 64bit - I tried the Raid and Quick bios's both had same result - crashing within 15 minutes with 4 or more drives using Burn In Test 8.1

2. However if you flash the cards using the 2720_c2 Bios the cards behave in Win 10 Pro - do not fail the Burn In Test using Ver 2720c.v10 of the bios, Ver 1.0.0.0 of the miniport drivers and Ver 2.5.2.4 of the Management Program from HPT.

3. However ( ) you need to configure the cards separately - preferably in another machine - to upload the bios and then use the management programme to amend you settings in the configuration screens - or use the CLI program to do the same thing - actually quicker. Do each card one at a time and remove before putting the other one in. Once both are configured - make sure exactly the same config changes as they are going to work as one card from now on.

4. However ( yes again!). Once both are back in your target machine - connect your drives and boot up - you will see nothing in POST as the cards have it appears no raid functionality once paired. In windows you will now find the management prog - will not connect to the cards and nor will the CLI - although that lets you login - then ignores any commands you type!!!!!! So you have no way or changing anything short of pulling the cards and putting them in another machine - see above.

 

I have a support call with HPT - being very dim and slow - but i think the have finally go the point its there software not windows thats the problem. If i get a resolution i will update this thread

 

Couple of final things - in theory up to four of these cards can work together - verified for what its worth with HPT prior to wasting over a month faffing around.

Even though you cannot monitor the cards they do work as one under windows and Scanner(3122 Beta) sees them as one controller with 16 disks and DP (727 Beta)is ok as well- so - at least the cards do perform - sorry function

 

Only time will tell if they continue to do so and maybe HPT will come through with a fix - yeah and the orange Tango muffin - respects women!

[Spider99]

Actually go to the top of a very tall tower and throw the RR cards as far as you can 

 

They crashed the server and now i can only boot if the rr drivers decide to load and then they forget half the disks!!

 

This is after running for two days copying data with no issues at all

 

Maybe windows updated itself and did something -actually dont care anymore had enough hassle with them

 

Anyway my new HP 410 arrives tomorrow hopefully with a sas expander so the RR are being rma'd or being sold on ebay

 

Avoid if you can!

[Christopher (Drashna)]

Well, I'm very sorry to hear that!

 

Hopefully, no more issues with the new controller!