Christopher (Drashna)

I'd like to start by saying I've been using DrivePool+Scanner for a while now, and they are seriously awesome. Everything WHS should have been. So much thanks for that.
 
I only use it for home use but the one thing I'm currently lacking is an offsite backup. That's where CloudDrive comes into play for me. Being able to use multiple cloud providers with DrivePool is really powerful. Before now I wouldn't have considered it as I don't want to trust the cloud with my private files, but the encryption part of CloudDrive fixes that.
 
So basically my use case is to pool as many providers as possible, and backup only the most essential data. Speed isn't important to me.
 
On to my first experience. I installed CloudDrive and hooked up dropbox, amazon and hit a wall with microsoft onedrive. I eventually found the forum post (grateful it was stickied) and enabled it. I realise that their service is poop, but would be nice to be able to enable in the UI with appropriate warnings.
 
I put about 10mb of test data in all of the providers and found that they all generated lots of errors and didn't seem to get anything uploaded. Eventually worked out by reading through the forums that it's probably because my connection isn't fast enough (8mbit down, <1mbit up). After massively increasing most of the timeouts, reducing the threads to 1 and destroying/re-creating the drives with 128k chunks instead of the default 1mb I managed to get it stable.
 
What would be cool is if you had a benchmarking feature in the app, a bit like scanner's burst test. Then you could potentially have an "autodetect" settings feature, as finding and editing a config isn't the most user friendly first experience. You could also warn the user that their connection is less than optimal.
 
The only other thing I messed up was that I assumed that CloudDrive was encrypted by default. Would I be right in thinking that with encryption unticked, the chunks in the cloud would be readable somehow? I couldn't find a way to enable encryption on my cloud drives after I had made them, so in the end deleted them and created them again.
 
It would definitely be cool to have an overview panel of all of the connected drives. At the moment you have to hop through all the tabs to see what, if anything, is going on.
 
Anyway hope this feedback is helpful, keep up the fantastic work.
 
 
Max

Here is my configuration along with a custom built rack.
 
Server
iStarUSA D-412S3-MATX 4U Rackmount microATX Server Chassis
Intel Core i7-3770K Ivy Bridge 3.5GHz (3.9GHz Turbo) LGA 1155 77W Quad-Core
Intel Media DQ77MK Desktop MB, Intel Q77 Express Chipset, w/TPM
Corsair Vengeance LP 32GB (4 x 8GB) 240-Pin SDRAM DDR3 1600 (PC3 12800)
Intel 80GB 330 Series Maple Crest SSD
Antec EarthWatts Green EA-380D, 380W
Noctua Ultra Silent CPU Cooler Cooling NH-U9B SE2
3x Noctua NF-R8 PWM 80mm Case Fan
LSI 9200-8e SASController Card, 2 x SFF-8088 mini-SAS External Connectors, JBOD only
Windows Server Essentials 2012 R2
Stablebit DrivePool v2.1.1.561
Stablebit Scanner v2.5.1.3062
IHomeServer v3.1.76.0 (iTunes 12.1.2.27) feeding 2x AppleTV 3
Crashplan
 
 
Array
NORCO DS-24E 24-3.5" JBOD Enclosure
 
Pooled Drives - 59.2TB
2x 8TB Seagate Archive HDD
1x 6.0TB WD Red
1x 4.0TB WD Red
7x 3.0TB WD Red
5x 2.0TB WD Green
1x 2.5TB WD Green
3x 1.5TB Seagate
1x 1TB WD Green
Non-Pooled drive
1x 1TB WD Green - OS Backup drive
 
UTM
ARK IPC-1.5U1525 Black 1U Rackmount Server Case
Intel S1200KP Mini ITX Server Motherboard LGA 1155 Intel C206 BIOS
Intel Core i3-2120T Sandy Bridge 2.6GHz, 35W Dual-Core
Stock Intel cooler
Noctua 60x25mm A-Series Blades with AAO Frame, SSO2 Bearing Premium Fan
G.SKILL Ripjaws X Series 16GB (2 x 8GB) 240-Pin SDRAM DDR3 1600 (PC3 12800)
Intel 60GB 330 Series Maple Crest SSD
100gb 2.5 HD
Intel Dual Port Pro 1000 PCIe NIC
picoPSU-160-XT, 160w output, 12v input DC-DC Power Supply
Ubiquiti Networks Unifi UAP-Pro Enterprise Dual Band AP
Untangle v11.2
 
Dell PowerConnect 2824 24port GB managed switch
Dell PowerConnect 2724 24port GB managed switch
 
VMware ESXi 5.5 Home Lab server
ARK 1U125 Black 1U Rackmount server case
Supermicro A1SAi-2750 mini ITX MB
Supermicro PWS-203-1h 200W 80 Plus Gold
Intel Atom C2750 2.4GHz Avoton 8 core CPU
32gb Kingston DDR3 1600 ECC memory
LeefSupra 16gb USB 3.0 flash drive
1x 480gb Intel S3500 SSD
2x 240gb Seagate 600 Pro SSD (200gb)
1x 128gb Samsung 810 SSD (cache)
1x 64gb Intel 330 SSD (cache)
 
APC Smart UPS SMT1500RM2U 1000W 2U rackmount
 
 
Ctopher

Someone has to have it, right?
 
How about we start with the largest pool of the members that participate here?
 
I'm taking an uneducated guess that theoretically Drivepool should scale indefinitely (or to some insane limitation imposed by the O/S, hardware or something else) however we all know the real world is where it's at, yes?
 
I'm sure I'm nowhere near the largest but I've maxed out my Norco 4220, and then some.
 
It's NOW time to start building the successor! 
 

 
 
 
Would love to see what everybody else has going on!
 
~RF
 
 

im definitely buying five

Well I'm in europe... and the Euromillions lottery is up to £95m this week...  and thats lump sum and tax free

The "File Placement Rules" feature can do this, actually. 
It may require extensive micromanagement, depending on how your pool is setup, but the UI should allow you to easily do this.
http://stablebit.com/Support/DrivePool/2.X/Manual?Section=File%20Placement
 
As for fragmentation, you can defragment the disks in the pool, without any issues.
 
 
 
So that is a drive does fail, you can be certain that you have all of the contents of a particular folder (or none at all). 
 
 
I've been there (not blue ray rips, but re-encoded files), and it sucks.  It may be worth investing in enough drives to duplicate everything.

I would be very interested in a product called "Stablebit Filesafe" !

Thank you, Christopher!  The 'shotgun' effect is exactly what I was trying to describe.  
 
Again, it's amazing the level of support and "don't feel like you're looking down" on people that you all provide here.  On the FreeNAS forums and some other places, it's really very hostile.  That alone gives you a huge leg-up, plus having a really polished product.  I'll be ordering the combo-pack tomorrow!
 
Mike

We're definitely using the new API.
However, in regards to throttling, but API's seem to be throttled.
 
And i you want to see just how badly, check out the dev site:
https://dev.onedrive.com/README.htm
Check the bottom part out, where it says "Throttling".
It delays you for 3600 seconds (one hour)! That's..... horrendous. 

Christopher,
Alex found that software (not sure what) was overwriting the GPT.  Alex ultimately made the physical disk read only so that it will not be overwritten.  Currently, the pool is functioning as designed.  Therefore, the issue appears to be resolved.
 
 
Thanks,
Todd

I ordered the motherboard and CPU yesterday. I decided to bump up the CPU to a Xeon E5-2620V3 2.4ghz processor $430.00. It has hyper threading and a few other goodies that the 1.9ghz Xeon does not have. Stupid CPU costs more than the motherboard! Newegg was out of memory, so I'll have to get it elsewhere. While I'm at it I might as well get a SATA III SSD because I have a 256gig SATA II SSD for the OS drive now. What I paid for my current SSD I can get one with double the capacity and faster for a little less money then what I originally paid!
 
Sigh... :-)

Honestly, it depends on how you want to do it.
 
If you have the space available, you can remove the drive you want to replace first. 
 
But you could absolutely add the new drive, and then remove the smaller drive from the pool afterwards.
 
 
 
However, what I recommend here:
Remove the drive from the pool. Use the "Duplicate files later" option. This will remove the drive quickly, as it's leaving the duplicates on the drive. After the disk is removed, it will recheck the duplication status of the pool, and reduplicate files as needed.
Before it starts doing that, physically remove the "old" drive, and connect the larger drive.
This way, when it starts duplicating the data, it will use the new drive most likely. This will minimize the reshuffling of the files on the pool.

Well, PCI-X is still pretty darn fast, so either way, you should be fine!
 
And to make sure, I recommend the IR (RAID) firmware over the IT (passthrough/HBA) firmware. Both can do passthrough, but the IR seems to behave better. Besides, extra options are great.
 
 
As for Intel SAS Expander card, great catch! And yeah, I've only been able to find them for $200 or so. So finding one for $60 is fantastic! (if you see another, link me!)
 
 
As for the BSOD, it indicates a CLOCK_WATCHDOG_TIMEOUT bug check. This indicates that an expected clock interrupt on a secondary processor, in a multi-processor system, was not received within the allocated interval.
https://msdn.microsoft.com/en-us/library/ff557211(VS.85).aspx
As for the hotfix, it may be OS specific. And hotfixes aren't usually pushed through Windows Update. They're usually option installs offered to fix specific issues.
This is the hotfix you're referring to:
https://support2.microsoft.com/default.aspx?scid=kb;EN-US;975530
And if you check lower down on the page, there are a couple of registry tweaks that you can try, and see if it works.
 
 
 
If it happens again, set the dump to us:
http://wiki.covecube.com/StableBit_DrivePool_System_Crashes

Well, if it is an educational excersize only then I would opt for the WS10 Tech release. I have that installed on an old machine for that purpose as preperation for the migration from WHS2011 sometime in the early 2020's

Yes it is for WHS 2011. Thanks for the link. I'll save that for future refernce. Everything is back up and running now. I was able to recreate the shares through the dash board. All the shares are where they where before WHS freaked out. I also ordered some AOC-SAT2-MV8 cards for spares. I found a great deal on them on EBay. $35 each for two new cards.
 
Well I'm happy that things are back to normal. I need to pick up some more 6TB drives over the next couple of weeks. Gotta replace one 2TB drive that has a some bad blocks and a couple other 2TB drives that Scanner says I should consider replacing. Funny thing is I've had the most problems with WD 2TB drives. The WD 1TB drives I have installed in the pool I've never had a problem with and they are older drives.
 
At least I have access to my music and movies and performance is snappy.

Yep, it really is magical.

I believe my wife thinks we are running a data center sometimes. I've made a lot of changes recently and everything is a mess, but I have it on my list of things to do to clean up the closet this weekend. When I do, I'll get some pictures and put them up. 

pclausen, would you be willing to detail out your hardware?  I'm just curious about how you have that many drives connected, cards/connectivity type used, if you'd do things differently a second time around, etc.
 
Thanks in advance
 

Which would not help you against a number of things. I would recommend offsite backups for important data (all data actually to the extent possible).

For what it's worth, I run BI on the same "server" as the rest of my stuff. But, what I've done is set up ESXi, and BI is on a separate VM, and each of my VMs have a dedicated NIC. I've only given it two cores and 4GB of RAM, and works just fine at about 50% utilization under full load. Also, it stores to a separate pair of drives than the rest of my machines. This way, no matter what BI does, it never affects the rest of my system. This will keep you from needing to have a separate system just for BI. I'm sure it work just fine under HyperV if you wanted to go that way.
 
My BI implementation utilizes 7 3MP cameras and two 1.2MP cameras. The 7 3MP are hardwired using PoE, the two 1.2's are wifi, all configured for direct to disk. All cameras come into a different switch on a different IP range, and as I said a separate dedicated NIC. The other VMs I have truly have no clue that the BI machine is even there. The same server is running WSE2012, so it backs up my workstations on the local LAN NIC, and it is also my IIS server for a few websites.
 
I just checked power consumption on my server - 77 watts. 6 hard drives, Xeon E3-1246v3, Intel server motherboard, etc. I am using a Seasonic gold plus power supply, that may help a little bit.

Another advantage of Drivepool is that if you have a specific folder that you want extra protection for, like pictures, you can set it to duplicate across more than just two drives. So you could have three copies if you wanted.

Using 'dir /o:n' resolves the issue in cmd.exe and doesn't rely on any default behaviour (which has changed between releases of Windows). Of course it means changing your batch files but that is a one time hit.

I'm going to jump the gun and post my conclusion here: my issues were most likely caused by my add-in controller, even after installing its driver. I haven't yet run the burst tests, but here is a list of events that point to it:
When I first built the pool and relocated some pooled drives onto the new controller, those exact drives began to exhibit file access issues. I didn't mention this when I originally posted because I didn't notice issues immediately after doing so. Removing those same drives and reading them with my USB HDD dock exhibited no access issues (nor corruption). When formatting a DIFFERENT set of drives on the suspect controller, the formats would freeze. This was attempted three times. After flashing the firmware, the formats appear to be progressing normally and are further along. (To be clear, updating the DRIVER seemed to make no difference.) Currently, my pool is rebuilt from drives using the onboard controller only. There are no I/O issues. I could have done more to isolate the controller as the point of failure, but at this point, noticing how everything works perfectly fine until it's connected to the Rocket 640L is good enough for me. I guess by shopping for a cheaper card, I paid the price. So far, the fix appears to be to flash the stock firmware. For anyone looking, the Rocket 640L (not to be confused with the RocketRAID 640L) uses the Marvell 88SE9230 controller.
 
I've started a separate pool for the controller to verify controller stability before I let that controller contribute to my main pool. Also, burst tests will be underway soon.
 
My thanks to Christopher for his patience and guidance. To show my appreciation, I've tried to be thorough in this thread in hopes that anyone who runs into a similar problem with the same card finds this thread.
 
- AJ

Thanks to eBay I was able to assemble the following parts for surprisingly little.
 
1. Supermicro SC846E16-R1200B 24 Bay 4U Chassis with  2x 1200Watt 80plus Gold PSU.
 
2. Supermicro X8DT6-F Motherboard
 
3. 2x Intel Xeon x5570 cpu's.
 
4. 32gb Hynix DDR3 Server Ram (4 x 8gb)
 
5. Supermicro passive heatsinks (brand new in the box)
 
I loaded Win Server 2012 R2 Datacenter (already had) and installed 17.5TB worth of HD's (already had). More to be added later.
 
Everything I got from eBay could have passed for new and it all works flawlessly.
 
I decided to go with Drivepool versus RAID / SS / other solutions for 2 reasons 1. Worst case senario all I can loose is a what's on the failed HD. The remainder of the pool leaves my data unmolested and accessible in any Windows computer (this is important to me). 2. I've been using it for quite some time now and it's proven to be totally reliable and simple (too simple but very good at what it does).
 
I'll be using this server for the following -
 
1. Data Storage
 
2. Media Server
 
3. Backups.
 
4. Blue Iris Server
 
Given the expandability of this system, it will probably outlive me.
 
Yes, it's noisy and sucks up the juice but it won't be running 24/7 and it's not located in a people area so neither matters.
 
Excluding the HD's and SW which I already had, The system cost me well under $1000. I'm happy.
 
 
 

Totally agreed.  I had a couple of USB enclosures.   Eliminated all of them due those very problems...   I do have two TR5M enclosures connected via eSATA.   Fortunately my server lives in the crawl space basement, and have never had a problem with disconnecting, especially since no one ever comes near the thing except me!