Christopher (Drashna)

Hi I ran into power issues once and it was related to the psu my server was set to do all its maintenance tasks in the early hours and was drawing too much power in doing so eventually the 5v went pop and I lost 2 drives and a few bad block issues on a few more of them since then I always go for a gold rated over the top psu. I also make sure the power supply has enough connectors without using splitters.

No, that wouldn't work. Since we store the actual files on each drive, there isn't a drive large enough in this situation to store the file.  
 
The OS may see all the space available, but the driver for the pool is intelligent enough to know that no disk in that pool as enough space to store the file, and gives you an out of space error.
 
 
Specifically, when you go to create/copy the file, Windows normally queries for free space (to see if it will fit).  The DrivePool driver gets that query, and then queries each individual disk (based on the "real time placement limits", if any), and looks at those values.  If no disk has enough space to put the 4GB file (and in this example, no disk has the space), the driver returns the "not enough space" error.  This is passed to the program querying, and causes the operation to error out with the appropriate error.
 
 
And if the entire pool was full? Then it would cause problems.
However, one of the balancers, "Prevent Drive Overfill" tries to keep all of the disks in the pool 10% free, or 100GBs free. 
This can also cause issues when using the SSD Optimizer balancer, if you have very small cache/SSD drives.
 
Also, you could use the Ordered File Placement Balancer to minimize this possibility, as it fills one (or two, with duplication) drives at a time.
 
 
If we used a block based solution (such as how StableBit CloudDrive, RAID or Storage Spaces works), then yes, that would work fine. But you're storing the blocks of RAW data, so it's not easily accessible without ALL of the blocks, and those would be divvied up between the different disks.

Now I am really jealous. Do you actually run them through an emulator?

Well, ideally, you want our products to just sit there and do their thing.  
So I'm glad to hear that StableBit DrivePool has been working quite well, in that regards!

Hi Carlo, welcome Philmatic.
 
Looks like Carlo is king but as he said 33TB isn't anything to sneeze at. Especially if it's running, working and reliable.
 
I can certainly see your point on duplicating everything Drashna however that's basically the reason I run DrivePool. I have my important stuff duplicated and the replaceable stuff unduplicated. It's nice to pick and choose with folder duplication. I haven't really done the math but I *think* if I wanted to take that big of a hit on drivespace I'd run a conventional RAID. Maybe. (Not that there aren't plenty of other advantages to Drivepool..and I do need to rethink some things)
 
I was kidding about my Drive Pool being ALL PORN however the majority of my pool is video. (my personal favorite being older TV) however I know I fit the definition of data hoarder. I've got every file and picture that's ever been on any of my machines since around 1997..heck.. even from before that. I've got all of my C-64/128/Amiga disc images, all of my old BBS files, clients, business' etc. all online.
 
I also bought the first consumer digital camera that ever came out and went through several girlfriends with that one so needless to say "stuff adds up!' 
 
This new 4224 build is going to be Server 2012 Standard and I think I'll have some fun with deduplication. I can see it has it's place.
 
I really want the disk subsystem to fly on the this new 4224 build.
 
On hand I have a HighPoint 2740, a LSI 9240-8I and a Dell PERC H710P I scavenged in addition to the ports available on the motherboard, a Supermicro X-10DRI-O however I am not opposed to spending some "real money" on a new controller...
 
Heck, here's a picture of most the hardware that has arrived so far, all sitting on top of my running 4220 on the mini "bench." I'm pretty sure the only thing missing in this picture are two additional EVO 850's that just showed up. (for a total of four)
 
I do also have six 600GB 15K Cheetah SAS drives but I don't know if they will make it into this build or not. They are used with about two years of power on time but barely any usage.
 
I'm pretty sure I'm going to populate most of the bays in the Norco with 7200RPM HGST NAS drives or WD Reds.. I haven't made up my mind yet.  
 
The 4224 case just arrived and is sitting next to this 4220 awaiting insertion of parts. 
 
Decisions, decisions.
 
~RF
 

 
 
 
 

That is not a shabby amount of storage by any means.  Guarantee you have more storage than 99.99% of your local friends!
 
You can store a HUGE amount of media on that storage pool for sure. Basically your own personal version of Netflix.

Depends for me on which way I want to go.
For 8 bay http://www.newegg.com/Product/Product.aspx?Item=N82E16817707367
For 15 bay http://www.istarusa.com/raidage/products.php?model=DAGE415U20-PM#.VYI2zflVhBd
 
The StarTech eSATA 8-Bay Hot-Swap SATA III Hard Drive Enclosure with UASP works quite well and pretty cheap at around $300 plus shipping. Connect it up via eSata, USB3 or UASP and it works quite well with DrivePool.  I picked one up a couple of weeks ago and put 8 4TB drives in it and haven't had a single issue with it.  I've first tested eSata, then UASP and then normal USB3.  I left it connected via USB3 and haven't had a single issue.
 
It's not quite as fast as the 15 bay could be since you have 8 drives on a channel vs 5, but for normal (non benchmark use) and Plex use it works just fine.  I stream to anywhere from 6 to 10 people in the evening and it handles that load without breaking a sweat.   What I like is that you can take this box and plug it into just about any other computer/notebook or router and be able to access your data.  One of these boxes and a higher end home router like an ASUS gives you a nice NAS box.
 
The reason I'm leaning toward it is that it's very quite and keeps the drives cool.  It's not a rackmount unit but I don't really care about that for a home unit since it's versitile and works well via UASP/USB3 so I don't need to worry about sata/esata ports.  I still have a few sata ports available (8 I think) but I'm reserving them for internal drives.
 
I could go SAS also but really just don't honestly see the need for a media server when a simple box like this can easily add 8 drives at a shot. Throw 6TB WD Reds in it for a reasonable large amount of data at a decent price.
 
Carlo

 
I've got a 24 core (48 with hyperthreading) SuperMicro server with a few 1 TB drives in it running over 75 virtual machines.  They are all 2012 R2 installs.  Machine flys.
 
Not only does the de-duplication help with storage but also helps the machines run faster as it helps with caching/memory and other important items (not trying to be technical).
 
Carlo

That may all be true for the HGST drives but that does not apply to the Seagates. The Seagates actially come with firmware that optimises write-behaviour. It's "the other approach"  that that article refers to. There are no compatability issues with the Seagates.
 
And yes, due to SMR you may suffer write penalties but if we are realistic, you would have to have quite some heavy I/O to actually suffer/notice this. Opening a word doc, changing it and saving it? No issue. Movie editing? Not sure, perhaps. OLTP-databases? You typically simply do not want to risk degraded performance so no. But I really believe the use-cases where write performance would be an issue are very limited.
 
 
 
OK, but who writes 100TB a year (and wants to keep only 8TB of that)? 
A good review was done here: http://www.storagereview.com/seagate_archive_hdd_review_8tb
 
In the use case of OP, writing in batches of 20 to 30GB at a time, mostly to retain, these are a great deal IMHO. Oh, and read speeds are crazy.

I also do the fly by the seat of my pants method..Everything ctitical is backed up three fold or more... If its a few TB of movies then c'est la vie.
This approach may change though when the 8TB drives come down in price a little more.
 
As for DrivePool itself... well (as another Plex user), the best endorsement I can give is that I have been using it since the very early beta stages. 
As someone who used to be extremely active on here I just don't have the time any more. BUT I don't need to be because it just works!!!
Im well over two years of not having touched my server (windows updates aside) in fact I actually feel guilty that I guess I just take DrivePool for granted. It sits there silently in the notification area doing its stuff. 

Hi all,
 
Just wondering if anyone is using the new Seagate Archive drives yet?
 
The 8TB options are extremely competitively priced here in Aus (were down to 35c/GB other day). The drives certainly have their limitations however, mainly that their sustained write speeds are horrendous. They have a cache which will work very well up to 20GB, then slowly degrade till ~80GB or so where it becomes atrociously slow; this is just the nature of the SMR technology being used in a consumer grade product. I personally have no qualms with this. Yes, it will take forever to copy to initially, and would be quite frustrating for RAID due to the rebuild times.
 
However for someone who typically does a one time write and multiple read (ie WORM), they are fantastic value drives and aside from the initial population this issue would likely rarely occur. I usually only copy over batches of 10-20GB files at a time, normally batches off the DSLR.
 
Looking at grabbing 2-3, but currently only have the currency for 1, so looking to combine with my existing WD greens for the time being.
 
What I am wondering though, is would any of this interact strangely with Drivepool? Are there any limitations I should be aware of or need to address? In particular mixing the archive and green drives in a pool? I pressume the green drives will be limited somewhat by the archive drives if I chew through the archive drives cache.
 
Anyways, any opinions welcome, just works out a crapload cheaper than purchasing WD red drives here in Aus.
 
 
Cheers,
Luke

Hey Guys -- Sorry to bail on this thread so long ago. After I read it Mr. Drashna got me thinking.... 
 
Let's say this:  It appears that it's a bad idea to have every piece of network hardware in the house(s) on battery back-ups --particularly when it's cheap hardware.   
 
The power in the neighborhood hasn't gone out for a significant amount of time which meant I had several "junk/dumb" GB switches (meaning Trendnet, TP-Link and the like.) that haven't been power cycled in many, many months.  
 
So.....I'm not sure which switch/router was the problem but after power cycling all of them (and doing nothing else) everything basically matches your guesstimates above.
 
So with this system now up, running & stable I'm about to begin a 4224 build.  Once that's up and running I can revisit the 4220 box and clean up a few loose ends that are bugging me..but at this point it's 100% stable.
 
With my new 4224 build (Meant to separate my media server from my storage server)  I can do things a bit more slowly and concentrate on a few key points. 
 
Originally I was just going to upgrade the motherboard in the 4220 and call it a day but at this point I can't afford any significant downtime and wanted to spend time tweaking (specifically the network side of things) so I decided a totally new build was in order and began collecting all the pieces which have mostly arrived.
 
I'm sure you don't remember me Drashna but you came to my rescue a couple times (On two separate Sundays no less) during the 4220 build which was my first "serious" experience with DrivePool & Highpoint controllers... I sincerely appreciate it.
 
~RF
 
 

RFOneWatt,
 
I just have to know.  What are you using all that space for??  Video?

I'd like to start by saying I've been using DrivePool+Scanner for a while now, and they are seriously awesome. Everything WHS should have been. So much thanks for that.
 
I only use it for home use but the one thing I'm currently lacking is an offsite backup. That's where CloudDrive comes into play for me. Being able to use multiple cloud providers with DrivePool is really powerful. Before now I wouldn't have considered it as I don't want to trust the cloud with my private files, but the encryption part of CloudDrive fixes that.
 
So basically my use case is to pool as many providers as possible, and backup only the most essential data. Speed isn't important to me.
 
On to my first experience. I installed CloudDrive and hooked up dropbox, amazon and hit a wall with microsoft onedrive. I eventually found the forum post (grateful it was stickied) and enabled it. I realise that their service is poop, but would be nice to be able to enable in the UI with appropriate warnings.
 
I put about 10mb of test data in all of the providers and found that they all generated lots of errors and didn't seem to get anything uploaded. Eventually worked out by reading through the forums that it's probably because my connection isn't fast enough (8mbit down, <1mbit up). After massively increasing most of the timeouts, reducing the threads to 1 and destroying/re-creating the drives with 128k chunks instead of the default 1mb I managed to get it stable.
 
What would be cool is if you had a benchmarking feature in the app, a bit like scanner's burst test. Then you could potentially have an "autodetect" settings feature, as finding and editing a config isn't the most user friendly first experience. You could also warn the user that their connection is less than optimal.
 
The only other thing I messed up was that I assumed that CloudDrive was encrypted by default. Would I be right in thinking that with encryption unticked, the chunks in the cloud would be readable somehow? I couldn't find a way to enable encryption on my cloud drives after I had made them, so in the end deleted them and created them again.
 
It would definitely be cool to have an overview panel of all of the connected drives. At the moment you have to hop through all the tabs to see what, if anything, is going on.
 
Anyway hope this feedback is helpful, keep up the fantastic work.
 
 
Max

Here is my configuration along with a custom built rack.
 
Server
iStarUSA D-412S3-MATX 4U Rackmount microATX Server Chassis
Intel Core i7-3770K Ivy Bridge 3.5GHz (3.9GHz Turbo) LGA 1155 77W Quad-Core
Intel Media DQ77MK Desktop MB, Intel Q77 Express Chipset, w/TPM
Corsair Vengeance LP 32GB (4 x 8GB) 240-Pin SDRAM DDR3 1600 (PC3 12800)
Intel 80GB 330 Series Maple Crest SSD
Antec EarthWatts Green EA-380D, 380W
Noctua Ultra Silent CPU Cooler Cooling NH-U9B SE2
3x Noctua NF-R8 PWM 80mm Case Fan
LSI 9200-8e SASController Card, 2 x SFF-8088 mini-SAS External Connectors, JBOD only
Windows Server Essentials 2012 R2
Stablebit DrivePool v2.1.1.561
Stablebit Scanner v2.5.1.3062
IHomeServer v3.1.76.0 (iTunes 12.1.2.27) feeding 2x AppleTV 3
Crashplan
 
 
Array
NORCO DS-24E 24-3.5" JBOD Enclosure
 
Pooled Drives - 59.2TB
2x 8TB Seagate Archive HDD
1x 6.0TB WD Red
1x 4.0TB WD Red
7x 3.0TB WD Red
5x 2.0TB WD Green
1x 2.5TB WD Green
3x 1.5TB Seagate
1x 1TB WD Green
Non-Pooled drive
1x 1TB WD Green - OS Backup drive
 
UTM
ARK IPC-1.5U1525 Black 1U Rackmount Server Case
Intel S1200KP Mini ITX Server Motherboard LGA 1155 Intel C206 BIOS
Intel Core i3-2120T Sandy Bridge 2.6GHz, 35W Dual-Core
Stock Intel cooler
Noctua 60x25mm A-Series Blades with AAO Frame, SSO2 Bearing Premium Fan
G.SKILL Ripjaws X Series 16GB (2 x 8GB) 240-Pin SDRAM DDR3 1600 (PC3 12800)
Intel 60GB 330 Series Maple Crest SSD
100gb 2.5 HD
Intel Dual Port Pro 1000 PCIe NIC
picoPSU-160-XT, 160w output, 12v input DC-DC Power Supply
Ubiquiti Networks Unifi UAP-Pro Enterprise Dual Band AP
Untangle v11.2
 
Dell PowerConnect 2824 24port GB managed switch
Dell PowerConnect 2724 24port GB managed switch
 
VMware ESXi 5.5 Home Lab server
ARK 1U125 Black 1U Rackmount server case
Supermicro A1SAi-2750 mini ITX MB
Supermicro PWS-203-1h 200W 80 Plus Gold
Intel Atom C2750 2.4GHz Avoton 8 core CPU
32gb Kingston DDR3 1600 ECC memory
LeefSupra 16gb USB 3.0 flash drive
1x 480gb Intel S3500 SSD
2x 240gb Seagate 600 Pro SSD (200gb)
1x 128gb Samsung 810 SSD (cache)
1x 64gb Intel 330 SSD (cache)
 
APC Smart UPS SMT1500RM2U 1000W 2U rackmount
 
 
Ctopher

Someone has to have it, right?
 
How about we start with the largest pool of the members that participate here?
 
I'm taking an uneducated guess that theoretically Drivepool should scale indefinitely (or to some insane limitation imposed by the O/S, hardware or something else) however we all know the real world is where it's at, yes?
 
I'm sure I'm nowhere near the largest but I've maxed out my Norco 4220, and then some.
 
It's NOW time to start building the successor! 
 

 
 
 
Would love to see what everybody else has going on!
 
~RF
 
 

im definitely buying five

Well I'm in europe... and the Euromillions lottery is up to £95m this week...  and thats lump sum and tax free

The "File Placement Rules" feature can do this, actually. 
It may require extensive micromanagement, depending on how your pool is setup, but the UI should allow you to easily do this.
http://stablebit.com/Support/DrivePool/2.X/Manual?Section=File%20Placement
 
As for fragmentation, you can defragment the disks in the pool, without any issues.
 
 
 
So that is a drive does fail, you can be certain that you have all of the contents of a particular folder (or none at all). 
 
 
I've been there (not blue ray rips, but re-encoded files), and it sucks.  It may be worth investing in enough drives to duplicate everything.

I would be very interested in a product called "Stablebit Filesafe" !

Thank you, Christopher!  The 'shotgun' effect is exactly what I was trying to describe.  
 
Again, it's amazing the level of support and "don't feel like you're looking down" on people that you all provide here.  On the FreeNAS forums and some other places, it's really very hostile.  That alone gives you a huge leg-up, plus having a really polished product.  I'll be ordering the combo-pack tomorrow!
 
Mike

We're definitely using the new API.
However, in regards to throttling, but API's seem to be throttled.
 
And i you want to see just how badly, check out the dev site:
https://dev.onedrive.com/README.htm
Check the bottom part out, where it says "Throttling".
It delays you for 3600 seconds (one hour)! That's..... horrendous. 

Christopher,
Alex found that software (not sure what) was overwriting the GPT.  Alex ultimately made the physical disk read only so that it will not be overwritten.  Currently, the pool is functioning as designed.  Therefore, the issue appears to be resolved.
 
 
Thanks,
Todd

I ordered the motherboard and CPU yesterday. I decided to bump up the CPU to a Xeon E5-2620V3 2.4ghz processor $430.00. It has hyper threading and a few other goodies that the 1.9ghz Xeon does not have. Stupid CPU costs more than the motherboard! Newegg was out of memory, so I'll have to get it elsewhere. While I'm at it I might as well get a SATA III SSD because I have a 256gig SATA II SSD for the OS drive now. What I paid for my current SSD I can get one with double the capacity and faster for a little less money then what I originally paid!
 
Sigh... :-)

Honestly, it depends on how you want to do it.
 
If you have the space available, you can remove the drive you want to replace first. 
 
But you could absolutely add the new drive, and then remove the smaller drive from the pool afterwards.
 
 
 
However, what I recommend here:
Remove the drive from the pool. Use the "Duplicate files later" option. This will remove the drive quickly, as it's leaving the duplicates on the drive. After the disk is removed, it will recheck the duplication status of the pool, and reduplicate files as needed.
Before it starts doing that, physically remove the "old" drive, and connect the larger drive.
This way, when it starts duplicating the data, it will use the new drive most likely. This will minimize the reshuffling of the files on the pool.

Well, PCI-X is still pretty darn fast, so either way, you should be fine!
 
And to make sure, I recommend the IR (RAID) firmware over the IT (passthrough/HBA) firmware. Both can do passthrough, but the IR seems to behave better. Besides, extra options are great.
 
 
As for Intel SAS Expander card, great catch! And yeah, I've only been able to find them for $200 or so. So finding one for $60 is fantastic! (if you see another, link me!)
 
 
As for the BSOD, it indicates a CLOCK_WATCHDOG_TIMEOUT bug check. This indicates that an expected clock interrupt on a secondary processor, in a multi-processor system, was not received within the allocated interval.
https://msdn.microsoft.com/en-us/library/ff557211(VS.85).aspx
As for the hotfix, it may be OS specific. And hotfixes aren't usually pushed through Windows Update. They're usually option installs offered to fix specific issues.
This is the hotfix you're referring to:
https://support2.microsoft.com/default.aspx?scid=kb;EN-US;975530
And if you check lower down on the page, there are a couple of registry tweaks that you can try, and see if it works.
 
 
 
If it happens again, set the dump to us:
http://wiki.covecube.com/StableBit_DrivePool_System_Crashes