Jump to content
  • 0

A few questions... DrivePool, BackBlaze, future migration path..


mnemonic0ne

Question

Guys,

 

Quick question, I think am at the end of my research and about to move into implementation over the Christmas holiday.  I have tried and invested in EVERY permutation that one can imagine on the home storage front... First gen WHS (HP), second gen WHS (HP), collosal custom WHS build (BIG processor, etc), unRAID, Synologys, etc, but I still haven't found my perfect all encompassing solution... I loved the DriveExtender idea, but I/O issues killed me on the streaming side (one of my 2 primary goals with WHS), thru a ton of cash AND hardware at the issue, but it was a structural DE issue.... Went to unRAID, but really didn't like the tempermant or LINUX foundation.... LOVE my Synology for a lot of reasons, but comes up short on streaming 1080p effectively and cloud storage options are too expensive (Amazon Glacier) or bad experience (Crashplan).

 

So once again I am on a quest.  Here are my requirements

 

1.  Simple pooled storage with some type of redundancy - Check -  I think I have also settled on using DrivePool over Storage Spaces and other 3rd Party Solutions after much research

2.  Easily Integrated into a robust backup strategy - Question??  I have settled on Backblaze after a VERY bad experience with Crashplan. .  

3.  Runs in a familiar environment - Check

4.  Allows integration with Plex - Check by default...

 

My setup will be fairly simple, Windows 7 Professional box (recycled i7-920 and 5 WD RE4 2TB drives that were removed from an unRAID box that has now been decommissioned) with...

 

Plex Media Server

iHomeServer (with iTunes)
DrivePool and possibly scanner

 

My first direct question is anyone using DrivePool and BackBlaze in concert for offsite backup?

 

I will be doing primary file hosting on my Synology DS1513+, and this will also be the device that is always on and accepting incoming backups from my other PC, etc.  With my new DrivePool server firing up only when I need to do hardcore media streaming and transcoding, as well as being a backup target for most of the files that are located on my Synology.  I have high bit rate reference material in ISO and MKV formats that will NOT be backed up, but have down-sampled M4Vs that will be backed up to both to my DrivePool server and then to Backblaze.... At least that is my plan.

 

And before anyone asks, my issue with Crashplan is it NEVER finished backing up my files even after I seeded my backup with an external drive they sent me..... And consequentially I lost a LARGE number of files.....

 

Lastly, I also wonder about future migration paths to a new OS (Windows 10) and replacing/upgrading drives.... From what I've read it is as simple as installing a new OS and installing DrivePool again and it will recognize the pool?  On the drive issue, I understand as long as I have enough free space in my pool or if I can easily add another drive then it is as simple as removing the old drive and adding the new (or if not enough space, adding the new drive and removing the old)?

Thanks in advance!

M

Link to comment
Share on other sites

6 answers to this question

Recommended Posts

  • 0

M,

I'm curious what your main reasoning behind having a NAS and a WHS? From my understanding the Synology has a iTunes service which, from what I've read but have no experience with, should be able to provide iTunes service to your AppleTVs/iPhone/iPads. What other devices are you planning on using? You mentioned for heavy steaming, how many simultaneous devices are you planning to have running? You also mentioned using iHomeServer (which runs your iTunes as a service and now provides DNLA services) why not use that to provide for both iDevices and other streaming devices?

 

I would additionally ask the following questions:

1. Will your WHS be a standalone server or used as a desktop?

2. What devices will need access to media? Is there a common media format between the devices?

3. Why the need for ISO and MKVs? Do you know those are not valid formats for iTunes? Is that why you mentioned Plex, if so that will only be used for non-iDevices? Why not encode with handbrake into a format that all devices can use?

4. Are you running any additional applications or services? example be VMs, handbrake, SQL, etc.

5. What are you data requirements? Total today and how much growth expected over the next year? Can you NAS handle that or will you need to upgrade drives or model within the next year or two? Will your current WHS be able to provide for all your data needs?

 

I ask all those questions because you seem to be caught between using a NAS and a WHS, both are valid solutions but why not focus on one instead of having multiple streaming devices? How are you going to organize media between the two devices? 

 

A single solid WHS should be able to handle all your needs. Leveraging DrivePool you can have duplicate or even triple copies of data, provide a platform for hosting user shares, leverage a offsite cloud backup software solution and a streaming platform. You don't need a lot of CPU power to stream to multiple devices, I was using a old P4 quad to stream to multiple AppleTVs, iPhone, iPads and computers with no performance issues. But you need to determine what else your WHS will be used for. 

 

And regarding crashplan backups, depends on how much data you have, your outgoing bandwidth and timeframe to backup you are willing to accept, backblaze will likely be similar to crashplan. Just to give you a little background I have been using Crashplan for about 3.5yrs and currently have 14TB backed up. I spent roughly the first 6months backing up all my data. I've had to do a restore of about 10-12gb a couple of years into it, but I didn't have any issues doing a restore over a couple of days. The key is creating backup sets and organizing your data and setting priority. I currently have 5 backup sets created and in order of priority. This way the most important items are always backed up first and cascaded based on relevance to me and acceptable risk. 

 

1. User Data & Pictures (this includes key user profile data like documents, emails, financial data, work documents, iOS backups and pictures)

2. Misc stuff (I have all my car photos, tune software, basically random but important items to me)

3. Software (basically any online purchased software, copies of software CDs, license keys, etc)

4. VMs (I keep quarterly backups of VMs. You will not want to have an active VM being backed up as the amount of changes will cascade and never catch up, if the VM if important consider every other day, week, or monthly backups)

5. Media (this is all my audio books, ebooks, music, movies, TV shows)

 

And to add an additional layer I have all my important used documents and pictures stored on my desktop(used daily) which gives me local access. These items are scripted to copy to my WHS weekly into my user share, that share is part of my User Data & Pictures backup set. So now I have 3 copies saved, 2 local and 1 offsite. Plus this saves me of only needing a single Crashplan backup license but still having data being backed up from multiple computers going offsite. So I would suggest considering setting up something similar to leverage your local copies and hopefully a single repository to be used to backup to Backblaze. 

 

These are just some ideas to consider. Its taken me a handful of years to get organized and have my data coordinated and streamlined. If you are interested in my hardware you can see my post under the DrivePool > Hardware > My Rack Server post. 

 

CTopher

Link to comment
Share on other sites

  • 0

That's a lot of information, and I'll see if I can answer it as best I can.

 

 

You mentioned Drive Extenders bad performance, we don't have that issue. Part of the reason that happened was that they were using some more obscure features of NTFS and implementing it in a non-standard way. This cased some pretty bad performance issues.

StableBit DrivePool is a virtual file system. Meaning we don't store the file system anywhere but in memory. This means that you will see closer to native disk speeds. In fact, if you are duplicating, you may see better than that, as we do "read striping" in this case.

 

As for backup strategy, it depends on how BackBlaze's client works.

I'm trying to test that currently (though it seems to not want to download the client ATM).

But I suspect that it shouldn't have a problem, but you never know.

 

 

As for CrashPlan, I'm sorry to hear that. I've heard of weird issues with it, in the past... which is why it's always a good idea to test out recovery, to make sure it's working as expected. (also, there is no such thing as too many backups).

 

As for migration, yes, it's as simple as reinstalling StableBit DrivePool and reconnecting the drives. It will recognize the pooled disks and recreate the pool automatically. 

And as for the drives, you can add or remote disks at any time. ANd this is (usually) a simple and straight forward process as well. Just connect the drive, and add it.  And removal is basically just a couple of clicks (and a long wait as it moves files off of the disk). 

 

 

Though, CTopherC has some good points in his post, so it may be worth reading all the way through that, as well.

Link to comment
Share on other sites

  • 0

Guys

 

Thanks for the feedback. Let me answer why I'm doing what I'm doing.

 

CT - As you said you have spent years to get to a solution... Me too... I started with the HP EX475, so I've been trying to find the magic bullet for a home storage solution for 6 years.

 

To your points.

 

iTunes server on Synology only works for audio... NOT video, so no joy on streaming to an Apple TV or iOS device without an intermediate step. iHomeServer or Plex would be that step for me.

 

1. Stand alone server - recycled i7-920 that just was taken out of my workstation, with 2 x 5 bay SATA iStar hot swap cages (would be starting pools with 5 repurposed WD RE4 2 TB drives, cages and drives from previous unRAID setup), 256 GB SSD for OS and apps

2. Devices - iOS mobile devices, Apple TVs, a couple of Samsung TVs, XBOX One, 2x OpenELEC HTPCs and 1 WMC - Most clients running Plex, WMC running MyMovies

3. ISOs and MKVs - For reference quality material being streamed high bit rate 1080p to capable devices (and can be transcoded on fly with enough horsepower to other devices)

4. Other Apps - Currently this will be purpose built for Plex/Media and Backblaze, no VMs, no other apps right now. I may transition my home automation platform to the box at some point, but that will be when I upgrade CPU to something with better power efficiency, but similar trancode capabilities (newer i5 with lower TDP but high Passmark scores)

5. Data requirements - Current Synology is 5 x 3 TB WD Reds in SHR, so 12 TB addressable space... Currently 2 TB free, so some room to grow and time to wait before 6 TB drives drop a bit more. And yes either solution would have the ability to handle my needs with upgrading drives to larger models or adding expansion bays

 

As for concentrating on a single solution, part of my desire is not to be reliant on a single point of failure, so redundancy is a overall goal. The real redundancy in the media department will be the smaller M4V (MP4) files will be on both solutions, allowing me to back them up locally and remotely. Organization will be done using simple rsync targets and commands built into my Synology and into DeltaCopy, or using GoodSync.

 

On the number of streams, it is all dependent. There are 5 of us in the house, so there could be 3-4 streams happening at any time.... Or none.... With my lower quality M4V files my Plex Synology service can handle the 1080p streaming natively to most of my devices (DirectPlay in Plex parlance), but for "reference" files I use uncompressed MKVs so I can enjoy a BluRay quality experience without the disk my Synology can't handle these at all, which is part of what kicked off this science project. Also if I want to bring a movie up REALLY quick I can just rip 1:1 to MKV (20ish minutes) and go until I transcode (1-2 hours).

 

In short I keep MKV for high quality reference material or favorite movies, and I trancode everything to M4V at some point to keep in a smaller package, but "lower" quality.

 

Running both my Synology and a new home server (will have to be Windows 7 or 8 for Backblaze) will allow me to put the big server to sleep when high quality streaming and media are not needed 16-18 hours a day, and the Synology will allow steaming of the M4Vs to mobile devices or kids devices where quality is not as demanded.

 

Additionally, as I mentioned before I am a redundancy fanatic, so the NAS and server option gives me the local redundancy I want plus I will get the offsite backup with Backblaze. My experience with Crashplan was horribly slow and my backups (~4 TB) NEVER finished after running it for over 16 months and seeding it with a 2 TB drive and a 100/30 Mbs Comcast Business connection. The server crashed and I lost the other 2 TB, needless to say there was much cursing and re-ripping time to replace the data (movies).

 

Now I might end up transitioning back to a more server centric setup if I find the stability, reliability and feature set on par with what I have with my Synology. But since I already have my Synology up and going, and it does a REALLY good job with its feature set, I don't have a real reason to take it out of the loop right now. And to be honest, my need for a Plex server for on the fly transcoding and a Backblaze client was the real genesis of this current quest so I started simply by looking at Storage Spaces as a simple bridge, but since I've done my research, DrivePool seems like it may allow me to swing back towards a server approach; and since I have enough surplus hardware a new mad science project just kind of blossomed.

 

So to wrap up.... I think I know where I'm headed and why I am headed there, I've taken many a path to get here and most likely there will be more paths in the future.

 

M

Link to comment
Share on other sites

  • 0

mnemonic0ne,

 

I can definitely understand your setup, and the desire for redundancy. 

However, the point that I think that ctopherc was trying to make here was this:

 

The synology NAS device and the PC are both working as NAS devices. Why have two NAS', where you have to split you storage between the two. This means that you need significantly more hard drives to store all of your data.

 

One of the nice things about StableBit DrivePool is the built in duplication option (which you can set to higher than just x2 if you so desire). And if you wish to store the data, then a USB or eSATA (ideally) enclosure may be a better idea. Just connect it when you need it. Additionally, a multi-bay enclosure could be used to host a secondary pool, which you could then replicate the important data to. This may be a better (or at least cheaper, and more power efficent) solution for you in the long run.

 

However, this is your setup, and you can do whatever makes sense for you. But it is a suggestion.

 

 

As for the transcoding, Plex is fantastic. Also, the Core i7 CPU you listed should be able to stream to several devices at the same time. However, if you're using OpenELEC, that should be able to handle the MKV (or other formats) natively.k

 

 

As for the BackBlaze client, I was able to get it installed eventually... and it appears that it will backup from the pool without any issues.

 

And again, you're not the only one that I've seen that has reported ... well the exact same issue with CrashPlan... :(

ALso, nice connection. Reminds me, I need to call Cox and renew my business plan (and see if I can get better bandwidth now).

 

 

 

But again, please do list to our suggestions, but figure out what fits you best. There are definitely a bunch of different solutions that are possible. But only one will fit you.

Link to comment
Share on other sites

  • 0

Storage Spaces is a block based solution, much like (if not almost identical to) RAID.

Like RAID, the data is stored in a "raw" format, and read and written to in that style. 

 

What does this mean? That the data is in a proprietary format, and may not be readable outside of the system (you should be able to migrate storage spaces to another system, but I've not tested this....).

Another issues, is that since is a software implementation, it relies much more on your hardware. You will want a good CPU (it does a lot of number crunching with a PARITY array), and that you will want to have good quality RAM (ideally ECC). 

 

 

DrivePool is a file based solution. Meaning that the data is stored on the disks as normal files. Accessible any time. 

Also, you should see native disk speeds when accessing the files (or potentially faster with duplication enabled, and Read Striping).

 

 

The biggest difference here is when it comes to disaster recovery and/or migrating away from the pool.

With Storage Spaces... there are a couple of companies (and only a couple) that boast the ability to restore data from Storage Spaces. And their software is $$$$.  With DrivePool, ANY recovery software that is capable of accessing NTFS can be used to restore the data.

 

As for migrating away from the pools... with DrivePool, you can just uninstall it and then move around your files as needed.

With Storage Spaces... it's a CF. THat's the nicest way I can describe it.

But basically, you need to have enough space to empty the Storage Spaces volume.... You may be able to remove some of the disks, so you can play "musical chairs" with the disks... but chances are that you may not be able to....

 

 

 

 

 

 

Another important point here, if you're on a Server OS, using Storage Spaces, and using parity, you should ABSOLUTELY be using ReFS. It supports automatic correction of damage data, by using the parity information. 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...