Jump to content
Covecube Inc.
  • 0

Question about duplicated folder (different speed drives)


Question

Hello all. I have not yet had a chance to download drivepool, since I am waiting for my synology NAS to come, but first I have a question. 

 

I am thinking of doing a duplicated folder setup. The source folder in my PC contains a VMWare VM on an SSD drive. The target folder for duplication will be on the synology NAS with a HDD.

 

Obviously there will be a huge speed difference between the write speed of the source and target. How will drivepool handle this huge discrepancy in write speed, is this even possible to do or inadvisable.

 

----

 

During heavy VM operations eventually there will be a big and continually growing discrepancy between the data written to the SSD and the NAS HDD. At some point I would guess DP will crash.

 

Or will DP limit the write speed of the SSD to match that of the target NAS HDD speed.

Link to post
Share on other sites

15 answers to this question

Recommended Posts

  • 0

...I am actually not certain that this can be achieved at all.

IMHO drivepool will build the pool on physical disks and only after that you can enable duplication on the pool.

In order to incorporate the NAS, I doubt that drivepool will allow you to build a pool with a network share in it.

The only way of using resources on the NAS, that I can think of, is using it as an iSCSI target and importing that iSCSI disk into your windows system with the iSCSI initiator.

Then drivepool should see it as a physical disk. However, that data will not be shared by the NAS in order to be seen by others.

 

What is your usecase behind that?

 

When using this for duplicating a virtual machine config and its virtual disks, I think there is a lot of risk that the duplicated copy of a virtual disk will be unusable.

This risk exists independent of the way on how you copy it (manually or by means of drivepool duplication with/without matching disks speeds)

The normal way of operating this, AFAIK is to create a VM-snapshot. When you are able to locate the snapshots in an other directory, you should be able to duplicate these safely

and use these to recreate the VM resume state later in case of a crash of the VM 

Link to post
Share on other sites
  • 0

I don't think you should go that route and combine these two things.

 

It is however not uncommon to use an external datastore for hosting VM disks, like via NFS.

If your NAS is available at the time you need to run the VMs and if the NAS performance over wire is enough (with link aggregation and RAID on the NAS there are possibilities that are within the range of a consumer SSD)

you could for example use the iSCSI target method and exclusively move the VMs to the NAS...freeing up all two local disks in your RAID1 VM-datastore

Link to post
Share on other sites
  • 0

Yeah, currently StableBit DrivePool does not support adding networked locations to the Pool. Only physical disks. The except to this may be iSCSI, as these show up as physical disks to Windows, and therefor to us.

 

As for VMWare, if performance is important, then you will be better off with RAID, I think.

Link to post
Share on other sites
  • 0

Thanks for all the responses.

 

Follow up question regarding diff speed drives:

 

Say I have 2 drives in the pool, one is HDD one is SSD. And I setup folder duplication from the SSD to the HDD.

 

During heavy and continuous operation to the SSD source folder, eventually the HDD will lag behind the SSD. If this discrepancy gets big enough, what happens then? (I assume this is not recommended)

Link to post
Share on other sites
  • 0

Well, if you have Real Time duplication enabled, then the writes are done in parallel. 

I beleive that the slower drive would slow down any transfer or access. So if this is an important consideration, then you may want to check out the Archive Optimizer optional Balancer, as this lets you specify feeder disks. These disks are written to first, and then later moved off of the pool.

 

For read access, if one disk is significantly slower, we'll read from just the faster disk, to speed up access. However, if they're close to the same speed, we'll use a technique called "Read Striping". It's basically what it sounds like. We read ahead from both disks, and cache the contents, to speed up access.

 

 

http://stablebit.com/Support/DrivePool/2.X/Manual?Section=Performance%20Options

Link to post
Share on other sites
  • 0

As far as duplicated file performance, here's how it works:

 

The Windows NT kernel is inherently asynchronous. In other words, performing an operation does not block the caller. You may have experienced this in other Windows applications, you click a button and then the whole Window freezes until some operation completes. This is not how the Windows kernel works (where our pooling is done). The original designers had the foresight to make the whole thing asynchronous. Which essentially means that nothing waits for something else to complete.

For example, if I need to read a file I issue a READ IRP (a command) to some driver (some code that someone else wrote). If the driver can't service the read request immediately it tells us "Ok, great! I'll get back to you when the read is complete. Do you have any other requests?". This is how everything works under the hood.

We take advantage of this and basically for duplicated files we issue multiple write requests in parallel. This means that the total time that it takes to complete the request is the turnaround time of the slowest drive.

Link to post
Share on other sites
  • 0

But... while the slowest drive is writing, new commands may be issued? So you could subsequently read another file and it would be read from the fastest drive?

 

Very astute. You've just struck a very important aspect of disk I/O, it's called synchronization (by us programmers). Obviously, if such a thing were allowed then you would end up with file corruption.

 

StableBit DrivePool has a well defined locking model to prevent such things from happening (as does the rest of the Windows kernel), and frankly this is most of the work of writing a reliable pooling solution.

Link to post
Share on other sites
  • 0

Thanks Alex for the enlightening answer. I am now in the process of my trial.

 

 



I don't think you should go that route and combine these two things.

 

It is however not uncommon to use an external datastore for hosting VM disks, like via NFS.

If your NAS is available at the time you need to run the VMs and if the NAS performance over wire is enough (with link aggregation and RAID on the NAS there are possibilities that are within the range of a consumer SSD)

you could for example use the iSCSI target method and exclusively move the VMs to the NAS...freeing up all two local disks in your RAID1 VM-datastore

 

 

Unfortunately my skillset does not yet cover iSCSI and related technologies, although the synology is here now.

 

So until I get there, currently I have my HyperV VMs running in a file-duplicated pool of 2 SSD partitions (of 2 SSD Drives).

 

My next question is going to cover best practices for running a 24/7 HyperV VM in a drivepool drive. If I can't find what I'm looking for from past posts then I'm going to create a thread.

 

-------

 

The reason I ask about different speed is that I am on a laptop where one of the SSD is sitting in my CD Drive with a caddy. And although the SSD is the same type, being in the caddy, the speed is about 30-40% less, probably due to the SATA port of the CD being SATA2 instead of SATA3.

 

The first image is SSD in HDD Port and second is the same type of SSD in a caddy in the CD-ROM slot of the laptop.

 

 

bx4Bpaf.png

 

vYmLV4N.png

 

Link to post
Share on other sites
  • 0

For those of you interested, here is the performance of the drivepool drives.

 

The read speed is similar to a single SSD drive (however does not seem to stripe the data and thus increase read speed) but the write speed is half that of a single SSD drive.

 

---

 

Remember that one SSD is in a SATA3 port and the other is in a SATA2 port.

 

In reading, the pool reads from the SSD in the SATA3 port but what is curious is the hit in the writing speed. I expected the write speed to be close to the speed of the SSD in the SATA2 port but it's significantly lower.

 

 

ihe2yUz.png

Link to post
Share on other sites
  • 0

Unfortunately my skillset does not yet cover iSCSI and related technologies, although the synology is here now.

 

Just check with your synology's manual if iSCSI is supported.

The NAS will be the iSCSI target and you will have to set it up / configure it there.

Your Windows machine will be using that iSCSI disk(s) on your NAS with the use of the iSCSI initiator application (which is available as standard from win7 onwards).

There are many howtos available on the net.

Link to post
Share on other sites
  • 0

Not to bother and I know you're very busy but how would reading _another_ file from the faster drive cause file corruption?

 

Basically you might get into a race condition, where the state of all the files are not the same, and another operation might get data that is inconsistent. So you want to make sure that reads / writes are synchronized properly, and you also need to make sure that the cache is consistent, these are probably some of the most complicated things to code when writing a file system.

Link to post
Share on other sites
  • 0

As far as performance, I did some informal testing today with a RAM disk, comparing file copying speeds of pool vs. direct to RAM disk. The numbers didn't show much of a difference when comparing pool I/O vs. non-pool I/O (about 700 MB/s). I tested both local file access and UNC file access. UNC access was slower both on the pool and direct to disk (~500MB /s). But this was with unduplicated files.

 

I'll try to do some more formal performance testing with duplicated files and see what I can come up with. Perhaps we can optimize duplicated files further.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...