Jump to content
Covecube Inc.
  • 0

Change allocation unit size of all drives in DrivePool: Best way?


Question

Hi folks,

 

I've got 4x8TB and 2x6TB drives in a 6 drive DrivePool with 2x duplication on all files, and 3x duplication for a specific folder.

 

Attached is a photo of current drive stats:

4Vc6s1P.png

 

I formatted these drives with 4098 byte unit allocation size. I want to change all 6 drives to 64KB.

 

I'm looking to find out the best way to perform this change on all 6 drives with minimal risk. Basically looking for a recommended set of exact procedure.

 

In my head I'd be doing this:

  • Remove a drive from drivepool
  • Wait for the 2x/3x stuff to propagate to the remaining drives
  • Format drive with new allocation size
  • Add back to DrivePool
  • (Presumably it won't be moving any data back to this drive)
  • Repeat for the remaining drives

However there must be an easier way. Is there a way to change it without formatting? Without removing it from the DrivePool?

 

I'm sure there's 20 ways to get this done, hopefully I can get some staff or user input on the issue. I'm tech savvy but never dealt with a situation like this before. Not to mention the added complexity of the drives being in a pool.

 

Thanks!

Link to post
Share on other sites

18 answers to this question

Recommended Posts

  • 0

LOL.  I just recently did this on my own pool, actually. 

 

 

There isn't a good way. You need to format the drives to do this (or use a 3rd party tool, and hope it doesn't corrupt your disk).

 

The simplest way is to use the balancing system.  Use the "Disk Space Limiter" balancer to clear out one or more disks at a time. Once it's done that (it make take a few hours or longer), remove the disk from the pool and reformat it. Re-add it and repeat until you've cycled out ALL of the disks.

 

 

Specifically, the reason that I mention the Disk Space Limiter balancer is that it runs in the background and doesn't set the pool to be read only. And is mostly automated.

Link to post
Share on other sites
  • 0

Thanks for your input Christopher, I'll be sure to utilize that balancer and have it move all files from that drive, then pull it out of the pool and format, then add it back.

 

I figured it'd have to be something like that but your idea sounds like the right approach. My main concern is I wanted to ensure that while I was doing all of this moving and formatting, I always had my 2x and 3x duplication intact (aka I wasn't manipulating my drives in a "degraded" state with a single instance of a file in the case of 2x, or 2x in the case of 3x).

 

I'll most likely do that, and as you said it'll take time so I'll have to find an optimal time to do it because I'd like to monitor it while I'm doing it all.

 

Thanks!

Link to post
Share on other sites
  • 0

I figured it'd have to be something like that but your idea sounds like the right approach. My main concern is I wanted to ensure that while I was doing all of this moving and formatting, I always had my 2x and 3x duplication intact (aka I wasn't manipulating my drives in a "degraded" state with a single instance of a file in the case of 2x, or 2x in the case of 3x).

 

Well, removing the disk from the pool normally would do that as well (as long as you're not using the "duplicate data later" option). 

 

But using the balancer is best way for this, IMO. and it clears out disks without having to remove them first (and the hassle with that).

 

And if you have enough space (and it looks like you do), you could clear out a couple disks at a time. Also, when this is going on, there should be an "X" and a ">>" button on the left of the pool status bar. Clicking the ">>" button increases the balancing priority, which should allow it to run a bit faster.

Link to post
Share on other sites
  • 0

Why would you want to change it to 64 kb?   Do you have large media files and looking fro better streaming performance?  

For a number of reasons. 

 

By default, Windows wants to use 4k clusters. This means that each allocation unit for files is this size. If you ahve a 10MB file, it's broken up into 4k chunks and placed on the disk.  The flip side is that each file uses 4k period. That includes folders, which have very little data.  

 

Additionally, even if the file (or the remaining chunk) uses a couple of bytes, it will take up the entire allocation unit. 

 

 

Now, why is this important? Well, a 64kb chunk is 16x the size of the 4kb one.  

This means that more data is guaranteed to be sequential.  This is of one main benefit: Fragmentation. 

 

Sinch each chunk is larger, that means that you're less likely to see fragmentation, Additionally, since more of that data is in once section of the disk, it means that there is less seeking being performanced on the disk, and may increase performance.

 

THis is the same reason some people recommend that you defragment your disk. 

 

 

 

The flip side, as I mentioned, that if you have a lot of small files, you're losing disk space. A cluster or allocation unit can only be used by one file. 

For an OS disk, or one with a lot of very small files, this can cause an issue, as you may run out of allocation units before you run out of disk space.   

However, for storage, a majority of the files are going to be much larger, and will benefit from the allocation unit size increase. 

 

 

 

 

So what does this equate to for you? Well, on my Seagate Archive drives, I saw fewer performance drop offs, and I saw about a 20MB/s speed increase on reads. 

Link to post
Share on other sites
  • 0

What's about SSDs clusters size if they are used with SSD plugin on a pool with 64k hard disks?

Should SSDs clusters match HD's one? or is it better to leave them at rhe default size?

 

It's better to not use the larger allocation unit size for SSDs.  

 

Specifically, because SSDs don't benefit from them. The main reason to use the larger size is performance and preventing fragmentation. Since neither affect an SSD, there is no benefit.

Additioanlly, since SSDs are smaller in capacity, you're more likely to lose a good chunk of space by doing so.

 

 

However, there is absolutely no harm in using the different sizes on the SSDs vs the spinning hard drives. 

 

 

So 64k allocation unit size for "spinners" and default for SSDs is good.

Link to post
Share on other sites
  • 0

For a number of reasons. 

 

By default, Windows wants to use 4k clusters. This means that each allocation unit for files is this size. If you ahve a 10MB file, it's broken up into 4k chunks and placed on the disk.  The flip side is that each file uses 4k period. That includes folders, which have very little data.  

 

Additionally, even if the file (or the remaining chunk) uses a couple of bytes, it will take up the entire allocation unit. 

 

 

Now, why is this important? Well, a 64kb chunk is 16x the size of the 4kb one.  

This means that more data is guaranteed to be sequential.  This is of one main benefit: Fragmentation. 

 

Sinch each chunk is larger, that means that you're less likely to see fragmentation, Additionally, since more of that data is in once section of the disk, it means that there is less seeking being performanced on the disk, and may increase performance.

 

THis is the same reason some people recommend that you defragment your disk. 

 

 

 

The flip side, as I mentioned, that if you have a lot of small files, you're losing disk space. A cluster or allocation unit can only be used by one file. 

For an OS disk, or one with a lot of very small files, this can cause an issue, as you may run out of allocation units before you run out of disk space.   

However, for storage, a majority of the files are going to be much larger, and will benefit from the allocation unit size increase. 

 

 

 

 

So what does this equate to for you? Well, on my Seagate Archive drives, I saw fewer performance drop offs, and I saw about a 20MB/s speed increase on reads. 

 

 

 

 

 

Sorry to bump an old post but I just wanted to say thanks for this. I just added another 8tb Seagate today and decided to format at 64k on the strength of this thread.

Without the speed boost button I'm seeing 60-90 MB/s constant. (It didn't drop below 60MB/s in a twenty minute period.)

 

With the speed boost enabled I do see drop offs, but much less frequently than on the previous 8tb (formatted at 4k) and by the same token I'm hitting 180-210 MB/s at times. Happy days. :)

Anyway this latest drive addition will give me available space to empty and reformat the other drives over the next few weeks so I'm gonna do it. .

 

Going back to your comment about fragmentation, I understand what you mean about larger clusters should lead to less fragmentation.. Am I also correct in thinking that if over time fragmentation occurs, the larger clusters will ALSO make it easier to defrag?

I will add that all the Seagate archive drives will hopefully be write once as intended, so this question mainly applies to some of the 4TB drives that are much more read/write intensive.

 

Either way thanks so much for your in depth reply to the guys in this thread. It all makes plenty of sense and definitely seems worth doing. :P

Link to post
Share on other sites
  • 0

Glad to hear that it's helped you (and no problem with bumping the thread). 

 

As for the fragmentation, yes and no. Chances are, there will be less fragmentation, and when defragging it, it will move around larger chunks of data.   But it won't make defrag-ing easier really. 

Link to post
Share on other sites
  • 0

Glad to hear that it's helped you (and no problem with bumping the thread). 

 

As for the fragmentation, yes and no. Chances are, there will be less fragmentation, and when defragging it, it will move around larger chunks of data.   But it won't make defrag-ing easier really. 

 

No worries Christopher, thanks very much.

Link to post
Share on other sites
  • 0

I am also doing this now and ran across this post. 

 

Question: is there any real benefit to doing a "hard" format, or is the quick format good enough? The hard formats of my 12TB 5400 RPM drives are taking forever. I was think maybe the covecube scanner app checks them out thoroughly anyway, so I could skip the hard format.

Link to post
Share on other sites
  • 0
5 hours ago, SCB said:

I am also doing this now and ran across this post. 

Question: is there any real benefit to doing a "hard" format, or is the quick format good enough? The hard formats of my 12TB 5400 RPM drives are taking forever. I was think maybe the covecube scanner app checks them out thoroughly anyway, so I could skip the hard format.

Hello SCB, welcome to the forum. If your HDD has never been formatted, it is still recommended to do a full format on a new drive. That way the OS will check the HDD for any bad sectors and mark them not to be used. A quick format should be good enough for a HDD that was formerly fully formatted. The quick format does not check for bad sectors, so if you have a new drive with only a quick format to save time, you could end up sending data to bad sectors. Bottom line, IMHO, it is far better to use the full format on a new drive even though it may take days. It would be far worse to find corrupted files on the HDD at a later date and sometimes those errors can cause the drive to crash.

Just a friendly suggestion, it would be better to open a new question/thread on the forum for your question(s) then to tag onto an old thread that ended 6 years ago. Even if the question has been asked before, the solution could have dramatically evolved over the years. Again, welcome to the forum and hope to hear more from you in the future. 

Link to post
Share on other sites
  • 0

Thanks. I am not unfamiliar with forum decorum. However my case is exactly the same as that which is discussed above: I am reformatting drives for 64kb. So I do believe it is pertinent to the topic in this case. 

I've been using these drives for awhile. I don't know whether I hard formatted a particular drive. Some I did, some I did not, when I started using them and I made no effort to label exactly how I commissioned them.

So what I am mostly looking for here is whether or not the drivepool scanner performs the same function (marking any bad sectors) as does a hard format. Likely only Christopher can answer this (another reason I posted it here). These drives have been deep scanned many times (once per month, averaging over a year or two) by SBscanner already. Hence, my question.

Link to post
Share on other sites
  • 0
On 4/20/2021 at 4:52 AM, SCB said:

So what I am mostly looking for here is whether or not the drivepool scanner performs the same function (marking any bad sectors) as does a hard format. Likely only @Christopher (Drashna)can answer this (another reason I posted it here). These drives have been deep scanned many times (once per month, averaging over a year or two) by SBscanner already. Hence, my question.

OK. Hope you get your answer on the forum. If not, maybe a support ticket....

Link to post
Share on other sites
  • 0
14 hours ago, gtaus said:

OK. Hope you get your answer on the forum. If not, maybe a support ticket....

Thanks. Yeah nothing so far. I don't want to jab the guy with a support request over something so small. Maybe he will choose to answer here though. 

I am still doing my drives, one by one, and they each take several days: about three days to evacuate a drive via drive usage balancer, another full day to hard format it, and then another three days or so to add it back and let it re-balance everything (this overlaps with the three days to evacuate the next drive for format). So really, four days per drive with the overlap. If I could quick-format that would cut it down a day. Windows auto-update has already boned me twice though by interrupting the drive format with a reboot (you'd think it would stop itself from doing that, but alas...) so now I have that shut off too.  Anyway- I have 16 large drives here, even saving one day per would mean saving many days. As it is now, this little adventure in optimization will take me 64 days to complete.

Sorry if I sounded a bit persnickety about being accused of a thread hijack. I always thought it best for one topic to remain in one place, rather than the same question in a dozen places with a dozen answer sets. I know that drives me nuts when I'm trying to figure something out with Google-Fu. Maybe not everyone feels the same way though

Link to post
Share on other sites
  • 0

Well, it is just speculation on my part but I really doubt a full format is the same as a Scanner run. The difference being that a format will actually try to write (and report when it fails) while scanner "just" tries to read.

Link to post
Share on other sites
  • 0
On 4/22/2021 at 4:53 PM, SCB said:

Sorry if I sounded a bit persnickety about being accused of a thread hijack. I always thought it best for one topic to remain in one place, rather than the same question in a dozen places with a dozen answer sets. I know that drives me nuts when I'm trying to figure something out with Google-Fu. Maybe not everyone feels the same way though

Well, I did not take it that way, so nothing to apologize for from my perspective.

Even though the full format routine might take awhile, I think you will be better off for the effort. From my experience, data loss can sometimes take a lot longer to recover than a few days. Data corruption on a pool can sometimes go undetected for a long time, and repair might be very difficult. So I always take the time upfront for a complete full format with new drives before I add them to my DrivePool.

 

On 4/23/2021 at 12:59 AM, Umfriend said:

Well, it is just speculation on my part but I really doubt a full format is the same as a Scanner run. The difference being that a format will actually try to write (and report when it fails) while scanner "just" tries to read.

I ended up getting Hard Disk Sentinel for monitoring my drives, but, in general, I also think these monitoring programs mainly read drives and look for problems that way. Having said that, I did buy the Pro version of Hard Disk Sentinel and it has testing programs that read and write to the drive. Sometimes you are able to put a "failed" drive back into service after you run a program to "fix" the hard drive. In that case, I think it hides weak sectors from being used and resets the drive less those failed sectors. I was able to put one drive back into use after it passed that test and recovery routine, so I figure Hard Disk Sentinel already paid for itself on that one drive recovery.

Most of my drives that go bad are not able to be recovered. I guess it depends on what is going bad with the drive and I won't pretend to know more than that. If I get a report that a drive is starting to fail, I just replace it if the Hard Disk Sentinel testing routines fail the drive. 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...