Jump to content
Covecube Inc.
  • 0

Ensure maximum usable space at all time?


Question

So I'm currently testing out DrivePool before buying it and I've stumbled upon a problem. So from what I understand, DP works based on file so the largest file that can be move to the pool will be limited by the biggest free space available on a disk in the pool.

For example:

Pool with 2 Disk

Disk A: 8TB with 7TB (multiple small files) used so 1TB free

Disb B: Same as disk A

This means that the pool would show that I have 2TB left but I won't be able to copy a 2TB file into the pool. But then why won't the pool move the multi small files from say Disk A to disk B so that:

Disk A: 8TB with 6TB (multiple small files) used so 2TB free

Disb B: 8TB with 8TB used 0TB free

Now I can copy my 2TB file over.

I tried using the Ordered File Placement plugin and it runs into another problem that it prioritize like this:
 

Pool with 2 Disk

Disk A: 8TB with 7TB (multiple small files) used so 1TB free

Disb B: 8TB Free

 

Now when I copy over my 2TB file over, it said no space left because the plugin seems to prioritize filling disk A first before Disk B and won't automatically move the new 2TB file over to Disk B even though disk A doesn't have enough space left for my new file. The plugin seems to want to fill disk A first before it move on to Disk B no matter what.

Can someone recommend me a solution?

My needs are:

Having multiple HDD in a pool.

I work with a lot of big file so it should always ensure the biggest usable space is available at all time.

I don't use duplication.

Link to post
Share on other sites

13 answers to this question

Recommended Posts

  • 0
3 hours ago, gogito said:

I tried using the Ordered File Placement plugin and it runs into another problem that it prioritize like this:

Pool with 2 Disk
Disk A: 8TB with 7TB (multiple small files) used so 1TB free
Disb B: 8TB Free

Now when I copy over my 2TB file over, it said no space left because the plugin seems to prioritize filling disk A first before Disk B and won't automatically move the new 2TB file over to Disk B even though disk A doesn't have enough space left for my new file. The plugin seems to want to fill disk A first before it move on to Disk B no matter what.

Could you share your Balancing settings and balancers? This can't be normal behaviour as this would always result in an error (at some stage, a file will have to go to the next disk).

Link to post
Share on other sites
  • 0
17 hours ago, Umfriend said:

Could you share your Balancing settings and balancers? This can't be normal behaviour as this would always result in an error (at some stage, a file will have to go to the next disk).

After around 20 times trying the balancing, it finally moved my existing file over and seems to be working now. Not sure why it was bugging at first. So sadly I cannot test it anymore but it's now solve. I will report if it happens again. Thank you.

Link to post
Share on other sites
  • 0

 

On 5/23/2021 at 7:35 PM, Umfriend said:

Could you share your Balancing settings and balancers? This can't be normal behaviour as this would always result in an error (at some stage, a file will have to go to the next disk).

I ran into the problem again, trying to copy a 100GB file from my Fast Pool to Slow Pool while there was 380GB free and it says out of space, I attach some picture below, including my Balancer settings

image.png

Link to post
Share on other sites
  • 0

I actually run into the same issue (because of Chia so weird use-case). What I think would help is if we had two limiting values to set:

1. Never fill above a certain level in any case or, if filled above, move some when rebalancing; and,
2. Don't write to this drive if free space is already below X GB.

I write single files of about 100GB. I would not mind if the disks got full up to 0, but let's say 50GB. However, if there is 120GB available, I want it to write to that disk even if, ex-post, there would be less than 50GB free. It's just when there is less than 103GB free that it should not write to it.

What I do not understand is that the last operation is, as far as I can tell, a simple copy so I *think* it should be possible to know how much will be written before determining which disk to write to.

I wonder if that combined add-in written by that DP-fan caters for something like that.

Link to post
Share on other sites
  • 0
3 hours ago, Umfriend said:

I actually run into the same issue (because of Chia so weird use-case). What I think would help is if we had two limiting values to set:

1. Never fill above a certain level in any case or, if filled above, move some when rebalancing; and,
2. Don't write to this drive if free space is already below X GB.

I write single files of about 100GB. I would not mind if the disks got full up to 0, but let's say 50GB. However, if there is 120GB available, I want it to write to that disk even if, ex-post, there would be less than 50GB free. It's just when there is less than 103GB free that it should not write to it.

What I do not understand is that the last operation is, as far as I can tell, a simple copy so I *think* it should be possible to know how much will be written before determining which disk to write to.

I wonder if that combined add-in written by that DP-fan caters for something like that.

Yea, the Chia plotting operation is having other trouble so I'm not talking about it, it's the copying operation. I'm simply copying a 100GB plot file over which should be simple enough but DrivePool just doesn't want to do it.

Link to post
Share on other sites
  • 0

Ah, I see. Ordered File Placement overrides the normal "write new files to the disk with the most free space" rule. You've got Ordered File Placement set to "fill each drive until there is only 1% or 5 GB free on it" and you've still got 5.25 GB free on the first disk (and 5.44 GB on the next).

So it's going to try to write files to the first disk until it's got less than 5 GB free. And since the file you're trying to write is 100 GB, it can't fit so DrivePool can't hit the limit  that will let it switch to the next drive (which in this case wouln't be able to fit it either).

Note that due to the way Windows (or any OS that allows files of pre-undefined size to be written) works, DrivePool can't know the size of any file before it's first (fully) written to the pool.

First option: if you know the incoming files will always be 100 GB in size then set the plugin's "Or this much free space" to be 101 GB (or whatever figure will end up such that DrivePool can fill the disk past the limit without running out of space on that disk altogether.

Second option: trying the SSD Optimizer plugin instead? It also supports ordered placement, so you could assign a fast disk that has enough space as an "SSD" to act as a buffer; set a very high fill limit on the SSD and archive drives in the plugin but then set the Automatic balancing trigger size (in the Settings tab) to be low enough, and either un-tick the "Not more often than every" option or make the time period low enough, that it can prevent the SSD from filling up.

That way DrivePool can see the size of the file in the SSD and thus be able to figure out whether it's got room on the first (archive) drive or needs to try the next (and so on).

So it (should) go: program -> writing files into "SSD" disk -> DrivePool can now see the finished sizes of each file and move them into the "archive" disks in order without getting stuck. Just make sure the "SSD" is fast enough that it can handle being the buffer.

Link to post
Share on other sites
  • 0

OK, I thought that if I set the limit to, say, 120GB, and I had 130GB free, it would not write to the disk as it would end up with less than 120GB free. I think, now, that that is not how it works: it will write it. However, once it is done, the resulting 10GB free will, I think, cause it to then offload/rebalance, no?

I like the "SSD" idea, will set that up sometime.

Link to post
Share on other sites
  • 0

I haven't tested whether it will rebalance in that way; hopefully it doesn't, as that would cause the same sort of problem as above, but if it does there is the option to set OFP to only place new files and leave existing files where they are.

Link to post
Share on other sites
  • 0

Yeah, it think I remember that yesterday I had the limit at 105GB, it wrote to it with 50GB left afterwards and then started to rebalance one 100GB file off that HDD. I did have the "OFP - Only control new file placement option checked. I *think* I fixed that by unchecking the "Allow balancing plug-ins to force immediate balancing." switch. But I feat that it will be a structural issue for this specific use-case. Either you set the limit to low and you'll get out-of-space issues or you set it high enough for one file to fit but the last file will then be offloaded if it ever start to balance.

Link to post
Share on other sites
  • 0
57 minutes ago, Shane said:

Note that due to the way Windows (or any OS that allows files of pre-undefined size to be written) works, DrivePool can't know the size of any file before it's first (fully) written to the pool.

Can you elaborate more on this? The file I'm trying to write is a static file with a pre defined size of 100GB. Or am I misunderstanding something?

Link to post
Share on other sites
  • 0

The OS writes files by going (approximately) "create/open entry for file on drive's index, stream data to file from program, write details (e.g. blocks used so far) to drive's index, repeat previous two steps until program says it's done or the program says it's encountered an error or the drive runs out of room or insert-other-condition-here, write final details in the index and close entry for file". Or in even simpler terms: at the system level all files are written one block at a time, no matter how many blocks they'll eventually involve.

Now a workaround for programs that deal with fixed file sizes is to ask the OS in advance "how much free space is on drive X" so that they can know whether there's going to be room before they start writing (well, "know" as in "guess" because other programs might also write to the drive and then it becomes a competition).

But the catch there is that when the OS in turn asks the drive pool "how much free space do you have", DrivePool reports its total free space rather than the free space of any particular physical drive making up the pool. This is because it can't know why it's being asked how much free space it has (DP: "am I being asked because a user wants to know or because a program wants to write one big file or because a program wants to write multiple small files or because some other reason, and oh also if I've got any placement rules those might affect my answer too?" OS: "I don't know").

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...