Jump to content
Covecube Inc.
  • 0
fattipants2016

Scheduled balancing >24 hours apart?

Question

Looking for a hack to make this work, otherwise I hope this could be added some day.

I'd like a way to balance less frequently than 24 hours (once a week would be ideal,) but still at a scheduled time (1:00AM) to coincide with my SnapRAID sync schedule.

 

BTW, not worth an extra post, but does DrivePool balance the recycle bin!? Is there a way to make it ignore it? There might be 100's of GB in my recycle bin at any one time.

Share this post


Link to post
Share on other sites

11 answers to this question

Recommended Posts

  • 1
 

 I'd like a way to balance less frequently than 24 hours (once a week would be ideal,) but still at a scheduled time (1:00AM) to coincide with my SnapRAID sync schedule.

You could use the "balance immediately" and the "not more often than every" options.  Set it to about what you want. 
Then .... if needed, script the service to be restarted at the time you want balancing to occur.  
That .... should work, I think. But don't quote me on that. 

But here: https://stablebit.com/Admin/IssueAnalysis/27852
I've added a feature request for this

 

BTW, not worth an extra post, but does DrivePool balance the recycle bin!? Is there a way to make it ignore it? There might be 100's of GB in my recycle bin at any one time.

Yes, it would balance the recycle bin on the pool. 

 

However, depending on your pool..... balancing shouldn't happen too often.  All of the default balancers are more of "edge cases.  Sothey shouldn't cause balancing to happen, most of the time. 

Share this post


Link to post
Share on other sites
  • 0
 

BTW, not worth an extra post, but does DrivePool balance the recycle bin!? Is there a way to make it ignore it? There might be 100's of GB in my recycle bin at any one time.

Almost positive Recycle Bin is part of "Other" data, which is stuff not managed by DP. So it shouldn't get balanced.
The hidden/system $RECYCLE.BIN folder is at the root of each disk, so is outside the DrivePool folders.

Share this post


Link to post
Share on other sites
  • 0
 

script the service to be restarted at the time you want balancing to occur.

That's actually a great solution. 

Would DP be clever enough to handle the process being stopped during a write or something? It's definitely not worth breaking anything.

 

However, depending on your pool.

Yeah, it really only comes up because I have a lot of folder placement rules which make my drives unevenly balanced.

Almost wasn't even worth mentioning, but I figured i may have overlooked an option to ignore it.

Share this post


Link to post
Share on other sites
  • 0
 

Would DP be clever enough to handle the process being stopped during a write or something? It's definitely not worth breaking anything.

Yes and no.  The service and the driver are two completely separate things.  stopping the service does not unmount the pool. So the pool will continue to work just fine.   There are some issues that can/will occur if you leave the service off, but just restarting it shouldn't be a problem. 

And yes, the service should stop any balancing/duplication/what not, if it's in the middle of something.  Even if it doesn't, the service appends ".COPYTEMP" to the file name when moving/copying files, so if something "does happen", it won't cause issues. 

 

Share this post


Link to post
Share on other sites
  • 0

Related Qs

Scenario: Low powered server & download box. Want to order tasks to avoid them running at same time. e.g. Download queue empties so then execute Balance to move files and then execute SnapRAID to update.

Q1: How can I call the DP Balance function (with DP running all the time) just when the Download queue is empty (the downloader can identify and script calling once queue is empty)?

Q2: Once Balance is complete how can I then call to SnapRAID to do its stuff?

Share this post


Link to post
Share on other sites
  • 0
7 hours ago, Methanoid said:

Related Qs

Scenario: Low powered server & download box. Want to order tasks to avoid them running at same time. e.g. Download queue empties so then execute Balance to move files and then execute SnapRAID to update.

Q1: How can I call the DP Balance function (with DP running all the time) just when the Download queue is empty (the downloader can identify and script calling once queue is empty)?

Q2: Once Balance is complete how can I then call to SnapRAID to do its stuff?

DPCMD.exe (from a powershell script) would be the way to do this, except I can't find a re-balance command among it's documented list.  You'd run this command, wait for it to finish & issue a pause x command (or simply pause for x minutes if it terminates immediately), then the command(s) for SnapRAID.  Perhaps Christopher/Alex has more information on whether or not dpcmd can trigger a re-balance.

However, if you have automatic on-the-fly balancing turned on, there should be little need to constantly re-balance the pool.  In fact - I find it detrimental to do that regularly, since it actually hurts the way SnapRAID works.  The more things SnapRAID has to re-calculate parity for (every time a file is moved it has to re-calculate across all files that moved/deleted file was linked to) the more work it does, and the greater a chance it will bomb.

Point in case:  I recently upgraded a few hundred tv shows from 720p to 1080p, and the combination of deleting of the old and adding the new actually caused SnapRAID to error out and never finish.  I had to delete all .parity and .content files and have it run a completely fresh parity build.

So from that perspective, if you have a higher number of additions/removals on the pool, I'd just let DP balance real-time instead of forcing it on a schedule.  Run manual re-balances during low activity times seems like a better course of action.

Share this post


Link to post
Share on other sites
  • 0
10 hours ago, Methanoid said:

Related Qs

Scenario: Low powered server & download box. Want to order tasks to avoid them running at same time. e.g. Download queue empties so then execute Balance to move files and then execute SnapRAID to update.

Q1: How can I call the DP Balance function (with DP running all the time) just when the Download queue is empty (the downloader can identify and script calling once queue is empty)?

Q2: Once Balance is complete how can I then call to SnapRAID to do its stuff?

I only have experience with your second question. If you figure out Q1 you could probably just use it to trigger my solution to Q2.

 

1.) enable drivepool's runningfile option

2.) use bigteddy's FileSystemWatcher script (available on technet) to monitor for the removal of the runningfile you've configured, and write an event

3.) use the event-log entry you set up to trigger snapraid via a scheduled task

 

(in a nutshell, DP will create a dummy file while it's balancing and remove it when it's done. You can use the removal of this file to trigger SR)

 

Some notes:

1. ) the filesystem watcher eats up some i/o, so I still recommend you schedule it and define a max. runtime

- If you let it run all the time, it will also trigger snapraid every time DP does any routine checks, not just balancing

2.) I recommend configuring snapraid-helper (from codeplex) rather than calling snapraid from command-line

- it will check for a user-defined number of missing files prior to sync, and e-mail you so you can decide what to do

- you can also have it email you with a list of added / removed / updated / restored files after every sync if you so desire.

 

I'd never touched powershell prior to configuring the scenaro above, and now I use it for all kinds of cool stuff. It's worth giving it a go.

I made quite a few posts, here, while trying to get it working. they might be useful

Share this post


Link to post
Share on other sites
  • 0

Thanks guys for 2 helpful posts...

It probably then makes sense to real time balance and work out how to get SABnzbd (my DL client) to call SnapRAID update (via helper) a few mins after the queue is emptied. That would mean files in right places, no DLs in progress or unpacks and CPU idle so max effort for parity updates etc

Share this post


Link to post
Share on other sites
  • 0

I've certainly no idea what SABnzbd is, but I'd personally configure DP & SR to run at the same time each night, and configure your DL client to pause your download or post-processing queues for however long it typically takes.

Simple = Good

And I personally use the SSD Optimizer plugin for DP, so even if files get added to the pool while SR is running, they simply get placed on the SSD, which SR doesn't care about, until the next balancing pass.

Share this post


Link to post
Share on other sites
  • 0

@Methanoid Yeah, there is no command to trigger a balancing pass, from command line.    Unfortunately, the software really isn't designed to work that way, so what does exist has been added on slowly, and may be incomplete (eg, definitely is). 

And as has been mentioned, there is the "RunningFile" thing.  However, the file is created for any task. That's balancing, duplication, measuring, etc.  But that should be fine for your case, I think.  

And you can find details on how to do that here: http://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings#Settings.json

 

And you should definitely should be able to script the rest (though reddit or their forums may be a better place to ask about that)

 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×