Jump to content
  • 0

Folder/File balancing, where destination is drive not part of the pool?


postcd

Question

Hello,

i am not user of the DrivePool yet.

When i create pool from several HDDs of various sizes, can i set the rule where all *.db files that are going to be saved a new or modified will be placed not on any drive that is part of the pool but on to my operating system (Windows 10) system drive which i bet i would be unable to make it part of the pool unless i create for example VeraCrypt encrypted container on it and after mounting this container it will appear as a new drive which i will be able to include into the pool and set it as a destionation for the .db File balancing feature?

Update: so far, from the comments, it looks like the DrivePool will not save .db files to the drive outside of the pool.

Link to comment
Share on other sites

15 answers to this question

Recommended Posts

  • 1

I don't think that is correct. It seems to me that you can cause DP to place all files that conform to *.db to a single HDD/SSD (that is part of the Pool though). This can be done through the File Placement options.

https://stablebit.com/Support/DrivePool/2.X/Manual?Section=File Placement

And that would allow for the use of an SSD as part of the Pool dedicated to *.db files only.

It is a bit complex to set up (see e.g. 

I can't say I would look forward to setting this up but it is intended to support what you want I think. Of course, if you have duplication then it is a bit more complex as you need two SSDs for this to work, unless the .db files only need to be read fast and writes can be "slow".

I am curious though as to what program you are using that has these *.db files.

Link to comment
Share on other sites

  • 0

The short answer is - not in the way you're describing within DP itself.

 

Now, the fundamental problem with what you're suggesting is that *.db is not a unique file type - so a load of Windows system files & odd drivers & bits of Adobe & many other pieces of s/w also create & modify them over time - & those need to be in specific folders...

...so if you randomly tried to move all of the new & modified db files to somewhere else then it would break loads of stuff.

 

Anyway, because the group of drives in the DP will need to be assigned a drive letter - let's say D, with the system drive being C... ...then if you want something saving onto the pool you'll save it to, for example, D:\DBFiles

...& if you want it on the system drive then C:\DBFiles

So this is something that you'll have to change within the 3rd party s/w you're using that's creating the db files - if it's a sensible thing to split those db files from other data of course.

 

Alternatively, given that I'm guessing you're looking at the system drive being a SSD & you want the db files on it for faster read speeds - I suppose you could create an additional partition on the SSD & add that to the pool - again D.

With a D:\DB Files folder, you could tell DP to limit the placement of that folder to the SSD partition (& also tell it not to store anything else on that partition) - but again you'd obviously have to direct the s/w creating the db files to save them in that folder - as opposed to C:\DBFiles

However, unless you also wanted to add (real time) duplication - so you've got the db files stored on both the SSD (for read speeds) AND a HDD (for duplication)…

(in this situation you'd limit the placement of everything in the D:\DBFiles to both the SSD partition & one of the HDDs - & add 2x duplication)

 ...this would seem pretty pointless; & partitioning the SSD to do this would have the major downside of limiting the flexibility of usage of the SSD's capacity.

Edited by PocketDemon
This is incorrect.
Link to comment
Share on other sites

  • 0

So, i can not make that DrivePool relocating feature save some data on a drive that is not part of the pool?

 

Quote

load of Windows system files & odd drivers & bits of Adobe & many other pieces of s/w also create & modify them over time - & those need to be in specific folders

this drive pool virtual drive (or how to call the drive that is result of DrivePool pool) would not be a system drive, it is rather data drive for videos, images and a few apps (not Program Files) (one is using files with .db extension). So i do not think it shoudl cause some conflict, unless DrivePool does not have this feature to save some data outside of the pooled drives or fails to properly create hardlink(?) from original location to new one? Or you mean to say that DrivePool's relocating feature would places all .db files no matter in which directory they were located into one common directory without using subdirectories?

Quote

I suppose you could create an additional partition on the SSD & add that to the pool - again D.

Yes, this seem to be good option, but currently my system drive is fully encrypted using Truecrypt and main bootable partition can not be shrinked from within Windows Drive manager. So i guess option is to create encrypted VeraCrypt/TrueCrypt container (file) on my system drive and mount this file using VeraCrypt/Truecrypt as a new drive.

Quote

you'd obviously have to direct the s/w creating the db files to save them in that folder

if i understand right, i think i can not do this, because that SW which is making .db files is maintaining some structure and i can not tell it not to do it, as i am not a programmer:

data/folder1/subfolder/randomfilename.db
data/folder2/subfolder/randomfilename.db
..
(all folder and file names except "data" are randomly created and i can not guess its name)
.db files is like maybe 100-5000 and total its size is maybe 30 GB atm, though can be growing.

Moreover that software is using very large amount of data not fitting todays SSD's capacity, though .db files are of relatively small size as mentioned.

Link to comment
Share on other sites

  • 0

Okay, to be clear, DP cannot do this...

...& if whatever s/w you're using *has* to have the db files in the same folder as the data then there's no workaround that I can think of - as either they're all in the respective folders or the db files would be completely useless & your s/w would just create a new one in the respective folders during whatever process it's using.

Yeah, with no info given as to what program is creating these files then it's impossible to try to find an answer - so I was simply working on the premise that most s/w allows you to alter the standard directories for specific things...

...however I'd suggest that you either look in the documentation &/or ask on a forum dedicated to that s/w to see if it's possible to relocate all of the db files into a single alt directory; as you're then getting the answer from people who have explicit knowledge.

Edited by PocketDemon
This is incorrect.
Link to comment
Share on other sites

  • 0
1 hour ago, Umfriend said:

AFAIK, there is no reason why programs could or should not be located on a DrivePool so perhaps you could relocate all to DP?

I can't speak for whatever s/w the OP is using, but taking, for example, most of Adobe, then forcing it to place the temp files/scratch disk/media cache on a decent SSD can make a significant difference; irrespective (within reason of course) of where the main data files are stored...

…&, naturally, that SSD doesn't need to be the system drive; which would be particularly relevant if budget limited the capacity of the SSDs you could afford to buy.

So, for example, with <=1080p editing in Premiere, I've personally seen no benefit in using anything better than using short-stroked 4 drive R10 HDD arrays for the main video & audio files for a project - but that certainly isn't true for all of the other files.

 

Then, as another example, with 16 threaded batch audio lossless compression/decompression (& sundry tasks), there was no speed difference whatsoever between using 2 reasonable SATA 250GB SSDs in R0 (repurposed 830 Samsungs that I'd bought when they were the new thing as a system drive) - vs using a 1TB 970 Evo...

...but the R0 set up was noticeably faster than using a single SATA SSD...

...& it's significantly quicker to move in the order of 100-300GB of audio from a DP'd HDD to the R0 array, do all of the batch processing I need to, & then move everything back.

 

Now these are just 2 examples of my experience with my main setup of course...

…but it's just about illustrating that whilst you're 100% correct that there's no reason why everything couldn't be on any drive type, that's not to say that different storage options can't be more appropriate for different processes/parts of processes.

 

That said, I now have no idea 'if' the s/w they're using will show a material benefit or not - as I originally assumed that there must be a very good reason for what was being proposed - so they were working with something like massive databases that, for some reason, needed to be in a generic *.db format...

...but based on what's since been written I'm really not certain what the gain would be - which is part of the reason for suggesting looking into the specific s/w they're using.

Well, I would imagine that on whatever forums are dedicated to the s/w, people would tell the OP if it was a worthwhile proposition.

Link to comment
Share on other sites

  • 0

Thank you both, if the answer is NO (DrivePool can not selectively put certain file types from certain folder to my system drive which is not part of the pool), then i have to put another SSD drive and make it part of the pool OR i have to create the encrypted container on my system SSD drive (which is not part of the pool) to create virtual drive which i make part of the pool.

If the DP will relocate these .db files, for example:

data/folder1/subfolder_def/randomfilename.db
data/folder2/subfolder_abc/randomfilename.db
data/folder2/subfolder_abc/nextfolder/randomfilename.db

i hope it will also duplicate them to other drive (i will always want any data of the pool be duplicated to one more physical drive).

> @Umfriend perhaps you could relocate all to DP?

No, because that data/ subfolders (in various depths) contains also very large media data and that is currently not suitable to store terabytes of data on a nowadays SSDs. I want to relocate .db files of that data/ folder subfolders because the app (Zer0net) has built in search functionality which is searching thru these .db and i want it to be fast (-> SSD), rest data are OK to be stored on the HDDs.

Link to comment
Share on other sites

  • 0

I thought I'd made myself clear, but DP also cannot put selected file types within a pool on selected disks.

The ONLY thing that you can do is to tell it to put a folder on 1 (or more) drive(s) - which it will carry out until there's not enough space on the drive(s).

So - D:\DBfiles\[all of the *.db files] - could be on a specific drive (or drives).

 

But with a structure akin to -

D:\MediaFiles\Media File0000001\[bunch of files, inc a *.db file]

D:\MediaFiles\Media File0000002\[bunch of files, inc a *.db file]...

...D:\MediaFiles\Media File9999999\[bunch of files, inc a *.db file]

- then EVERYTHING in the D:\MediaFiles\ folder hierarchy would follow the same drive limitations.

 

So, again, you would need to look at the documentation for or discuss it on a forum about Zer0net to see if it's feasible to do the former... ...as there is no solution within DP that will move ONLY the *.db files; UNLESS you can set it up to put them in a separate folder (or folders) from the media files.

 

Otherwise, the only other option I can think of that 'might' work for what you're doing is look at SSD caching s/w to see if any of that can meet your needs.

So something where you can set up a SSD to cache the most used data from a specific HDD (or DP pool or array) 'might' work - since they would normally tend to ignore very large files - which would then to prioritise your *.db files...

...but if you could find something that was more explicitly controllable then it would obviously be better.

Link to comment
Share on other sites

  • 0

@Umfriend , program name was mentioned in the last sentence of my previous comment.

> if you have duplication then it is a bit more complex as you need two SSDs for this to work, unless the .db files only need to be read fast and writes can be "slow"

Yes, the duplication of the .db files can be handy to 1 more drive, but not necessary i think. I would expect DP not to delay write to a "primary" destination just because "replication" drive is slower..

Link to comment
Share on other sites

  • 0
30 minutes ago, Umfriend said:

I don't think that is correct. It seems to me that you can cause DP to place all files that conform to *.db to a single HDD/SSD (that is part of the Pool though). This can be done through the File Placement options.

https://stablebit.com/Support/DrivePool/2.X/Manual?Section=File Placement

And that would allow for the use of an SSD as part of the Pool dedicated to *.db files only.

It is a bit complex to set up (see e.g. 

I can't say I would look forward to setting this up but it is intended to support what you want I think. Of course, if you have duplication then it is a bit more complex as you need two SSDs for this to work, unless the .db files only need to be read fast and writes can be "slow".

I am curious though as to what program you are using that has these *.db files.

Thanks for finding this & proving me wrong - as, along with give the OP proper advice, it's honestly really useful to learn something new.

Yeah, I can't think of an application for it for my personal use, but it's much better to be aware of useless (to me) capabilities than believe a load of nonsense.

Link to comment
Share on other sites

  • 0
1 hour ago, postcd said:

Yes, the duplication of the .db files can be handy to 1 more drive, but not necessary i think. I would expect DP not to delay write to a "primary" destination just because "replication" drive is slower..

I am not certain but to the extent a program postpones execution until a write is completed, I would expect it to receive the write-finished confirmation only when all duplicates are written. One thing to remember is that in DP, there is no primary/secondary idea. All duplicates are equal.

 

1 hour ago, PocketDemon said:

I'm truly sorry, as it clearly can be done.

Well, I think DP is intended to support this but it may not be that straightforward to get to work easily.

Link to comment
Share on other sites

  • 0
5 hours ago, Umfriend said:

I am not certain but to the extent a program postpones execution until a write is completed, I would expect it to receive the write-finished confirmation only when all duplicates are written. One thing to remember is that in DP, there is no primary/secondary idea. All duplicates are equal.

 

Well, I think DP is intended to support this but it may not be that straightforward to get to work easily.

Having just had a quick test, in terms of setting up a rule then it's dead simple, & you then get the same options for each rule as you do for directories in terms of drive usage & %ages.

However, having checked, what you don't get is anything additional in the duplication options - so each file clearly still inherits the duplication from its parent directory.

So that means that with 2x duplication on the parent folders you'd have to set limit the rule to using SSD + one other drive to get the read boost on all of the db files...

(& then disable the SSD as a storage option for all of the directories)

...I believe we know for certain from other threads the DP will prioritise reads from the fastest drive(s) with Read Striping enabled.

Link to comment
Share on other sites

  • 0

If there is x2 duplication then you need two SSDs for tj SSD optimizer/write cache. I would advise against using an OS drive for data. But certainly you could do what you propose with 2 SSDs. It can even be done with one if you use hierarchical Pools but I would advise against that too.

What I do not know is whether you can use two SSD as cache _and_ designate the same as targets for, say, *.db files.

Chris would know i guess.

 

Link to comment
Share on other sites

  • 0
17 minutes ago, Umfriend said:

If there is x2 duplication then you need two SSDs for tj SSD optimizer/write cache. I would advise against using an OS drive for data. But certainly you could do what you propose with 2 SSDs. It can even be done with one if you use hierarchical Pools but I would advise against that too.

What I do not know is whether you can use two SSD as cache _and_ designate the same as targets for, say, *.db files.

Chris would know i guess.

 

I'd deleted part of what I'd written... but obviously not quickly enough.

 

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...