Jump to content
Covecube Inc.
  • 0
The_Saver

150 GB database on system drive to be split with pool SSD's

Question

Hello everyone,

I have a 150 GB database on the system drive (C:\Users\Admin\AppData\Local) that I want to be split with pool SSD's.  Note that I cannot move the location of the database, so I was thinking about using a hardlink?

The database goes through heavy operations every day which can really slow down the other services running on the system due to high system drive utilisation.

I have 4 SSD's in the pool which I want to use for the database. Right now those 4 SSD's are used for incoming files to speed things up before they are moved to one of the 10 HDD's I have to store the data. So the database should not be treated as ordinary files - the database should stay on the 4 SSD's at all times.

I also don't want the database to be visible as a folder (hidden is still not okay) on the drivepool.

How would I go about setting this up?

 

I was thinking about creating a new pool and adding the 4 SSD's. However, I can only see the drivepool I already have under non-pooled drives and therefore it would add all the drives and create the hidden poolpart folder.

 

I hope this makes sense. Thank you for taking your time to read this!

Share this post


Link to post
Share on other sites

12 answers to this question

Recommended Posts

  • 0

You're sorta painting yourself into a corner by wanting to use pool drives, but not the pool, AND split the database across all four of the drives.

What I'd recommend is making a RAID 0 (RAID 0+1 if you can afford the 50% space drop) stripe out of your 4 SSDs, then deciding if you want a separate volume on them just for the DB, or if it can share space with the files you move on/off and simply be in "it's own directory".  I'd think sharing a single logical volume would be okay.

You can have files/folders outside the hidden Pool directory that sit on the drive and behave normally, and which aren't seen by Pool users.  But you can't break up that DB onto separate drives without some type of underlying span mechanism, which in this case would be RAID.

You could then mount that RAID drive to both a letter, and a folder under the "C:\Users\Admin\AppData\Local" path.  Drivepool could use the letter, and you'd still be compliant using the system path for the DB.

No matter what happens, you'll want good timely backups running, since you'll be exposed to either a 4x failure rate (RAID 0) or 2x failure rate (RAID 0+1).

Share this post


Link to post
Share on other sites
  • 0
49 minutes ago, Jaga said:

You're sorta painting yourself into a corner by wanting to use pool drives, but not the pool, AND split the database across all four of the drives.

What I'd recommend is making a RAID 0 (RAID 0+1 if you can afford the 50% space drop) stripe out of your 4 SSDs, then deciding if you want a separate volume on them just for the DB, or if it can share space with the files you move on/off and simply be in "it's own directory".  I'd think sharing a single logical volume would be okay.

You can have files/folders outside the hidden Pool directory that sit on the drive and behave normally, and which aren't seen by Pool users.  But you can't break up that DB onto separate drives without some type of underlying span mechanism, which in this case would be RAID.

You could then mount that RAID drive to both a letter, and a folder under the "C:\Users\Admin\AppData\Local" path.  Drivepool could use the letter, and you'd still be compliant using the system path for the DB.

No matter what happens, you'll want good timely backups running, since you'll be exposed to either a 4x failure rate (RAID 0) or 2x failure rate (RAID 0+1).

Thank you for your reply!

This is a lot to think about. I would very much like to avoid going down the RAID route as I ended up using this software as a software RAID

Maybe I should just stick to a simpler approach.

5 minutes ago, Christopher (Drashna) said:

First, is the database a single file?  
If so, then you can't split it.

Second, you could not use hard links, since these cannot span volumes.  You'd need to use junctions (ideally) or symblinks. 

 

 

The database files are about 1 KiB to 20.000 KiB. So there are tens of thousands of them :) (this is why I need SSD's for the database)

Right now I am testing with a folder on my drivepool where I have placement rules with only the SSD's. I put about 150 GB of random files in there so it should be a realistic test. I remember testing this a some point where it didn't work. Maybe the SSD Optimiser broke some stuff and moved the files to the HDD's.

Which would be better, junction or symlink? I don't want the files stored on the C drive, only on the pool. I was thinking about using this software to make my life easy: http://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html#download

 

But is there a solution where I have one Drivepool for everything and then underneath that I create one for the database and one for all the other stuff I already have? (It is imperative that the 4 SSD's will store the database and take care of all incoming files for speed).

Let me know if I am not making sense, and thanks for your help! :D 

Share this post


Link to post
Share on other sites
  • 0

I guess if you wanted to keep it simple, you could just dedicate one of the four SSDs to the database, leaving the other three for Pool use.  You won't get multi-disk-IO speeds, but you'll still get the raw IOPS speed of a single SSD.

Normally where data stores are considered I wouldn't even think of suggesting RAID due to failure rates on rebuilds and so on, but you're not dealing with RAID 5/6/etc or large storage drives scenario with the SSDs, so it would work well here if you wanted to use it.  Plus with RAID 0+1 or 1 you get the benefit of double-read speeds and redundancy.  But I can respect it if you want to go the simpler route.  Just be prepared to keep multiple daily snapshots of the data as backup points.

 

20 minutes ago, The_Saver said:

Which would be better, junction or symlink? I don't want the files stored on the C drive, only on the pool. I was thinking about using this software to make my life easy: http://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html#download

Wait... I thought you didn't want the DB files on the Pool at all.  :wacko:

 

Share this post


Link to post
Share on other sites
  • 0
1 hour ago, The_Saver said:

But is there a solution where I have one Drivepool for everything and then underneath that I create one for the database and one for all the other stuff I already have? (It is imperative that the 4 SSD's will store the database and take care of all incoming files for speed).

Let me know if I am not making sense, and thanks for your help! :D 

Yes, that's called Hierarchical Pooling, which DrivePool supports.  The problem there, is that when you add the DB child pool to the master pool, all it's files become visible there to anyone with access to that master pool.  I was under the impression from your first post you wanted to keep the DB files completely out of any pool at all.

 

Perhaps re-defining your goals and giving a little architecture detail would help us to help you:  You have a DB with a lot of files ranging in sizes, that you want the 4 SSDs to support in a pool-style fashion (aka software RAID), but which you don't want people to see.  You also have a regular Pool of disks that hold your main non-DB style data.  And, you're using the 4 SSDs as a front-end cache - are they setup to utilize the DrivePool SSD Optimizer plugin?

Are you against having completely separate pools - one for the DB, and one for your main data store?  If not you can accomplish what you need rather easily.  Partition each of the four SSDs into two logical volumes:  the first part holds 1/4 of the DB, the second is used to cache the main data pool.  You'd make a Pool for the DB by combining all the 1/4 volumes together.  You could utilize the second volume slices on all four as your SSD Optimizer front-end.

 

Share this post


Link to post
Share on other sites
  • 0

Sorry if I wasn't clear before.

What I mean is that I don't want the database files in the same pool as the other files I have :)

It sounds fine making a new pool as long as it will not be visible for the other people who have access to the current pool.

So I can have a master pool with the database and then inside it have the pool I have now.

 

I did some testing where I created a folder with random files and set file placement rules to only be the SSD's. But it did not work - the files were moved to the HDD's. How do I set this up properly? This is important as this is what I would have to do if I have a master pool and a pool inside it.

 

How would I proceed to have the best possible outcome? I was thinking to do the following:

1) Figure out why the file placement rules do not work and fix it

2) Create a new master pool

3) Inside the master pool, set up a folder for the Database with a rule saying that only the SSD's can contain the data

4) Move the existing pool inside the master pool.

4a) Making sure that files for the existing pool (which is now inside the master pool) continues the same was as they currently do with new files still being placed on the same SSD's as the Database before being moved out onto the disks for improved performance

 

If you need more information from me, please let me know.

Thank you for the assistance! :)

Share this post


Link to post
Share on other sites
  • 0
23 hours ago, The_Saver said:

The database files are about 1 KiB to 20.000 KiB. So there are tens of thousands of them :) (this is why I need SSD's for the database)

 Right now I am testing with a folder on my drivepool where I have placement rules with only the SSD's. I put about 150 GB of random files in there so it should be a realistic test. 

That should work, then.  Just wanted to make sure.

And yes, that would definitely need an SSD (and maybe even an NVMe based SSD) for that. 

23 hours ago, The_Saver said:

 I remember testing this a some point where it didn't work. Maybe the SSD Optimiser broke some stuff and moved the files to the HDD's.

If you want to use file placement rules AND the SSD Optimizer, there are some settings that you MUST change. 

Quote

* If you've created file placement rules that are attempting to keep files on drives that are designated as SSDs,
  then you should disable the "Unless a drive is being emptied" option, under the "File placement settings" category, 
  on the "General" tab. Otherwise, your File Placement rules will not be respected by this plugin (because it is 
  emptying the SSD drives).

 

23 hours ago, The_Saver said:

 Which would be better, junction or symlink? I don't want the files stored on the C drive, only on the pool. I was thinking about using this software to make my life easy: http://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html#download

Junctions.  If you have the option, always junctions.  Symlinks can be handled oddly in some cases.

And yeah, that software is nice :)

23 hours ago, The_Saver said:

But is there a solution where I have one Drivepool for everything and then underneath that I create one for the database and one for all the other stuff I already have? (It is imperative that the 4 SSD's will store the database and take care of all incoming files for speed).

You can create multiple pools, actually.  That may be the best for what you want. 

But keep in mind that each disk can only be part of a single pool. 

 

Share this post


Link to post
Share on other sites
  • 0
5 hours ago, Christopher (Drashna) said:

But keep in mind that each disk can only be part of a single pool. 

Logical disk, right?  It could simply be a formatted partition the way I understood it.  It would make spreading multiple pools out over the same set of drives way easier.

Edit:  Just tested this myself, it'll take logical volumes (partitions), so that you could structure multiple pools across the same set of physical drives.  B)

Share this post


Link to post
Share on other sites
  • 0
23 hours ago, Christopher (Drashna) said:

That should work, then.  Just wanted to make sure.

And yes, that would definitely need an SSD (and maybe even an NVMe based SSD) for that. 

If you want to use file placement rules AND the SSD Optimizer, there are some settings that you MUST change. 

 

Junctions.  If you have the option, always junctions.  Symlinks can be handled oddly in some cases.

And yeah, that software is nice :)

You can create multiple pools, actually.  That may be the best for what you want. 

But keep in mind that each disk can only be part of a single pool. 

 

I will make junctions instead of symlinks! Question 1) What is the difference?

I was thinking that I will go with multiple pools which you also say might be the best for me. I hope that it is alright with me asking all these questions. How would be the best possible way to create a new pool that contains the database and have a pool inside that with all my files on my current pool?

I was thinking of doing the following (Pool A = my existing pool)(Pool B = the new pool I will create residing inside Pool A):

1) Shut down all services using Pool A.

2) Create a new pool called Pool B with Pool A as its only non-pooled drive. (This means that all drives will be attached to Pool A. Pool B will use Pool A for storing files and thus creating the hidden poolpart directory)

3) Drag and drop all the folders from Pool A to Pool B.

4) Move the database to a folder on Pool A with file placement rules so only SSD's can store that data.

5) Restart server and start up all services again and praying that it works.

 

Question 2) Would this work? Did I miss anything?

Question 3) I am worried that in Step 3 it moves the data by reading and writing all the data instead of doing a regular move command. This would wear down my HDD's and take a very long time. Do you know what will happen?

Question 4) Will new files placed on Pool B first be stored on the SSD's and then moved to the HDD's afterwards?

 

Thank you for helping me!

Share this post


Link to post
Share on other sites
  • 0
On 8/7/2018 at 2:41 PM, Jaga said:

Logical disk, right?  It could simply be a formatted partition the way I understood it.  It would make spreading multiple pools out over the same set of drives way easier.

 Edit:  Just tested this myself, it'll take logical volumes (partitions), so that you could structure multiple pools across the same set of physical drives.  B)

Yup.  :)

On 8/8/2018 at 12:08 PM, The_Saver said:

I will make junctions instead of symlinks! Question 1) What is the difference?

How they work, and how Windows handles them, ESPECIALLY over a network.  

On 8/8/2018 at 12:08 PM, The_Saver said:

Question 2) Would this work? Did I miss anything?

That looks right.

On 8/8/2018 at 12:08 PM, The_Saver said:

 Question 3) I am worried that in Step 3 it moves the data by reading and writing all the data instead of doing a regular move command. This would wear down my HDD's and take a very long time. Do you know what will happen?

That depends on how you move the data, but yeah, this is likely what will happen.  If you move data from volume to volume, this is what happens.  You only get the "smart move" if the data is on the same volume. 

On 8/8/2018 at 12:08 PM, The_Saver said:

 Question 4) Will new files placed on Pool B first be stored on the SSD's and then moved to the HDD's afterwards?

If the SSD Optimizer is enabled for that pool, yes.

Share this post


Link to post
Share on other sites
  • 0
10 minutes ago, Christopher (Drashna) said:

If the SSD Optimizer is enabled for that pool, yes.

Thank you for your reply, it answers all my questions. I just want to be sure I understood this corretly:

 

Pool A will have ALL my drives "attached" to it. It uses SSD Optimiser to put new files on SSD's first.

When creating Pool B, the only drive Pool B can use for data is Pool A. In other words, 4 SSD's and 11 HDD's will be attached to Pool A. And Pool A will be attached to Pool B.

Will a new file placed on Pool B first be written to an SSD's before being to put to an HDD?

 

I hope it makes sense and thank you!

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...