Jump to content
  • 0

DrivePool shows 1TB of extra Other files


cryodream

Question

As the title says, Drivepool shows ~1TB too much of Other files. I have files on only 1 drive outside of the pool. The size of those files is ~1.5 TB. But Drivepool shows there is 2.5TB of Other data. And I can see Other data on pretty much every disk in the DP UI, which varies from ~200MB to ~250GB.

 

I've read FAQ - Unduplicated vs Duplicated vs Other vs Unusable - Nuts & Bolts - Covecube Inc., and I understand there should be some additional Other data, but a Terabyte? This feels way too much.

 

Is this normal or is this a bug?

 

Windows 8.1

Drivepool 2.2.0.651 beta

Pool size: 76.4TB, Unduplicated: 60.2TB, Duplicated: 2.94TB, Other 2.58TB

 

Link to comment
Share on other sites

11 answers to this question

Recommended Posts

  • 0

Remeasure the pool.  (under Pool options, in the UI).

 

If the issue persists, use something like "WinDirStat", and analyze one of the pooled disks (not the pool itself). And files outside of the hidden "PoolPart" folder are considered "other" data by the pool.

 

 

Another thing to check is the shadowcopy storage. If you have a lot of snapshots, it could cause this. Also, if you're using data deduplication, it could also cause this. 

Link to comment
Share on other sites

  • 0

Thanks for answers, Christopher.

 

Nope, remeasuring does not help. Also I do not use any shadowcopy, snapshots, deduplication or somesuch.

 

Checked with windirstat and here's an example for one of the drives:

there should be pretty much NO other data, but instead there is 284GB, which is a huge amount, huh...

 

kRUNvgy.png

 

EJ1sB12.png

Link to comment
Share on other sites

  • 0

Could you open up the disk itself, right click on the "PoolPart.xxxxx" folder, and select properties.  let it sit for a long while (so it stops counting up). Check what it says is the Size, and the "Size on disk" values.  See if these match. If they don't, I'm guesing that the discrepancy will be ~284GBs.

 

If they do match, then let me know.

Link to comment
Share on other sites

  • 0

I reshuffled the drives in my backplanes, and after restarting the server and after DrivePool remeasured the pool the "other" size has decreased from considerably: from 2.58TB to 1.92TB. This is still too much though, it should be something like 1.7TB. Why the size decreased - no idea, I tried remeasuring like 5 times before and it did not change. Now it did, though still to big. I hate not knowing what the hell is going on :)

 

Ok, here's the screens of PoolPart on one of the disks, that should not have any other data, but has 51.6 GB.

 

6W9qpPx.png   LQAnWms.png

 

 

As you can see, it's clearly not 51.6 GB difference :)

 

aOa51Q3.png

 

o0GElzr.png

 

 

It would be very helpful if DrivePool had a view, where you could clearly see a list of files on the drive and the type of the files, like: unduplicated, duplicated, other...

Maybe a feature for the future? :)

 

Seriously though, what are those hundreds of GBs, that DP considers as Other? I mean, these sizes are not trivial...

Link to comment
Share on other sites

  • 0

For auditing the pool.... check out the end of this post:

http://community.covecube.com/index.php?/topic/1587-check-pool-fileparts/

 

(just added this week, actually). 

 

 

To be honest, 50GBs sounds more reasonable here. 

http://community.covecube.com/index.php?/topic/37-faq-unduplicated-vs-duplicated-vs-other-vs-unusable/?p=3635

 

 

However, it sounds like you definitely still have waay too much "Other" space on the other drives. 

 

 

Specifically, the measuring is done in the kernel driver, meaning at a very low level. If it's coming back with inconsistencies, there may be disk errors.

 

But please try enabling file system logging and remeasure the pool. Once it finishes (or finishes with a "problem drive", end the logging and upload the logs to us. 

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

 

Also, it may be worth checking the Event Viewer ("eventvwr.msc") logs (specifically, in the Windows Logs, the "System" section) to see if you're experiencing any disk, ntfs or controller related errors. If so, that could be the cause here.

Link to comment
Share on other sites

  • 0

Christopher, to be honest, I almost got scared another drive(s) dying... But then I was re-reading one of my older questions and found this:

 

 

And to clarify, there should be no issue with pointing uTorrent to the Pool directly. For locked/option files.... the real time duplication feature writes all files/modifications in parallel, so there shouldn't be an issue with downloading to or seeding from the pool. Though, any locked file may show up as "other" until the file is unlocked.

 

 

And of course, I had utorrent seeding 200+ torrents. I closed utorrent, added some more drives to the pool (by seeding the pool), and after the pool was up and running again - only ~12 GB of Other data.  :rolleyes:

 

So, just to confirm - utorrent locks the files adn DP sees them as Other?

 

I was thinking of seeding from the pool. Won't that be a constant problem, if the files are actualy Duplicated, but DP sees them as Other half of the time? Is that gonna be a constant needless balancing and reduplication/deduplication hell?

Link to comment
Share on other sites

  • 0

Aaah! Okay, that would definitely do it.  And I'm sorry for missing that.

 

Specifically, the way that utorrent locks the file is the way that you are supposed to lock open files (eg, for writing).  THis prevents other programs from modifying the contents or messing with them, so that you avoid corruption.  The issue is that this does cause some issues with StableBit DrivePool.  There isn't a good way to handle this because of what is going on, so we leave it as "other" until we can be sure about the file size and other details.

 

 

 

As for seeding them, I think that once you've stopped the torrents and then restart them to seed them, it may fix the issue here.  If it doesn't, it shouldn't affect duplication, but it may affect balancing. Specifically, "other" data is ignored when calculating the balance ratio and determining if data needs to be moved around.  

But for the most part, I suspect that should be fine.

If you do have issues with it, let us know.

Link to comment
Share on other sites

  • 0

Aaah! Okay, that would definitely do it.  And I'm sorry for missing that.

 

Yep, Chris from the day before yesterday is better than Chris from yesterday, Such a blunder  :rolleyes: Scared me half to death for no good reason. You should be ashamed of yourself...  :D

 

But seriously, thanks for confirming about ut behavior.

 

So, I should remember to stop the torrent, after it finished downloading and then resume it for seeding. Hoping, that ut would release the write-lock on the files. I gather ut should not re-lock the files with write-lock, just for seeding, but we'll see, I guess.

 

Thanks again for all your answers, Chris.

Link to comment
Share on other sites

  • 0

Yep, probably gonna need to use dedicated disks for utorrent. Arrgh, I added all the spinners to the pool, just yesterday :) I love having only SSDs and one huge pool of spinners in "This PC" :)

 

Can I "un-seed" the drives from the pool? I mean I stop the service, move the files out of the pool into the drive's root (these files will not have duplication, of course) . Restart the service, re-measure, and then remove the drive from the pool. Which should be quick, as there's no files to move on removing the drive?

 

Pitty though, I used to seed TBs upon TBs. Now if I have duplication, and still need another dedicated copy for ut seeding... f*** that.

 

Anyway, now that you told me that there's such a thing like dpcmd.exe, I'm thinking, maybe I could write a windows service or a just a simple console app and use task scheduler to manage the duplication for me?

 

- Scan the utorrent drives and hash-identify all seeding files.

- Scan selected duplicated folders on the pool (like movies, tvshows, etc) and hash-identify the files.

- Compare and find duplicated folders on the pool, that have all the files inside them with a 3rd copy on ut drives.

- Unset duplication on those folders.

- Save space. There is most definitely NO need for 3 copies of freakin movies and tv shows. Even 2x duplication is killing my spirit, when looking at all the space I'm loosing :)

 

Would that be doable with dpcmd.exe?

 

Would that create any sort of problems or performance hit on Drivepool? I mean, micro-managing the duplication this way, would create hundreds or maybe even thousands of individual duplication settings on folders...

 

The positives would be:

A separate copy of files for ut on dedicated drives.

No need for tons of folder junctions, which I used before to point to the files in my library.

Could seed even those torrents where people have f***ed up naming the files. There are always those "special" ones, that simply cannot comform to the naming rules, and need to be "different". Jeez, thats annoying.

Could once again use hard links with utorrent and cross-seed those torrents, that have files renamed on one tracker but not on the other.

 

Phleeease, tell me that's possible :)

 

Oh, and btw, it would be nice, to have commands in dpcmd.exe to force re-measuring, re-balancing and duplication run. So it could be automated properly.

Link to comment
Share on other sites

  • 0

Yeah, it's is really nice. ;)

 

 

As for the "unseeding", absolutely. Just move the folder out of the hidden PoolPart folder.  However, since this may be spread out between multiple disks... it may be more complicated than that, and may require moving files around.

However, I would recommend stopping the "StableBit DrivePool Service" while you're doing this to make sure it doesn't move files around. And after you're done, you'll want to remeasure the pool, to make sure it's up to date.

 

As for dpcmd, you could absolutely run a batch/script file and call it. 

There is a small list of what the utility does here:

http://community.covecube.com/index.php?/topic/1587-check-pool-fileparts/

And you can just run it and get a list of the commands and help for the various commands, as well. 

 

 

Though, I'm not entirely sure what you want to do here... but the utility includes the option to force a consistency check basically. That sounds like it's what you want here. This will cause the pool to run a duplication pass, if it finds any issues. 

 

 

As for saving space .... I have my entire Video collection duplicated. That's ~35TB of content. Yes, I am losing out on a lot of capacity by doing so. But .... what happens when you lose a large chunk of the files? Rebuilding it isn't easy, and is very time consuming. I've been there, I've done that. And I'd rather pay for the additional storage than the stress of having to rebuild my collection.  Though, I do understand that not everyone feels the same. 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...