Jump to content
  • 0

Files are missing from pool, but they are on the drives in the correct place (...\PoolPart...\...etc)


cryodream

Question

Yesterday my gf asked me to get her a movie, so I downloaded it for her. Today she wanted to watch it, and asked why it's not there, did I forgot? I said, of course no, I did, it's there. She says no... I say, fuck yes, it is. She says no. I'm like, give me that...(the remote). I look through her Kodi library and... WTF? It's not there. The problem is, I KNOW I have it, I fucking got it yesterday. I use "Everything" to look for it, and there it is. I'm like WTF? The problem, that "Everything" does not work on the DrivePool pool, only on the mounted drives themselves. I go into the pool and what do you know - its not fucking there!
 
So, the movie folder with the files are missing from the pool, but they actually are on one of the drives in the correct place. I have the pooled drives also mounted into folders, though I never use them, it actually helped this time to find out that's something is messed up. Here's a list of files for this movie:

D:\Drives\b3s05\PoolPart.eb4558cc-3700-4add-95b0-09e7f9286bef\Movies\Good\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89\clearart.png
D:\Drives\b3s05\PoolPart.eb4558cc-3700-4add-95b0-09e7f9286bef\Movies\Good\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89\logo.png
D:\Drives\b3s05\PoolPart.eb4558cc-3700-4add-95b0-09e7f9286bef\Movies\Good\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89.mkv
D:\Drives\b3s05\PoolPart.eb4558cc-3700-4add-95b0-09e7f9286bef\Movies\Good\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89.nfo
D:\Drives\b3s05\PoolPart.eb4558cc-3700-4add-95b0-09e7f9286bef\Movies\Good\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89-fanart.jpg
D:\Drives\b3s05\PoolPart.eb4558cc-3700-4add-95b0-09e7f9286bef\Movies\Good\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89-poster.jpg

This never happened to me with DrivePool before, afaik. I searched the forum and all I could find was this thread: "Missing" files - General - Covecube Inc., where the guy was having the same problem, but there was no answer with a solution.
 
Anyway, I did check the list of file system filters as per instructions here StableBit DrivePool Q2159701 - Covecube - Wiki:

C:\Windows\system32>fltmc

Filter Name                     Num Instances    Altitude    Frame
------------------------------  -------------  ------------  -----
WdFilter                               49       328010         0
luafv                                   1       135000         0
npsvctrig                               1        46000         0
FileInfo                               49        45000         0
Wof                                     0        40700         0

It seems, there are more than those 2 (luafv and FileInfo) on my server, but i googled the others and they seem legit:
WdFilter - Windows Defender Mini-Filter Driver - Windows 8 Service
npsvctrig - Named pipe service trigger provider - Windows 10 Service
Wof - Windows Overlay File System Filter Driver - Windows 8 Service
 
So, the questions:

  • How the fuck do I fix this?  I mean, re-copying or moving the files to the pool manually is not a solution I am looking for. It may help for this one missing folder, that I found by accident, but what if there are more? What if this has been happening for days, weeks or months? How many "missing/hidden" duplicated files may there be on all the drives? Please don't tell me I will need to do this manually, as even a thought of doing this with 80TB of files is giving me a headache.
  • Is this a bug? Why is this happening and how do I avoid it in the future?
  • I did the file system logging thing, as per: StableBit DrivePool 2.x Log Collection - Covecube - Wiki. My logs are 228MB. What the hell is in those logs? And from how long ago? Why the hell do you need the whole folder? Why not the most recent log, from today, when this shit happened.

 

Thanks in advance for any help.

Link to comment
Share on other sites

6 answers to this question

Recommended Posts

  • 0

First, I'm sorry to hear about the issue you are experiencing, and any trouble that it has caused. 

 

 

What OS are you using, (for the system that StableBit DrivePool is on)?

What version of StableBit DrivePool are you using?

 

Also, have you made any changes to the StableBit DrivePool advanced config setting?

 

 

  1. As for fixing the issue, well, it's a very odd issue and really shouldn't be happening.  Grabbing the logs and trying to list the folder should help us to see what is going on. 
    I'd also recommend running "chkdsk c:\Drives\b3s05 /f" just in case there are any disk errors that may be causing this issue. 

    Also, upgrading to the beta version may help out here. But it may not.
     
  2.  Definitely a bug. It should always list the full contents of all of the pooled disks. 
    Though, could you confirm that the specific disk is in the pool? (yes, I know this may be a stupid request, but just in case). 
    Also, does the folder show up (eg "[PoolDrive]:\Movies\Good\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89\")? And if it does, does it show any content, or none at all?
     
  3. Lots. It includes three main folder:
    • ​Error Reports -  These are handled errors, but may indicate specific issues. Not as relevant here, as these are not generated by the driver, which is the problem here
    • Store -  This is the local settings store, where balancing, pool info and other settings are kept. 
    • Logs - There are two folders in this folder:
      • Service - This is the text output from the background service, and likely why the files are so large.  The last few days are all that is really necessary, so you can skip the older logs, and this may cut down on a lot of size. 
      • CoveFS - kernel driver tracing (debug) logging.  This is what will likely catch the specific issue in question, and only generated when file system logging is enabled

However, grabbing all of the logs is preferred, as there may be related issues that show up in different parts of the logs. 

 

 

 

 

Also, something else to check is the Event Viewer (run "eventvwr.msc"). Specifically, check the "Windows Log" and the "System" section.  See if there are any disk, ntfs or controller (iaStor, ahci, etc) related errors.  If there is, and it's on the same disk, it may be that the disk is experiencing a problem, and causing the files to not be shown by the pool drive. 

Link to comment
Share on other sites

  • 0

Chris, as always, you're awesome, thanks for answers.

 

What OS are you using, (for the system that StableBit DrivePool is on)?

What version of StableBit DrivePool are you using?

OS: win 8.1 Pro x64

DrivePool: 2.2.0.659 Beta

Scanner: 2.5.1.3062

 

Also, have you made any changes to the StableBit DrivePool advanced config setting?

As much as I can recall, nope. Take a look:

 

<?xml version="1.0"?>
<configuration>
    <configSections>
        <sectionGroup name="applicationSettings" type="System.Configuration.ApplicationSettingsGroup, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=REMOVED">
            <section name="DrivePoolUI.MySettings" type="System.Configuration.ClientSettingsSection, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=REMOVED" requirePermission="false"/>
        </sectionGroup>
    </configSections>
    <system.diagnostics>
        <sources>
            <!-- This section defines the logging configuration for My.Application.Log -->
            <source name="DefaultSource" switchName="DefaultSwitch">
                <listeners>
                    <add name="FileLog"/>
                    <!-- Uncomment the below section to write to the Application Event Log -->
                    <!--<add name="EventLog"/>-->
                </listeners>
            </source>
        </sources>
        <switches>
            <add name="DefaultSwitch" value="Information"/>
        </switches>
        <sharedListeners>
            <add name="FileLog" type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=8.0.0.0, Culture=neutral, PublicKeyToken=REMOVED, processorArchitecture=MSIL" initializeData="FileLogWriter"/>
            <!-- Uncomment the below section and replace APPLICATION_NAME with the name of your application to write to the Application Event Log -->
            <!--<add name="EventLog" type="System.Diagnostics.EventLogTraceListener" initializeData="APPLICATION_NAME"/> -->
        </sharedListeners>
    </system.diagnostics>
<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.1"/></startup>
    <applicationSettings>
        <DrivePoolUI.MySettings>
            <setting name="UI_VerticalOnly" serializeAs="String">
                <value>False</value>
            </setting>
            <setting name="UI_HorizontalLite" serializeAs="String">
                <value>False</value>
            </setting>
            <setting name="UI_NoVersion" serializeAs="String">
                <value>False</value>
            </setting>
            <setting name="Wss_AnimationFps" serializeAs="String">
                <value>30</value>
            </setting>
            <setting name="Comm_ForceLocalTcp" serializeAs="String">
                <value>False</value>
            </setting>
            <setting name="UI_TroubleshootingMode" serializeAs="String">
                <value>False</value>
            </setting>
            <setting name="Comm_ForceLocalTcpPort" serializeAs="String">
                <value>27525</value>
            </setting>
        </DrivePoolUI.MySettings>
    </applicationSettings>
</configuration>

As for fixing the issue, well, it's a very odd issue and really shouldn't be happening.  Grabbing the logs and trying to list the folder should help us to see what is going on. 

I'd also recommend running "chkdsk c:\Drives\b3s05 /f" just in case there are any disk errors that may be causing this issue.

Where and how should I send the logs?

About chkdsk - is it for fixing the corrupted file system? Will running it "fix but also loose" any offending files?

 

Also, upgrading to the beta version may help out here. But it may not.

Already running beta. Waiting for new version of DP for ages... Presonaly, methinks Alex has buried himself with CloudDrive for good, while DrivePool very much needs some vital features and updates.

 

Definitely a bug. It should always list the full contents of all of the pooled disks. 

Though, could you confirm that the specific disk is in the pool? (yes, I know this may be a stupid request, but just in case).

Heh :) It is in the pool. But I very much understand the reason for you asking this anyway. Been there, seen that, many a time.

 

Also, does the folder show up (eg "[PoolDrive]:\Movies\Good\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89\")? And if it does, does it show any content, or none at all?

Nope. Neither folder nor any files show up in the pool. I can only find/see them if I browse through the individual drive via NTFS mountpoint.

If my GF wouldn't have wanted that movie, I most definetely would've missed this shit completely. No "hicups" while browsing the pool, no Scanner reports, that anything is wrong. File system - OK, S.M.A.R.T. - OK. Hence, I'm thinking, how long have this shit been going on?

 

However, grabbing all of the logs is preferred, as there may be related issues that show up in different parts of the logs.

K, done. Where to send?

 

Also, something else to check is the Event Viewer (run "eventvwr.msc"). Specifically, check the "Windows Log" and the "System" section.  See if there are any disk, ntfs or controller (iaStor, ahci, etc) related errors.  If there is, and it's on the same disk, it may be that the disk is experiencing a problem, and causing the files to not be shown by the pool drive.

Awesome. I totally forgot about checking the event log.

 

And I think or rather hope I found the culprit. The event log is littered with these warnings:

 
The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{32abe257-2f68-4778-8927-670457764ce3}, DeviceName: \Device\HarddiskVolume41.
(The I/O device reported an I/O error.)
The volume id 32abe257-2f68-4778-8927-670457764ce3 is for the drive itself, and HarddiskVolume41 is actually the pool. So I guess drivepool tries to write something in on that drive and the drive errors out?

Also it shows some of these:

The IO operation at logical block address 0x172df9a0 for Disk 22 (PDO name: \Device\00000051) was retried.
 

 

Questions:

  • Is this the cause of these missing files?
  • Why the hell there are no S.M.A.R.T. warnings then?
  • Should I retire this drive or do you recon, it's still safe enough to keep it online for duplicated files only? Am I stupid even thinking about keeping using it?
  • How do I go about trying to fix this drive? chkdsk and thats it? Or anything better out there? Should I clear the drive before hand?
  • This drive is used by uTorrent. I have uTorrent seeding ~450 torrents, most of them have their files on the pool. This drive is one of 3, that are dedicated for uTorrent only. I mean, I remember reading that uTorrent and DrivePool does not play nice. Maybe that's the problem?
  • Any good software to keep an eye on windows event log and report things like this automaticaly in the future?
Link to comment
Share on other sites

  • 0

Where and how should I send the logs?

About chkdsk - is it for fixing the corrupted file system? Will running it "fix but also loose" any offending files?

For the logs: http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

 

As for CHKDSK, yes, it's for fixing file system damage or corruption. As for losing files, it shouldn't, but in extremely rare cases, yes it can. However, you can run it in a read only mode by omitting "/r" and "/f".

 

Not entirely sure this will help though, but it's worth trying.

 

Already running beta. Waiting for new version of DP for ages... Presonaly, methinks Alex has buried himself with CloudDrive for good, while DrivePool very much needs some vital features and updates.

 

Yes, and no. He does tend to focus on a single product at a time. But he does look at issues, especially when they're stuff like this.

 

 

Heh :) It is in the pool. But I very much understand the reason for you asking this anyway. Been there, seen that, many a time.

Well, if the settings got corrupted, it could cause odd behavior, so I wanted to make sure that wasn't the issue here, actually.

 

But thank you for confirming.

 

Nope. Neither folder nor any files show up in the pool. I can only find/see them if I browse through the individual drive via NTFS mountpoint.

If my GF wouldn't have wanted that movie, I most definetely would've missed this shit completely. No "hicups" while browsing the pool, no Scanner reports, that anything is wrong. File system - OK, S.M.A.R.T. - OK. Hence, I'm thinking, how long have this shit been going on?

That's ... really odd. If you can access the files and folders normally, it really should show up in this list. Specifically, the driver reads from the underlying disks and combines the contents (when listing folders), or reads from one or both copies of the file directly. So this really shouldn't happen.

 

 

K, done. Where to send?

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

There should be a Box.com upload widget at the bottom of the page. Use that.

 

Also, just in case, could you run the following command, and post the log from it?

dpcmd check-pool-fileparts [PoolDrive]:\Movies\Good\The.Danish.Girl.2015.1080p.BluRay.DD5.1.x264-SA89\ 4 > missingpart.log
This should output the disk info, and what files reside on which disk, and duplication level.

 

 

Awesome. I totally forgot about checking the event log.

 

And I think or rather hope I found the culprit. The event log is littered with these warnings:

 
The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{32abe257-2f68-4778-8927-670457764ce3}, DeviceName: \Device\HarddiskVolume41.
(The I/O device reported an I/O error.)
The volume id 32abe257-2f68-4778-8927-670457764ce3 is for the drive itself, and HarddiskVolume41 is actually the pool. So I guess drivepool tries to write something in on that drive and the drive errors out?

Also it shows some of these:

The IO operation at logical block address 0x172df9a0 for Disk 22 (PDO name: \Device\00000051) was retried.

 

 

 

The I/O error is may be the key, but I'm not 100% sure. That it's seeing that is a bit odd, but that you're seeing an error on the an underlying disk (I'm guessing that Disk 22 is in the pool? and may the disk with the files on it), it could be that it's taking too long to respond with the file properly and erroring out (I'm not sure of that though).

 

Questions:

  • Is this the cause of these missing files?
  • Why the hell there are no S.M.A.R.T. warnings then?
  • Should I retire this drive or do you recon, it's still safe enough to keep it online for duplicated files only? Am I stupid even thinking about keeping using it?
  • How do I go about trying to fix this drive? chkdsk and thats it? Or anything better out there? Should I clear the drive before hand?
  • This drive is used by uTorrent. I have uTorrent seeding ~450 torrents, most of them have their files on the pool. This drive is one of 3, that are dedicated for uTorrent only. I mean, I remember reading that uTorrent and DrivePool does not play nice. Maybe that's the problem?
  • Any good software to keep an eye on windows event log and report things like this automaticaly in the future?
  • Specifically, that the driver isn't grabbing the info from the underlying disk. While this sounds simple, it really is a very complex thing. The logging above is done in the driver, at the kernel level, so it is very detailed and may indicate exactly why this is occurring.
  • They don't always. Some timeouts and retries are normal. They happen, and usually seamlessly. Ever seen a disk taking forever to load a file? That may be why. Though, depending on the disk, it may report Soft Read Errors when this happens.
  • I'm not convinced that the disk is the problem. Also, keeping it "in play" means that we can better troubleshoot the issue (this is one of those really, really hard to reproduce issues, so a "live" system with the issue may be very helpful). That, and if the disk isn't the issue ...
  • CHKDSK, yeah. It does wonders, and it is very reliable (it may not have been in the past, but NTFS is very, very resilient).
  • uTorrent should be be fine with the Pool. The issues with it are open files (seeding), and IIRC, BT Sync may show all copies of the duplicates. Aside from that, it should work on the pool just fine.
  • Not really. You can create scheduled tasks that trigger on specific events, and do stuff like emailing you. Otherwise, ... nope.
Link to comment
Share on other sites

  • 0

@Chris thanks for answers.

 

So I cleared the drive of unduplicated files (with the Drive Usage Limiter). It's been 4-5 days and there's no new warnings in the event viewer. Good sign? :)

I guess I'll wait and see what happens :)

 

This drive is one of 8 Seagate external drives I bought on Black Friday in 2011. The prices for drives have jumped sky-high back then, after the flood. I would've never bought external drives, but these went for ~$90, so I jumped on them in anticipation, that the prices will not go down quickly.

Anyway, the first one of these died like 5 months ago and I thought, here it goes, now they probably gonna drop like dominoes. Sadly, I was right - 5 drives dead in 5 months. And it seems 6th is showing signs of giving up :)

Well, I guess 4-5 years is kinda OK for these drives - ST3000DM001 have awful reviews everywhere and warranty was only 2 years.

Link to comment
Share on other sites

  • 0

Sorry for bumping an old thread, but i think i'm experiencing this as well chris.

Was looking for a movie earlier this morning.

But I couldn't find it despite kodi listing the movie, and the utorrent has the file downloaded 

So i redownloaded it again in utorrent

Strangely when i try to move the file to the exact same path, the movie reappears!

.. and prompt me if i want to replace the same file or not...

 

Drivepool is the latest version 2.20 742 beta

Link to comment
Share on other sites

  • 0

Sorry for bumping an old thread, but i think i'm experiencing this as well chris.

Was looking for a movie earlier this morning.

But I couldn't find it despite kodi listing the movie, and the utorrent has the file downloaded 

So i redownloaded it again in utorrent

Strangely when i try to move the file to the exact same path, the movie reappears!

.. and prompt me if i want to replace the same file or not...

 

Drivepool is the latest version 2.20 742 beta

 

 

Try the latest beta then:

http://dl.covecube.com/DrivePoolWindows/beta/download/StableBit.DrivePool_2.2.0.829_x64_BETA.exe

 

Also, note, that it may take a bit for the full list to populate, as the pools is handing back results as soon as it gets them.

 

Worst case here:

http://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings

Set "CoveFs_SortedDirectoryQueries" to "True" and reboot the system.

This will make directory enumeration take longer, but it will/should be a complete list.

 

 

 

If you're still having issues, get tracing logs (link) and reproduce. 

Also, try using the dpcmd's "check-pool-filepart" for the specific file (x:\path\to\file.ext), and see what it reports. 

 

Also, run the StableBit Troubleshooter, just in case:

http://wiki.covecube.com/StableBit_Troubleshooter

 

 

 

Also, as for the torrents, are you downloading the file to the pool and then moving it's location on the pool?

 

If so, try to make sure you set the download folder to have the same NTFS permissions as the destination folder, or use a separate disk altogether for downloading) 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...