Jump to content
  • 0

After Migrating to Drivepool, network access to pool VERY slow.


crankycowboy

Question

I recently migrated from Storage spaces to drivepool mostly for the rebalancing feature.  A big thanks to Chris for helping me get converted over. He answered more questions than he should have had to and even responded to me at all hours.

 

In any case, I'm migrated over and functioning.  However, I have noticed a dramatic drop in performance regading the pool

 

I run a media server with about 20 tb of data within the pool (mostly movies, music, tv shows etc).  The server is a dedicated system that is accessed via various clients (Roku, PCs, apple TV, mobile devices etc). On my main PC, I have the pooled "drive" mapped and can access it. When I click the mapped drive there is about a 3-4 second delay before the contents show up. And when I click on any folder it shows it's empty....and then 3-4 seconds later it shows the contents.  A similar issue is now happening when I try to watch a movie on a client (such as a Roku).  It will stream fine and then it will freeze and say "loading".  Furthermore, on the server itself, occasionally when I click on the pool (drive "d"), it displays similar behavior (there is a delay before showing the contents).  I NEVER had this or similar issues with Storage spaces so I have to assume the issue is directly related to drivepool. I am also running scanner. Right now, I don't have any files "duplicated" as I was working through "baby steps". Any help or suggestions are greatl appreciated.

 

Thanks in advance!

 

Cranky

Link to comment
Share on other sites

21 answers to this question

Recommended Posts

  • 0

Are you seeing a lot of disk activity on your server?  I use Procmon on the server to see what files are being accessed and by which programs.  Excess disk activity might be causing some of the issue.  I know for the first full day after installing DP, there was a lot of activity for various reasons.  But now that it has settled down, my speed of access and transfer rate is actually higher with DP.  Before I would get about 60MB/s at best.  Now I am getting 90-100MB/s regularly.  

Link to comment
Share on other sites

  • 0

You are very welcome! :)

 

As for your issue, that sounds more like a networking issue than an issue with DrivePool. I suspect that if you opened up the share locally, that you wouldn't experience this issue at all.

I say that, because I'm very familiar with networking issues. In fact, I significantly helped with this forum post:

http://forum.wegotserved.com/index.php/topic/8335-before-you-post-media-stuttering-playback-issues-performance-irregularities/

 

you should definitely check that out. That may solve most or all of your problems, actually.

 

If it doesn't, then please do this:

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

And replicate the issue you're seeing.

 

Also, I don't recall the CPU you were using or the amount of RAM. I mention this, because there are a few options that depend on this. Namely, "Network IO Boost". If you have a decent system, then I would recommend enabling this option (off by default, because it does use more resources).  If that doesn't help, try toggling "Read Striping", if any of the folders in question are duplicated.

Link to comment
Share on other sites

  • 0

Thanks for the replies guys. And thanks once again "Darshna"!

 

Actually, to the contrary, it doesn't seem to be a network related issue, and I can replicate the problem on the host machine itself.  For example, if I open a folder with all my photos (probably 100 folders, each with several pictures in it), initially when I open the directory it says "this folder is empty", and then the scrubber bar at the top slowly starts filling and the "please wait" icon (used to be the hourglass) starts spinning, and eventually the folders start showing up and the thumbnails become visable.  I had a similar problem where I had a symbolic link "forwarding" my appdata directory to "Plex" to the large "pool" drive. None of my movie posters or data from that folder were showing up within plex. I read that Drivepool had problems with the symbolic links and Plex (in the plex forums), so I copied the data to a drive within the pool but placed the data directly on the drive, and the problem was fixed. As a test, I did the same thing with my "pictures" directory described above, and then re-mapped the drive to my network. On the host itself, the folder refreshed very quickly and also did so over my network. This appears to be directly related to drivepool and the symtoms didn't start until I migrated away from storage spaces. Oh, and in quick settings on drivepool, I selected "server" which auto-populates the Network IO boost and read striping.  And to answer your question, my server is an I5 3570 3.5 GHz with 8GB ram. Running windows 8.1.  I started reading the link you referenced, but it's over 20 pages so I'll have to continue with that shortly. Any other suggestions to my particular problem since it doesn't seem to be network related?

Link to comment
Share on other sites

  • 0

Okay, if this is happening on the local machine, then there is definitely an issue.

 

Could you enable file system logging, reproduce the issue and send the logs to us?

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

 

As for Plex.... don't do it. Plex requires hard links for the metadata, which we don't support on the pool (it's a very technical issue). Plex relies on that instead of "cross linking" the files in the database it stores...

So definitely don't place the plex metadata on the pool.

 

As for that link, only the first post is important. It's been edited to include ALL of the important information.  But if this is happening on local storage, then it may not help.

 

Also, are any of your disks on a different "bus" than the others? Such as on a different controller, or on USB? IF so, this could slow down the pool.
You may be able to pin down a single (or a couple of disks) by accessing the hidden pool part folders directly.

Link to comment
Share on other sites

  • 0

Thanks again Christopher, I have enabled logging. The problem is there isn't a moment where there is a defined error. Accessing the pool (even on the host machine) is just extremely slow. I have noticed that now when streaming, it will stop to buffer or freeze completely where it would NEVER do that before. I'm assuming it's because plex on the host server is having trouble reading the data (same delays I can witness live when on that machine).  Like I previously said, if I move the data out of the pool and place it directly on one of the disks that's part of the pool, the problems goes away...so it appears to be directly related to drivepool's pool.  Should I just let it run for a while and try to access various disks or stream data and then send the log? Or what do I need to do since I'm not technically seeing an error, just delays in access. Please advise. Thanks!

Link to comment
Share on other sites

  • 0

Well, the file system logging... well does exactly what it sounds like. But, it is incredibly detailed. It may contain information about what is causing the issue, actually.

 

also, could you do this:

http://wiki.covecube.com/StableBit_DrivePool_Q2159701

 

And do you have any antivirus or disk tools installed (such as defragmentation or backup)?



And yes, enable logging, try to stream, and then stop logging. Then upload the log files to us. The shorter, the better (as these logs are VERY detailed). 

Link to comment
Share on other sites

  • 0

I have posted a video showing the crazy "lag" associated with accessing the pool on the host computer (my server).  That video can be found here: 

 

Obviously with that type of delay accessing the pool on the host, streaming over my network has no chance.  I have copied a folder out of the pool and placed it directly on one of the disks within the pool and shared that drive over my network and then shared the folder with plex. I can stream fine again, so directly related to the pool for sure.  As previously stated, I have enabled file logging and have sent that file to you. I also tried the option you suggested below regarding filters. That result is attached here as well.

 

No anti-virus installed, and no disk tools or any other utilities installed/running. I try to keep my server as stripped down as possible.

 

Something that is weird regarding this situation is I initially just did the "trial" version of both drivepool and scanner. Both worked fine for my entire demo period. I ran it in "trial" mode until the trial period only had one day left and then purchased both. Almost immediately, i  started having problems.  I'm not insinuating that my issues have anythign to do with the license, but just wanted to verify that functionality hasn't been flawed the entire time....Please help!  As this simply isn't functional for me in it's current state and I need to figure out what direction I need to go with this. Thanks again Christopher!

post-1698-0-94275500-1410095444_thumb.jpg

Link to comment
Share on other sites

  • 0

Okay, that is definitely really bad.  And yeah, would kill any sort of streaming.

 

The filters are "fine" (default ones installed on Windows 8), so that should be the issue.

 

Do any of the HDDs in the pool exhibit the same behavior? If they do, it could be that drive (or drives) that are the issue.

ANd if so, it could be something as simple as a bad cable or loose connection. Which is simple to fix.

Also, if youve StableBit Scanner installed, try running a "burst test" on each of the disks in the pool (right click on the disk and select "Burst test"). This is good for identifying problems with the drive/cable/controller.

 

Also, have you tried turning Network IO Boost off and see if that helps?

 

As for the trial vs full, the trial is fully featured. The only difference between the two licenses is that one is time limited, so that should not be an issue at all. I'd say much more coincidental than anything.

 

Also, one thing to try here, do a complete reinstall of StableBit DrivePool:

http://wiki.covecube.com/StableBit_DrivePool_Q3017479

 

Normally, I'd say just do a completely reset (gear icon-> troubleshooting -> reset all settings) and see if that helps. But the link above is much more thorough, and removes the drivers as well. 

If you're still having issues after that, let us know.

Link to comment
Share on other sites

  • 0

Thanks again Christopher. I did the manual install as you suggested.   Didn't seem to help...actually if anything, things seem to be worse now as "scanner" is stuck on the "initializing" screen and drivepool has been stuck on "starting services" for the last 15 minutes. Although, I can access my pooled drive, but exhibiting the same slow behavior. I was thinking of disconnecting each drive one at a time and restarting to see how things shape up.  Another "weird" thing, when I was in device manager after the "reinstall" I noticed some weird looking "errors"  See attached.  I have errors associated with portable devices which look to be drives. It says windows has stopped the device because it has reported problems. Another weird caveat is that I don't have anything assigned letter "g" but it's reporting a problem.  This is the strangest thing!

 

On the burst test, when I started it (when I could still get scanner to start), it said to run for at least 24 hours on each disk...is this what you were suggesting?  Regarding the trial vs licensed...I know it was completely coincidental. I was just trying to state that everything had worked as expected at one point and hadn't been displaying this odd behavior from the get go.  Any other suggestions? Thanks!



Just an update, both scanner and drivepool have started. Displaying the same behavior, but both are running.

post-1698-0-89621000-1410124025_thumb.png

post-1698-0-93699800-1410124028_thumb.png

Link to comment
Share on other sites

  • 0

Do you have any removable devices and/or card readers?

This includes USB drives or devices?

Or any optical drives?

 

As for the burst test, longer is better, but if you can get it to run overnight, that's fine.

 

 

Also, have you ran a memory test on the system?

And it may be a good idea to run "chkdsk c: /r /scan /perf" and then, "sfc /scannow" to make sure that the system is intact.



Also, do you ahve any disk or controller errors in the event viewer's "System" section?

Link to comment
Share on other sites

  • 0

Thanks again for sticking with me on this Christopher.  I don't have any removable devices or card readers...no usb drives or optical drives. I do have the typical peripherals (ie mouse, keyboard, etc connected via usb).  I also have a PCI-e sata card for the additional drives.  Ironically, the exclamation points in device manager are now gone, and now say they are "functioning properly".

 

I haven't run a mem test <yet>.

 

The chkdsk came back with no errors; however the sfc /scannow returned some errors:  "Windows REsource Protection found corrupt files but was unable to fix some of them".  Any idea what that is?

In the event viewer, I do have some errors/warnings. One that jumped out at me was storahci error 129.  Not sure what that is, but looks like it could be disk related.  Another piece of information that could be helpful is I had disabled scanner just because some people have claimed compatibility issues with some hardware so I wanted to eliminate that. However when I re-enabled it, I have an error on one of my disks that looks like this:

 

  • ST3000DM001-1CH166 - 3 warnings
    • There are currently 18960 reallocated sectors on the hard disk. A reallocated sector is created when a sector cannot be read or written to. In such a case, the next time the sector is written to it is swapped in for a good one from the spare sector pool. Having reallocated sectors decrease read/write performance and indicates drive trouble.
    • These are currently 232 unstable sectors on the hard disk. An unstable sector is a sector that can't be read. The drive will automatically swap the bad sector for a good one whenever new data is written to it, however, the original data may be lost.
    • These are currently 232 uncorrectable sectors on the hard disk. An uncorrectable sector is counted when the drive cannot read or write to a sector. This can indicate mechanical drive trouble.

 

This is on a drive that has quite a bit of data on it. Would you suggest "moving " the data to another drive?  Any additional help is greatly appreciated as this whole thing seems to be falling apart and it makes it painfully obvious how much my family relies on that media server :)  Thanks again!

Link to comment
Share on other sites

  • 0

Hi

 

I had a similar slow down when I had a failing harddrive try removing the drive and see if it clears up your problem if it does you may still be able to save the drive. Bad sectors don't always mean a drive is knackerd but running scanner/chkdsk/scannow doesn't really fix the problem the best way and way I have done it in the past is remove the drive from drivepool using the option to only remove unduplicated files it's the quickest way. Once that's done do a full format of the drive either using windows or a vender utility then run chkdsk and stuff and 99% of the time because the full format writes a zero to a sector the drives auto remap kicks in and the bad sectors gets removed then add the drive back to the pool and let it auto balance I have done the above a few times now and still using drives that would have been binned with no problems.

 

Lee

Link to comment
Share on other sites

  • 0

Lee is absolutely right here.

 

Also, first, reallocated sectors may not be a "bad" thing, but they can DEFINITELY cause a disk to slow down. Instead of the sectors being sequentially, they've been remapped to slack space. This means that hte read heads have to jump around when they hit these reallocated sectors, reducing performance.

 

Coupled with the uncorrectable and pending sectors... You definitely have a bad disk here. This disk could definitely, single-handedly be the issue here.

 

And yes, I would highly recommend moving the data off of this drive. If only because of the uncorrectable/pending sectors. These can lead to corruption or damage to the data.

 

 

As for the error from SFC, that's generally not a good thing. It indicates that there are issues with system files that it cannot resolve. One option to fix that is to run "DISM /Online /Cleanup-Image /RestoreHealth". However, if this is a mostly clean system, a reinstall may be simplier.

Link to comment
Share on other sites

  • 0

I had to go out of town on business so this got put on the back burner. However, it's still <somewhat> of an issue.  I probably just need to format to resolve the SFC findings (hopefully). Is anyone familiar with the "refresh your PC" option in Windows 8? Anyway,  a couple of observations. While I was gone, the performance increased substantially on the pool <not quite back to where it was, but much better>. However, recently resorted back to the poor performance. I noticed that when the pool is performing terribly, it is "measuring" (which typically takes a day and a half). During this time, it acts like it did in the video above. When it's done and the bar at the bottom of the interface turns from the yellow "measuring" to blue...it gets MUCH better. However in a day or two, it starts "measuring" again. What initiates the "measuring" procedure if I am not doing it manually? Just back to the troubleshooting side of this and wanted to ask since that is dramatically impacting my issue. I will probably resinstall windows in the near future, but would also like to take any steps to avoid being in this same boat after the install. Thanks!

Link to comment
Share on other sites

  • 0

Well, it shouldn't be measuring that often, actually.

 

That it is, is problematic. THis can happen if one or more disks temporarily disappears....

 

 

Also, if the pool is slowing down that much by measuring the pool (scanning files), that implies that one or more drives are having a serious performance issue here. 

And this would seem to indicate a hardware issue with the drives more than an issue with DrivePool itself. 

 

However, could you enable file system logging and let this issue repeat itself?

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

Once it does, upload the logs to us, and we'll take a look.

But I suspect that it will indicate an issue with the hardware.

Link to comment
Share on other sites

  • 0

I have had this issue as well from time to time, I have about 50 TBs online, in three pools, I suspect that disks that may have bad sectors are slowing down drive pool - I have used the method that Lee posted as trying to resolve the issue on the server hasn't seemed to work.

 

Has crankycowboy's issue been resolved?

 

Would really like to see an update on his issue, it is very much the same I have experienced.

Link to comment
Share on other sites

  • 0

I believe it was.

 

 

If you have bad sectors, it will adversely affect the performance on that drive.

 

Also, if you have antivirus software installed, it can adversely affect the performance of the pool. Ideally, exclude the hidden "PoolPart.xxxx" folders on each drive. Or uninstall it.

 

Additionally, a number of disk tools can cause issues as well.

In fact, if you run "cmd /k fltmc", you can get a list of the file system filters. If you copy and paste that list, we can look up the filters and see if any one of them may be the issue.

 

And do you have StableBit Scanner installed on the system? It will report the SMART errors on the drive, and perform a surface scan, to identify any bad/unreadable sectors on the drive.

 

And worst case, you can do this:

WHS2011 (v1.3): http://wiki.covecube.com/StableBit_DrivePool_Log_Collection

Windowss (v2.X): http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

Link to comment
Share on other sites

  • 0

Well I formatted and everything seemed to be resolved....but now it has reared it's ugly head again. I think I have narrowed down part of the problem.  I have a huge draw on my office with all the electrical equipment in there, so a breaker will pop pretty often (this new house has AFI breakers which tend to pop more easily with dc power supplies).  My breaker will trigger about once a week or more. Yes, I know this is part of a larger problem, which I'm trying to address.  But every time the breaker "pops", drivepool goes back to "measuring". Which takes two days or more typically. During this time, everything goes to a crawl again. If the breaker triggers again during this time, it extends things. Should drivepool start "measuring" every FREAKING time power is lost unexpectedly?  Please advise as I've loved the community and Christopher has been a saint, but I just can't keep dealing with not being able to access my pool while "measuring" all the time.  Like I said, I'm addressing the "afi breaker issue"...but this is crazy.  Let me know your thoughts please.

Link to comment
Share on other sites

  • 0

Is the system losing power when the breakers trip? You didn't mention this specifically. If it is, then it could definitely do that.

 

However, if it's just fluctuating, it could cause the drives to lose power enough to drop out for a second or three, and that would cause the pool to be remeasured.

 

The first thing I'd recommend is a UPS (Battery backup). Even if the breaker isn't fully tripping, it will "smooth out" the power, to make sure that there are no issues related to power. In fact, this is a good idea for any important computer system.

Link to comment
Share on other sites

  • 0
3 hours ago, AlexOb said:

Having this same problem.  Doesnt look like a solid answer was figured out here.  Curious if anyone's had this issue and resolved it?

Mostly, because this is a complicated issue, and there isn't a "one size fits all" type solution. 

There are a number of things to try:

  1. Run "netsh interface tcp set global autotuninglevel=highlyrestricted" on both systems, from an elevated command prompt, and reboot the system.
  2. On the Network Adapter properties, disable any setting with "offload", or "checksum". Disable "green" features, and Interrupt Moderation, if present.  Enable flow control, and disable jumbo packets (or toggle them, as they can help in some cases)
  3. Update the drivers for the network adapters
  4. Check for any antivirus software, and (temporarily) remove them
  5. Disable Windows Search on both systems.

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...