Jump to content
  • 0

Removing drive from pool


Edward

Question

Am in the process of removing a drive from my pool due to repeated SMART warnings (head parking on a Seagate 3tb drive).

 

The process has apparently been going on for some time now (ca. 3 hours).  However, except for the bar at the bottom saying removing drive, there is no evidence that anything is happening. In particualr there is no 'disk activity' being indicated.

 

Is there anything I can do to actually verify that the process is working as it should in the background? Is there any way I can ascertain how long the process is likely to take?  Ballpark figure would be fine.

 

Am running drivepool 2.1.1.561 under WHS11.

 

thx

Edward

 

 

Link to comment
Share on other sites

Recommended Posts

  • 0

Since I'm facing the same issue, please allow me to continue under this same topic. If a new topic is required, I will do so. Thanks

 

Ref: Contact Request 1828677 that submitted to Support 9 hours ago, but not yet received help.

Our case:

We are using a trial version of DrivePool version 2.2.5.1237 for the past few days on Server 2019. Test details:
. 4 drives (2 x 4TB, 3TB, 2TB) in a Mediasonic Probox for data, USB connection.
. 2 x 240 GB SSD's in a separate enclosure for Optimizer, USB connection.
. DrivePool started well. Added SSD Optimizer + Order File Placement Plug-ins.
. Loaded 6.79 TB data, unduplicated. Current data distribution:
  2.34 TB on each of the 4TB drives
  1.43 TB on the 3TB drive
  531 GB on the 2TB drive.
  175 GB on one of the 2 SSD drives.
 
We wanted to test the disk removal. So, 24 hours ago, we clicked the removal button next to the 3TB drive. Next screen we left all 3 options unchecked, and confirmed the drive removal. The button changed to 'Waiting to Remove'. Message at bottom of the screen 'calculating' with the yellow progress bar moving from left to right continuously.
More than 24 hours later, the screen remains the same, nothing changes.
 

Update:

Since there is no reply from the request, I decided to cancel the process. DrivePool dashboard frozen that required Task Manager to close. DrivePool won't start that required a reboot. Upon reboot and starting DrivePool again, the 'balancing' initiated. When that completed, I was able to start the 'Dish Removal' process again. There is indeed a progress with percent counter. That is better.

So, DrivePool needs to 'balance' first, which requires to empty all contents on the SSD cache. That makes sense. However, DrivePool is not robust enough to give a warning and lacking on the balancing task. I'd like to see both these incorporated on the next release. That would give more confident to switch over from Storage Spaces.

Sincerely,

Edited by TPham
Update
Link to comment
Share on other sites

  • 0

Stablebit does not have a large support department; as I understand it, your support request is going directly to the developers. As much as we'd all like personal 24/7 valet service for our USD$29.95, they do need to do other things from time to time. :)

That said however, DrivePool simply should not need to balance first to remove a drive (caveat: so long as there's room), and certainly shouldn't sit there for 24+ hours doing nothing obvious and then require a reboot to get it unstuck! I hope that Stablebit is able to find the cause and fix it. Please keep us updated?

Link to comment
Share on other sites

  • 0

I've done it. It's literally plug-and-play. So long as DrivePool is installed on a computer and you have the physical ability to plug in your pooled drives, it'll automatically recognise the pool on those drives so you can play with your files straight away. Doesn't matter whether it's Windows 7/8/8.1/10/2008/2011/2012/2016/2019, Home/Pro/Enterprise/Server.

In fact, even if every single copy of both Windows and DrivePool somehow disappeared overnight (cue spooky music), you would still be able to access your data with any other OS that can read standard NTFS volumes (they just wouldn't be conveniently pooled anymore).

Also, if a drive in your pool fails, it'll be marked in DrivePool's GUI as missing and still displaying the drive letter (if any) and volume label (I set the latter to match the drive's serial number), and if you hover the cursor over the missing drive it'll show you the model (edit: and serial number if possible), and the pool will still be accessible (albeit read-only until the problem is resolved and any files unique to that drive missing, so if you've duplication enabled for the entire pool you won't have lost anything).

Link to comment
Share on other sites

  • 0

Thanks, Shane, for confirmation about moving DrivePool. It is most IMPORTANT to know.

Setting drive volume label to match the drive's serial number is a great practice. Better than a bunch of 'New Volume' labels. Also thanks for the other tips. I'm a magnet for troubles, so every tip, big or small, I collect them all, just in case :).

Since you mentioned about duplication, I think this feature on DrivePool is COOLER than Storage Spaces, as we can set selective folders, not the whole pool.

Best regards

Link to comment
Share on other sites

  • 0
On 1/7/2021 at 9:14 AM, TPham said:

Setting drive volume label to match the drive's serial number is a great practice.

If setting the drive volume label to match the drive's serial number works for you, then stay with it.

My approach is different and works better for me. First of all, I currently have 18 USB HDDs in my DrivePool. I don't use Drive Letters on any of the pool drives. There are just too many drives in the pool and DrivePool does not need Drive Letters anyway. I preferred to clean up my Windows File Explorer listing, so the Drive Letters had to go.

What I do is just name the drives in logical order as they sit on the shelf. Being not too creative, my pool drives are labeled as DP01, DP02, DP03, etc.... I also put a label on each drive case. I have DrivePool GUI sort the pool list by name for easy reading. If there is any problem with a HDD, I immediately know which drive is affected.

On 1/7/2021 at 9:14 AM, TPham said:

Since you mentioned about duplication, I think this feature on DrivePool is COOLER than Storage Spaces, as we can set selective folders, not the whole pool.

I have lots of unduplicated home media files in my DrivePool. But I also have a few folders that I want 2X duplication. Not only do I find the duplication options better in DrivePool than Storage Spaces, but the net result is that I am saving lots of money by not duplicating my entire pool when it is not required for about 85% of my stored media files.

Also, I have had a couple HDD failures in the past month, and I have been able to recover almost all my data off the drives. In the meantime, DrivePool was still serving up all my other files like nothing happened. When my Storage Spaces crashed, it could takes weeks to rebuild. I don't miss Storage Spaces....

 

 

Link to comment
Share on other sites

  • 0

@gtaus

I'd love to know how to reply by sectional quotes as you and others did. Since I don't know how, I'm responding by the old numbering system, sorry.

1> 'Drive Letters had to go'. Agreed. How do you add the drives in DP without drive letters? Without drive letters, explorer won't show the drives, is that so? Can you give some samples how this could be done. In fact, I've been wondering about this subject, as there are only 23 letters left after A & B & D.

2> 'Duplication options better in DrivePool than Storage Spaces'. Agreed. I've thinking about organizing DP in three folders. Media no duplication. Data 1x2 duplication. Documents 1x3 duplication. This should work well, I think. I do have 2 x 250 GB SSD for cache. I think I'm going to like this setup.

Additional observations & considerations:

3> Last week, I had to nuke the DP setup as the media streaming got a hit. I notified Christopher and he reached out to me with advises. I just started to setting up DP again. To my delight, the streaming has gotten better. I placed order for additional 16 GB RAM and methodically built up DP. I hope to finally Activate it soon. So, if there is any tips (Do & Don't), I would most appreciated. Our fist priority is Reliability. Some of us can't live without it. :)

4> I plan on retiring the some virtual disks from storage spaces for use in DrivePool. Afterward the Pool enclosure can be disconnected from Server & saved remotely with the remaining virtual disks (just in case). The problem is I can't identify the drives in order to properly do the removal process. Get-Disk command does not show actual serial numbers. Here is what I got:

Listing as seen in Storage Spaces with 'wmic diskdrive get model,name,serialnumber' command.
HGST HDN 724040ALE640 USB Device, Model 724040ALE640, SN 152D00539000
HGST HDN 724040ALE640, Model 724040ALE640, SN 152D00539002
TOSHIBA MD04ACA400 USB Device, Model MD04ACA400, SN 152D00539003
TOSHIBA MD04ACA400 USB Device, Model MD04ACA400, SN 152D00539002
HGST HDN 724040ALE640 USB Device, Model 724040ALE640, SN 152D00539003
HGST HDN 724040ALE640 USB Device, Model 724040ALE640, SN 152D00539001
WDC WD40 EZRX-22SPEB0 USB Device, Model EZRX-22SPEB0, SN 152D00539001
WDC WD40 EZRX-19SPEB0 USB Device, Model EZRX-19SPEB0, SN 152D00539000
etc . . .

Thanks

 

Link to comment
Share on other sites

  • 0
1 hour ago, TPham said:

I'd love to know how to reply by sectional quotes as you and others did.

@TPham

You simply click the mouse and drag the cursor over the text you want to copy. The text should turn to blue background. When you get to the end of your selection, release the mouse and a popup box should appear "Quote selection". Click on the popup box and it will transfer that quote to your response box.

1 hour ago, TPham said:

1> 'Drive Letters had to go'. Agreed. How do you add the drives in DP without drive letters? Without drive letters, explorer won't show the drives, is that so? Can you give some samples how this could be done. In fact, I've been wondering about this subject, as there are only 23 letters left after A & B & D.

Unlike Windows File Manager, you don't need a drive letter for DrivePool to see and list the drive. I just added another HDD to DrivePool this morning, without a Drive Letter, and all I had to do was simply click the add button on the Non-Pooled drives list on the DrivePool GUI.

I currently have 18 HDDs and 1 SSD in my DrivePool. So I wanted to remove all drive letters from my pool drives and free those Drive Letters for flash drives and/or other devices I might want to temporarily add to my system.

If you ever need to work with any drive in Windows, you can go into Disk Management and add a Drive Letter to any pool drive you want. DrivePool does not care if you add/remove a Drive Letter to the drive, it identifies the drive by the hidden PoolPart folder it writes to the drive itself.

1 hour ago, TPham said:

2> 'Duplication options better in DrivePool than Storage Spaces'. Agreed. I've thinking about organizing DP in three folders. Media no duplication. Data 1x2 duplication. Documents 1x3 duplication. This should work well, I think. I do have 2 x 250 GB SSD for cache. I think I'm going to like this setup.

That is one way of organizing your duplication folders that I have also considered. It certainly would clean up the root directory to only 3 folders.

I really noticed a performance boost in DrivePool when I added a front end SSD. DrivePool is limited to the speed of the drive it is writing data to at that time. If it is writing to a SSD cache, then it's very fast. When I transfer large amounts of data, I am glad I have that SSD front end cache.

1 hour ago, TPham said:

I placed order for additional 16 GB RAM

DrivePool can use a SSD as a front end cache for writes. That works good enough for me. If you are a power user and need more speed in your server, I would suggest you take a look at PrimoCache. PrimoCache can use both your system RAM and your SSD as cache/buffers for reads/writes. I believe PrimoCache still offers a free trial period to use their software to see if it works for you.

Years ago I tried PrimoCache, but I only had 4GB RAM and no SSD on my system. Writing/reading from RAM was lightning fast with PrimoCache, but with limited RAM, the buffer filled up in no time with large data transfers (>3GB at that time) and dropped back down to the slow write speed of my archive HDD. So it was not worth it for me at the time to buy the program. Your situation is different with 16GB RAM and 2 SSDs. I'd recommend checking out the program at least for the free trial period.

I use DrivePool primarily as my home media storage server, so using the SSD as a front end cache is enough for me. Even without using the SSD, DrivePool was just fine for my needs and I had no complaints. The SSD just caches the writes faster on large data transfers. Sometimes that comes in handy.

2 hours ago, TPham said:

The problem is I can't identify the drives in order to properly do the removal process.

Yeah, I had a hard time trying to figure out what drive/serial number I was working with in Storage Spaces. If you buy a few of the same HDD models, only the serial number may be different, which was a real pain for me.

Anyway, when I set up DrivePool, I just named my HDDs DP01, DP02, DP03, etc... I placed them in physical order on my shelf and I also tagged each drive/case with the drive name. The drives I have in my ProBox enclosures are also in physical order, so I know exactly which drive is in the slot. My life was easier when I got away from using serial numbers to identify the pool drives.

I don't have any magic way to identify your HDDs with serial numbers in your enclosures. Sometimes the software will have a feature to "identify" your drive which might flash a light on the drive slot if your enclosure has that feature. Mine did not. If you have a USB disk caddy, you could pull your enclosure drives and identify them in the caddy. I'd label the drives at that time, if needed, before returning them to the enclosure. But, like I said, with DrivePool, I just tagged and named the drives in the physical order that they are placed in the enclosure and/or on the shelf. Anything to make like simpler works better for me.

Link to comment
Share on other sites

  • 0
2 hours ago, gtaus said:

Unlike Windows File Manager, you don't need a drive letter for DrivePool to see and list the drive. I just added another HDD to DrivePool this morning, without a Drive Letter, and all I had to do was simply click the add button on the Non-Pooled drives list on the DrivePool GUI

COOL ! I'll try that. Thanks

2 hours ago, gtaus said:

DrivePool can use a SSD as a front end cache for writes. That works good enough for me. If you are a power user and need more speed in your server, I would suggest you take a look at PrimoCache. PrimoCache can use both your system RAM and your SSD as cache/buffers for reads/writes. I believe PrimoCache still offers a free trial period to use their software to see if it works for you.

I've heard about PrimoCache and was tempted to try, but was reluctance to add third party apps. DP took a large percentage of memory on our server hence I nuked the installation & returned to 'storage spaces'. Since then, and with Christopher's options, I am trying again. If additional memory works, then I'm most happy.

2 hours ago, gtaus said:

If you buy a few of the same HDD models, only the serial number may be different

Well, 'storage spaces' does not return REAL serial numbers. I have 8 disks in the Pool. Get-Disk command returns 2 x SN 152D00539000. 2 x SN 152D00539001, 2x SN 152D00539002, 2 x SN 152D00539003. I guess that reflects the drives according 'parity' setting (if it means anything). So, without real SN, it requires trial & error to locate the desired drive it its location among several enclosures.

So, your way, no drive letter and naming drives as DP01, DP02, etc . . . makes perfect sense. I'll try that with new drives and the ones already in DP. KUDO to you.

It is a pleasure discuss with you and others in this forum.

 

Link to comment
Share on other sites

  • 0
5 hours ago, TPham said:

I've heard about PrimoCache and was tempted to try, but was reluctance to add third party apps. DP took a large percentage of memory on our server hence I nuked the installation & returned to 'storage spaces'. Since then, and with Christopher's options, I am trying again. If additional memory works, then I'm most happy.

From what I have read, many people use PrimoCache without any problems. If you have lots of RAM, it just works better. PrimoCache lets you designate how much, if any, RAM and/or SSD you want to set aside for the cache/buffer. The free trial period is long enough to see if it works for you. I only run my own home systems, so third party apps is not as issue as it might be at the office.

I am surprised to hear that DrivePool is taking up so much memory. I just checked my server's Task Manager and my DrivePool is reported as using only 40MB of memory. I have 8GB RAM, so no issue for me on my 6+ years old HP desktop running Windows 10.

5 hours ago, TPham said:

Well, 'storage spaces' does not return REAL serial numbers.

If you don't have StableBit Scanner, which will report real serial numbers, you can download the free version of Hard Disk Sentinel and that will also report real serial numbers. I don't know if that will help you locate and identify your drives in your enclosures, but if you pull a drive and put it into a USB disk caddy, you would be able to identify each drive that way.

I can't remember exactly how Storage Spaces lists the drives, but somewhere along the line I got smarter when I moved to DrivePool and the simpe naming of the drives as DP01, DP02, DP03 works better for me. Don't forget to put a matching tag on the drive/case itself. 

I ran Storage Spaces for ~7 years and when it works, it is great. But when it fails, it can be epic. I had 3 catastrophic failures with Storage Spaces. On my last failure, I had 26 pool drives, Storage Spaces was set up for 2 drive failure, but I had only 1 HDD fail and it crashed my entire pool. OK, my home media server is not vital to life and I had HDDs as backup in the closet. But I just lost confidence in Storage Space's promise to rebuild itself after a HDD failure. 3 strikes and I was out with Storage Spaces.

I moved to DrivePool after Storage Spaces. The read/write speed of DrivePool is limited to the speed of a single drive whereas Storage Spaces can read/write packets from multiple drives at once. So, at first, Storage Spaces appeared to be faster than DrivePool. However, I added a SSD as front end cache to DrivePool and now I get faster write performance on DrivePool than I ever got with Storage Spaces. If you have 2X duplication on your DrivePool folder, it can take advantage of read striping to increase read speeds. Occassionally, I will set up a 2X TEMP folder just to take advantage of Read Striping for some temp folders of working files.

Since I have moved to DrivePool, I have had 3 HDDs fail. Those have been older drives I moved over from my Storage Spaces days and they were at the end of their life cycle. On 2 of those drives, I recovered almost all my data except for a few corrupt files. That made recovery really easy. On the other drive, DrivePool recommended that I run Scandisk on the drive because there was a problem reading the directory. So I did, but Scandisk (with repair) wiped out the entire directory and I then lost access to all the data on the drive. DON'T run scandisk with repair on a HDD until after you have transferred as much data off of it as possible. At any rate, DrivePool stores complete files on the HDD and not just packets of files like Storage Spaces. So, under many failures, you can still recover almost all your data off a DrivePool drive.

5 hours ago, TPham said:

It is a pleasure discuss with you and others in this forum.

I am still relatively new to DrivePool (less than a year), so I am reading posts here all the time trying to learn more. I try to give back if I am able. @Shane , a moderator, has been very helpful on many of my questions. Responses are always appreciated.

Link to comment
Share on other sites

  • 0
16 hours ago, gtaus said:

At any rate, DrivePool stores complete files on the HDD and not just packets of files like Storage Spaces. So, under many failures, you can still recover almost all your data off a DrivePool drive.

Good to know. Thanks

16 hours ago, gtaus said:

The read/write speed of DrivePool is limited to the speed of a single drive whereas Storage Spaces can read/write packets from multiple drives at once. So, at first, Storage Spaces appeared to be faster than DrivePool. However, I added a SSD as front end cache to DrivePool and now I get faster write performance on DrivePool than I ever got with Storage Spaces.

Good to know. I use 2 x SSD as instructed for duplication feature. They are good for writing.

16 hours ago, gtaus said:

I ran Storage Spaces for ~7 years and when it works, it is great. But when it fails, it can be epic.

Luckily, I have not had any failure yet. But sooner or later, there will be. My luck continues with the help of DP and people like you who are willing to share knowledge. Thanks

16 hours ago, gtaus said:

If you don't have StableBit Scanner, which will report real serial numbers, you can download the free version of Hard Disk Sentinel and that will also report real serial numbers. I don't know if that will help you locate and identify your drives in your enclosures, but if you pull a drive and put it into a USB disk caddy, you would be able to identify each drive that way.

I have Hard Disk Sentinel. I also spent days & weeks searching & reading the topic. Some gave scripts samples. Others gave links to Microsoft to read, and of course, to get more confusion. At the end, it is something like your suggestion would be the answer. I plan to systematically pull a drive out, run 'Get-Disk' command, record & label each drive. Many ways to skin a cat, your way & my way would be at 'down-the-earth' level. Microsoft would be laughing at us, idiot basterds.

16 hours ago, gtaus said:

DON'T run scandisk with repair on a HDD until after you have transferred as much data off of it as possible.

I got to remember this. Thanks

16 hours ago, gtaus said:

I am surprised to hear that DrivePool is taking up so much memory. I just checked my server's Task Manager and my DrivePool is reported as using only 40MB of memory. I have 8GB RAM, so no issue for me on my 6+ years old HP desktop running Windows 10.

I misplaced the notes when I had to nuke DP, but it was somewhere in the 30 - 40 % memory usage. Media streaming was terrible. I knew something was wrong, and DP was the last app adding to the Server. After uninstalling DP, all were well. Then, what is the best options moving forward? Continuing with Storage Spaces requires advanced knowledge. The only alternative is DP. Second attempt at DP gave me lot of hopes, especially with quick 'down-the-earth' support from you, this forum and Christopher.

Media streaming returned to normal under second DP installation. This time, I got 4 x 4 TB hard drives + 2 x 250 GB SSD formatted with NTFS + 16 GB Allocation. Then, I installed DP (the Beta version from Christopher) + DrivePool.AllInOnePlugin.Setup_1.0.0.2_x64.msi. Next was the settings of DP.  Very smooth & quick.

16 hours ago, gtaus said:

I am still relatively new to DrivePool (less than a year), so I am reading posts here all the time trying to learn more. I try to give back if I am able. @Shane , a moderator, has been very helpful on many of my questions. Responses are always appreciated.

Glad to have your help

 

P.S. Removing drive letters & naming drives to be DP01_PB3, DB02_PB3, etc . . . worked. The new names give information about drives and its enclosure.

Link to comment
Share on other sites

  • 0

I made a BALANCING' change in 'DrivePool.OrderedFilePlacementPlugin' from 'new files only' to 'including existing files', or something like that.

A long yellow bar appeared at the bottom of the screen. I clicked on 'Re-Balance' button on the side of the bar. Some activities appeared briefly, then the bar changed to green and Pool Organization label. It remained in this state for 4 - 5 hours now. Cancelletion option not visible anywhere.

What would be appropriate next steps ?

Thanks

Stablebit.Re-Balance.jpg

Link to comment
Share on other sites

  • 0
13 hours ago, TPham said:

Second attempt at DP gave me lot of hopes

I don't know why you saw so much memory usage on your first install of DrivePool. I have never seen anything like that. Is it possible that DrivePool was performing some intensive task at that moment? I am glad that Christopher was able to get you back on track and that things are going much better (normal) now.

5 hours ago, TPham said:

A long yellow bar appeared at the bottom of the screen. I clicked on 'Re-Balance' button on the side of the bar. Some activities appeared briefly, then the bar changed to green and Pool Organization label. It remained in this state for 4 - 5 hours now. Cancelletion option not visible anywhere.

What would be appropriate next steps ?

A long yellow bar usually means DrivePool activity or an error state - like duplication needs to be re-checked. The green bar usually means everything is OK with DrivePool and does not need any attention. If you have a green bar, then I don't understand why you would be looking for a cancellation option. Is there some task running in the background? Your screenshot of DrivePool looks all good to me.

Speaking of background tasks, I have suggested that if would be nice to have an option to select that more information be displayed of what is going on in the background tasks running in DrivePool. Many of those tasks and status on them seems to be hidden, maybe for a good reason. Most of the time, I'm just happy to let DrivePool do its thing in the background and I don't need to see the inner workings. But, it would be good to have an option to get more info and status on those tasks for those few times I do care to understand what is going on behind the curtain.

Under normal circumstances, DrivePool pretty much takes care of itself. However, I have noticed a few times that it appeared to me that a task like balancing might not be behaving properly. Don't know if it was a DrivePool issue, or a Windows issue, but I rebooted the computer and everything went back to "normal." That's pretty much my first step to take if I notice something fishy going on with my computer. Again, I don't know if it's a DrivePool issue, or a Windows issue. As you know, Windows can lock itself up for unknown reasons and a fresh reboot seems to correct the OS. And that certainly has nothing to do with DrivePool.

I would also suggest keeping an informal log (note on piece of paper) of what was wrong and how you corrected it. If you see a pattern of misbehavior, you could share that with Christopher and maybe he could look for a bug to correct. One thing I appreciate with DrivePool is that the programmer(s) are looking for ways to improve the software and they issue new releases. 

Link to comment
Share on other sites

  • 0
On 1/15/2021 at 7:39 PM, gtaus said:

A long yellow bar usually means DrivePool activity or an error state - like duplication needs to be re-checked. The green bar usually means everything is OK with DrivePool and does not need any attention. If you have a green bar, then I don't understand why you would be looking for a cancellation option. Is there some task running in the background? Your screenshot of DrivePool looks all good to me.

It used to be a transparent bar, I think. So after the balancing activity which was yellow, the bar became green. Therefore, I am not sure if this is normal, or some i/o is still going on.

On 1/15/2021 at 7:39 PM, gtaus said:

Speaking of background tasks, I have suggested that if would be nice to have an option to select that more information be displayed of what is going on in the background tasks running in DrivePool. Many of those tasks and status on them seems to be hidden, maybe for a good reason. Most of the time, I'm just happy to let DrivePool do its thing in the background and I don't need to see the inner workings. But, it would be good to have an option to get more info and status on those tasks for those few times I do care to understand what is going on behind the curtain.

Good point. In my case, it is a Server for storage and streaming. DrivePool is struggling with the latter over wi-fi network. I'd like to know a bit more about background process as well. Just like those balancing, the bar went from transparent, to yellow then stayed green. Should I worry about, or just disregard. Weird. So I did a comparison between DrivePool and Storage Spaces. Storage Spaces with Parity is REAL bad in writing, but its streaming capability is far superior to DrivePool. That is why we never experienced with streaming until I implemented DrivePool.

I tried PrimoCache as you suggested to see if streaming improved. Initial set up with Level-1 using RAM memory. It is helpful ONLY when writing tens or hundreds GBs. No need for a few random files now and then. Added Leven-2 cach with an SSD showed no change.

Actual writing to Storage Spaces was around 40 mb/s. With PrimoCache, the initial burst over 110, but quickly dropped down to between 40 - 60 mb/s. Actual writing to DrivePool (2 x SSDs for caching) is around 100 - 110 mb/s, with or without PrimoCache. PrimoCache is not likely to help my case. By the way, PrimoCache Server Ed. is quite expensive, whether or not it's use is for home or commercial.

Look like I can't give up Storage Spaces, as yet. Until I can find a way to improve streaming.

Picture 1: Read & Write capability to Stablebit DrivePool

Read.Write.Stablebit DrivePool.jpg

Picture 2: Read & Write capability to Storage Spaces

Read.Write.Storage Spaces.jpg

Picture 3: Write capability with PrimoCache From DrivePool to Storage Spaces. Initial Burst Speed

Write with PrimoCache Level-1 (initial burst.jpg

Picture 4: Write capability with PrimoCache From DrivePool to Storage Spaces.Soon Dropped to Normal Speed

Write with PrimoCache Level-1 (stable-out).jpg

Edited by TPham
Added clarification to the pictures.
Link to comment
Share on other sites

  • 0

If the activity bar is green and not moving, the DrivePool service is not performing any scheduled/requested tasks such as remeasures, balancing runs, consistency/duplication checks, etc. (This is not the same as whether other apps are reading/writing the pool - that's indicated by the Performance section).

The fact is DrivePool's read performance simply can't match Storage Spaces because the former merges the logical volumes file-by-file while the latter stripes the physical disks block-by-block; you just cannot beat that kind of "bare metal" approach for raw read speed and low latency, though DrivePool tries its best. The tradeoff as you can see with your own benchmarks is write speed (as well as reliability; if Storage Spaces goes bad, it tends to REALLY go bad).

All that said:

You're still getting pretty good read speeds out of DrivePool? If you're streaming music or video, most reads should be sequential anyway (assuming that your disk data isn't fragmented*; you might want to check the individual drives in the pool with a utility like UltraDefrag, which is also capable of defragging the MFT of your volumes) and your sequential reads are multiple times faster than needed to feed even wired gigabit while your random 4K Q1T1 is still managing 22.46 MB/s which should be enough for 4K video streaming.

Which if you're still getting problems with (e.g.) stuttering in your streams... leads me to think the problem is latency/buffering, whether DrivePool's (which can't "cheat" like Storage Spaces can), your drives (if fragmented as above) or that of your streamer and/or wifi and/or receiver.

On DrivePool's end, try Manage Pool -> Performance and ticking "Network I/O boost" and/or "Bypass file system filters" to see if that makes any difference to your situation (please read: link to documentation). For your streamer and receiver, see if they have options to increase their buffering (if you're using VLC, there's a bunch of advanced settings, some may do nothing, some may help a lot). For your wifi network, see if you get better results on a different channel or frequency (if your wifi is fancy enough, it may even offer QoS options for streaming), or whether it's being limited by less-capable devices sharing the connection.

LATE EDIT: *Note: SSDs shouldn't be defragged (though it may be worth using the manufacturer's diagnostic software to check if they need anything else done). However, sometimes they can suffer significant performance hits if their free space is too low (anywhere from 10% to 25% depending on model).

Link to comment
Share on other sites

  • 0
11 hours ago, TPham said:

In my case, it is a Server for storage and streaming. DrivePool is struggling with the latter over wi-fi network.

I don't understand how DrivePool read speed can test out 441MB/s and this is causing a problem. When I stream from DrivePool to my Amazon Fire TV stick, the streaming rate on the DrivePool GUI is at about 4MB/s and I don't get any stuttering or caching of the movies up to 1080p. What kind of speed do you need on your system for streaming without problems?

11 hours ago, TPham said:

I tried PrimoCache as you suggested to see if streaming improved. Initial set up with Level-1 using RAM memory. It is helpful ONLY when writing tens or hundreds GBs. No need for a few random files now and then. Added Leven-2 cach with an SSD showed no change.

I guess that makes sense. If you have 14GB of RAM cache set aside in PrimoCache, then you would see an initial burst of speed and then it would drop significantly after the 14GB RAM cache fills up. When I tried PrimoCache, years ago, I only had enough memory to set aside 4GB RAM. So, small files transferred to the buffer very fast, but large transfers over 4GB quickly slowed down to the write speed of the destination drive. For me, saving 20 seconds on a large file transfer was not worth the cost of the program.

As to PrimoCache and streaming, maybe there is no benefit to using PrimoCache considering my above example where my Amazon Fire TV Stick is only needing about 4MB/s streaming speed. At that rate, you would never even fill your Level 1 RAM cache. Adding an SSD as Level 2 cache for streaming would not provide any additional benefit if your destination device is only requiring 4MB/s, like my TV Stick.

11 hours ago, TPham said:

Actual writing to Storage Spaces was around 40 mb/s. With PrimoCache, the initial burst over 110, but quickly dropped down to between 40 - 60 mb/s. Actual writing to DrivePool (2 x SSDs for caching) is around 100 - 110 mb/s, with or without PrimoCache.

I only have 1 SSD drive on the front end of my DrivePool, but, like you, I get faster writes with DrivePool+SSD then I used to see with Storage Spaces. You probably don't see any benefit to PrimoCache because after filling up your 14GB Level 1 RAM cache, you drop down to the speed of the SSDs. I guess if you routinely transfer less than 14GB of data, then PrimoCache would speed things up. Usually, when I transfer data, it's a lot more than 14GB so PrimoCache would not offer me much time saved. Which is why I have not purchased PrimoCache.

11 hours ago, TPham said:

PrimoCache Server Ed. is quite expensive, whether or not it's use is for home or commercial.

I was thinking of the $29 version. But if the software does not provide you any additional benefit, then it's just not worth it. At least they let you try it before you buy it.

11 hours ago, TPham said:

Look like I can't give up Storage Spaces, as yet. Until I can find a way to improve streaming.

When Storage Spaces works, it works great. After running Storage Space for ~7 years, I was spending more time in PowerShell trying to manually fix problems with my Storage Spaces then I was actually using it. So I made the move to DrivePool and my life is better.

I don't understand why you are having issues with streaming from DrivePool. Back in the day, I was having issues streaming from Storage Spaces to my Fire TV Stick, but I upgraded my wifi router to 1 gig service and that took care of those issues. I don't have any problem streaming movies from DrivePool. I suspect your bottleneck is somewhere else in your chain. Again, my streaming device only pulls down about 4MB/s and DrivePool has no problem with that.

Here is a simple test you could try. Dump some files into a DrivePool 2X folder, turn on Read Striping, and see if that somehow makes a difference in your streaming experience. Also, watch your DrivePool GUI Performance Read/Write graph to get an idea of how fast your streaming device is pulling data. Your DrivePool CrystalDiskMark read speed test was 480MB/s, and I cannot imagine any streaming device needing anything close to that speed. 

Link to comment
Share on other sites

  • 0

@Shane Thanks. Great advises. I did some of the trouble-shooting tips you mentioned, as well as tried the options "Network I/O boost" and "Bypass file system filters". For the time being, I maintain both systems, and continue to monitoring DrivePool. One weird thing just happened though. Last Friday, I made a BALANCING' change in 'DrivePool.OrderedFilePlacementPlugin' from 'new files only' to 'including existing files'. Immediately afterward, the long bar turned yellow then to green and stayed there (as mentioned is earlier post). No activity whatsoever till this morning, after I activated DrivePool.  Suddenly DrivePool started writing & writing non-stop. It has been going on for a while now. The green bar stayed as it was before. Strange.

12 hours ago, gtaus said:

Here is a simple test you could try. Dump some files into a DrivePool 2X folder, turn on Read Striping, and see if that somehow makes a difference in your streaming experience. Also, watch your DrivePool GUI Performance Read/Write graph to get an idea of how fast your streaming device is pulling data. Your DrivePool CrystalDiskMark read speed test was 480MB/s, and I cannot imagine any streaming device needing anything close to that speed. 

I did the read stripping and the other options in performance tab since day 1. The drives in DrivePool are new, the server is powerful with Xeon processor and now 32 GB of ram, Lan & wi-fi speeds are great, and the receivers for streaming have more than sufficient power. The server is pure, no third-party app, only windows defender. So, I'm baffled about the poor streaming ability of DrivePool. I like the app and really want it to work.

Although CrystalDiskMark read/write speed tests in the 400 mbps, DrivePool showed an average around 100. This is still good. Until I can get to the bottom of this, Storage Spaces will be for streaming and DrivePool for archiving. My life became sooooo complicated now.

As for PrimoCache, we are on the same page. I reached out to their support staff on the performance and pricing. They suggested using 'delay-write' option, which allows the cache to build up. This option improved the initial burst, but dropped down even worse afterward. As for the pricing, server edition is $119. Eric Chen offered 10 % discount as my server is stand-alone and of no commercial use.

Well more later. I enjoyed reading yours, but last night I was up all night replacing the hard drive in my laptop.

Cheers,

Link to comment
Share on other sites

  • 0
6 hours ago, TPham said:

No activity whatsoever till this morning, after I activated DrivePool.  Suddenly DrivePool started writing & writing non-stop. It has been going on for a while now. The green bar stayed as it was before. Strange.

@Christopher (Drashna) is this a bug, or are there situations (other than "real-time" placement) where the Pool Organization bar not reacting to (presumed) balancer activity is considered working as intended?

Link to comment
Share on other sites

  • 0
10 hours ago, TPham said:

I did the read stripping and the other options in performance tab since day 1. The drives in DrivePool are new, the server is powerful with Xeon processor and now 32 GB of ram, Lan & wi-fi speeds are great, and the receivers for streaming have more than sufficient power. The server is pure, no third-party app, only windows defender. So, I'm baffled about the poor streaming ability of DrivePool. I like the app and really want it to work.

Yeah, it sounds like your system is magnitudes better than my ~8 year old HP desktop with a low powered AMD chip and 8GB RAM. But, DrivePool has no problem streaming my movies. I just don't understand why it's not working for you on a much better system.

10 hours ago, TPham said:

Until I can get to the bottom of this, Storage Spaces will be for streaming and DrivePool for archiving. My life became sooooo complicated now.

That's too bad. I hope you get the tech help you need with DrivePool. Life should not be so complicated.

As I said, I ran Storage Spaces for ~7 years and when it works, it is great. But when I had problems, Storage Spaces was a disaster and there was no help from the MS community. I have had similar HDD failures using DrivePool, but the difference is that I have been able to recover almost all my data on the failing DrivePool drives. In the meantime, DrivePool continued to work just fine with all the other drives in the pool, so I was never really down in that respect.

I hope you continue to update this thread because I really believe that your problem is most likely somewhere other than DrivePool. It would be great to see if you are able to resolve this issue and let us know what is was causing the problems. Best wishes.

Link to comment
Share on other sites

  • 0
On 1/20/2021 at 12:45 AM, gtaus said:

As I said, I ran Storage Spaces for ~7 years and when it works, it is great.

Same here. I've been using it since Server 2012 was released. Since then I went thru 2 upgrades, 2016 & 2019. Never a problem. I anticipated that some of the hard drives could reach the end of their life soon and will need replacement. If I know how to identify the drive when the time comes, I would move on without a second thought. But reading some bad experiences by others, I wanted to have a solid alternative ready. That is why I am here. Thanks for your continuing support.

Continuing with DrivePool streaming improvement, I spent much much hours in past few days analyzing and testings, as I do want DrivePool to work. I have a list of many more things to do. One of them relates to wi-fi setting and QOS (as Shane mentioned in an earlier post).

I have fiber optic with AT&T. The gateway/router combo is a Pace 5268A. I have an Asus router set up in AP mode which controls all wireless connections. The main receiver is an i5 and is within 25 ft direct sight to the Asus. Signal strength is excellent every corner of the house. Streaming has always been good. The main receiver and other laptops, phones and ipads always have 5 or 4 solid bars.

Pace 5269 gateway/router combo has no setting for QOS and/or any traffic control as far as I know. It can be set up as a bridge, but the speed will drop by haft and ip6 will stop. There is a way to copy its mac address and replace it with the Asus router. By doing this, I can set the Asus in router mode and utilize all its settings. However, this option is too extreme. AT&T certainly would not approve it. I am not ready for it yet, too much exposures. So, let's leave the Pace alone for now.
 
The AP mode in Asus has no support for QOS. It is running the latest Merlin firmware. Chances are there is other firmware that provides something like adaptive QOS, but I can't find one and/or if it would work. Perhaps it could be setup as a second router but proof it as an AP. I heard about but don't know how. There could be other alternatives. So, this is question #1.

I can have access to a spare Asus router and get it configured in repeater or any mode that supports QOS. Is this possible and would it help any, at least to the main receiver? This is question #2.

There was a notification for new 2.3.0.1244 release. I downloaded it and later found out it is a Beta. Any possible impact to my streaming and/or other functionalities? This question #3.

So, I would like to address these three questions first, prior to planning any thing else. For the time being, streaming from Storage Paces is our only choice.

Cheers,

 

Link to comment
Share on other sites

  • 0
6 hours ago, TPham said:

Continuing with DrivePool streaming improvement, I spent much much hours in past few days analyzing and testings

Have you determined what speed your TV streaming device pulls movies from your Storage Spaces or DrivePool? For example, when I watch my DrivePool GUI, I can see that my Fire TV Stick is pulling about ~4 MB/s tops for streaming 1080p movies. I don't suffer any stuttering or caching on my system. If I try to stream movies >16GB, then I start to see problems and caching issues. But, at that point, I know I have reached the limits of my Fire TV Stick with limited memory storage and its low power processor. It is not a limit of how fast DrivePool can send data over my wifi.

6 hours ago, TPham said:

The main receiver and other laptops, phones and ipads always have 5 or 4 solid bars.

Well, there is how many bars are available to indicate how strong the connection is, but bars does not equal speed. On my old 56K router, I would also have 4 or 5 bars indicating a strong connection, but I was constantly fighting buffering issues while streaming. I upgraded to a 1 gigabit router, which is much faster, and that took care of my buffering problems.

6 hours ago, TPham said:

So, I would like to address these three questions first, prior to planning any thing else.

Well, good questions but beyond my level of tech expertise with that equipment. I get my internet service from a local telephone company, and they have a computer support team on staff to answer questions and help customers with their equipment. If you are leasing your equipment from ATT, then they might have a support team you could contact for assistance.

 

6 hours ago, TPham said:

For the time being, streaming from Storage Spaces is our only choice.

At least you have something that is currently working for you, so it's not like you are in a panic. After years of running Storage Spaces on my system, and now with DrivePool for just less than 1 year, I don't yet understand why you are experiencing streaming issues with DrivePool. On my system, it made no difference at all in regards to streaming, which I have stated runs at about 4 MB/s tops and usually much less on my system.

Link to comment
Share on other sites

  • 0
1 hour ago, gtaus said:

when I watch my DrivePool GUI, I can see that my Fire TV Stick is pulling about ~4 MB/s tops for streaming 1080p movies. I don't suffer any stuttering or caching on my system. If I try to stream movies >16GB, then I start to see problems and caching issues. But, at that point, I know I have reached the limits of my Fire TV Stick with limited memory storage and its low power processor. It is not a limit of how fast DrivePool can send data over my wifi.

Ha!Ha!Ha! Nice trick. I did not know and did not even think about that.

Playing an 1080P from DrivePool to the main i5 receiver, DP reads 1000 kbps (just barely over 1 MB/s) for 1 second, then 0 for the next 2 - 3 seconds. Cycle repeats with some variations such as 500 or 700 kbps for 1 second, then 1 - 2 seconds at 0. Is this the issue ? The receiver is the root cause ?

Is there a way see how it reads in Storage Spaces for comparison ?

I have a firetv which my wife hated. I got kodi on it but never knew that it could stream from a network folder. How you did it? Complicated ? Give me some lead and I'll try.

1 hour ago, gtaus said:

I get my internet service from a local telephone company, and they have a computer support team on staff to answer questions and help customers with their equipment.

Lucky for you. Dealing with large company is a venture. Sometimes I got lucky. Most were disaster. Get them to fix something for free may still end up with a big charge. Better fix it myself.

Link to comment
Share on other sites

  • 0

A1: I'm not familiar with the Pace 5268A (or any other AT&T routers), sorry. Different country!

In theory, generally, it should be possible to have router "A" handling internet traffic while router "B" handles local traffic without them fighting about it; in practice some routers are (much) less capable than others and some ISPs also lock down the functionality of routers they provide / mandate.

For whatever it's worth, here's an example very basic setup:

  • Internet <-> [WAN port] Router A [LAN port] <-> [LAN port] Router B [LAN port(s) and Wifi] <-> Local devices
  • Router A (WAN DHCP on + LAN DHCP off) is configured to receive the necessary WAN addresses from the ISP but to not manage the local devices.
  • Router B (WAN DHCP off + LAN DHCP on) is configured to deliver the necessary IP configurations so local devices use Router A as the internet gateway.

Obviously this is a simplified summary and you'd need to get all the fiddly parts lined up, but that's my day job not my forum job. ^_^

A2: As a general rule of thumb you want to minimise the number of wifi devices (including stations) operating in any given area, and another is that you want all devices operating on any given channel to support the same wifi standards (e.g. if a gen 1 device and a gen 2 device are sharing the same channel, you may end up with the gen 2 device limited to gen 1 capability). That said, if your plan is to have Router A handling wired traffic while Router B handles wireless traffic, that might work for you.

A3: The new Beta functionality mostly relates to supporting Stablebit Cloud connectivity, but the changelog indicates there was work done on DrivePool functionality too. The great thing is you can install the Beta and if it doesn't help you can uninstall it and reinstall the Stable release without losing anything.

23 hours ago, TPham said:

Is there a way see how it reads in Storage Spaces for comparison ?

Resource Manager is built into Windows and can be used to monitor a variety of metrics including file access/throughput.

Link to comment
Share on other sites

  • 0
7 hours ago, TPham said:

Playing an 1080P from DrivePool to the main i5 receiver, DP reads 1000 kbps (just barely over 1 MB/s) for 1 second, then 0 for the next 2 - 3 seconds. Cycle repeats with some variations such as 500 or 700 kbps for 1 second, then 1 - 2 seconds at 0. Is this the issue ? The receiver is the root cause ?

I just got done watching a 720p movie and I don't think my stream ever got over 1 MB/s. When I said the 1080p stream was 4 MB/s tops, that is maybe a short burst at the start of the file which I assume is filling the Fire TV Stick's onboard memory cache and then, like you noticed, it drops below 1 MB/s. Since you see more or less the same streaming transfer rate that I do on my system, maybe you can see why I am always saying that I doubt if your problem with streaming is coming from DrivePool.

If I copy files to/from a remote computer to/from my DrivePool computer on my home network, I can reach 80 MB/s transfer rate. So, clearly, it's not DrivePool that is slowing down my system.

7 hours ago, TPham said:

Is there a way see how it reads in Storage Spaces for comparison ?

@Shane already suggested using the Windows Resource Monitor, which I access via the Task Manager. You should be able to view the data streaming from your Storage Spaces drive volume.

7 hours ago, TPham said:

I have a firetv which my wife hated. I got kodi on it but never knew that it could stream from a network folder. How you did it? Complicated ? Give me some lead and I'll try.

I use Kodi as my main media interface. If your DrivePool computer is on your network, it probably already has network sharing set on it. If not, turn on network sharing for your DrivePool drive (J: on my computer). Then, in Kodi, you need to add the media folder by going to the browse network command. Is it complicated? Well, it's a bit more complicated than Plex to setup, but I just like the Kodi interface better than Plex so I was willing to put a little more effort into the project.  I had to go online and search for Kodi tutorials on how to setup a network folder for Kodi, but following the directions, I got it to work without too much difficulty.

Just another thought comes to mind... I think you once mentioned that you were using IP6 protocol for wifi. IIRC, I had all kinds of problems with IP6 and shut it off so I only use IP4. At the time, there were many people complaining about the IP6 protocol not working correctly. It might have been too new and the IP6 standards were not firmly set. Anyway, going back to IP4 worked for me and I have not changed it since (if it ain't broke, don't fix it). If you are using IP6, you might want to turn that off and see if the older, more stable, IP4 works better for you. It could be that your devices are trying to communicate via different versions of IP6 and they have a hard time talking, which would lead to dropouts. Anyway, just a thought.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...