Jump to content
Covecube Inc.
  • 0

Poor write speed and poor balancing speed


Question

I have a pool of 15 disks, each with a read write speed of 150MB/s. No duplication. I can test and each drive does perform very close to it's rated speed. when transferring files to array over SMB my write speed is 62MB/s. I have added three SSD for cache and I am writing to them at 225MB/s over SMB multipath. but when balancer runs I only get 40MB/s unless I increase priority then it jumps up to 60MB/s. so SSD Optimizer has actually slowed my workflow down as it's limited to 40MB/s unless I increase priority. I have changed the setting in config.json for it's priority but it's still only 60MB/s. also why is it only emptying one SSD at a time? I end up with all three SSD's full and waiting on one to empty sequentially when they could all be dumping to pool drives at the same time. my workflow is transferring a about half a dozen TB a day and this is bottlenecking me. Please help! Thanks

Link to post
Share on other sites

9 answers to this question

Recommended Posts

  • 0

For me a pair of 12tb got me 400mb/sec which is decent enough for me to dump the ssd cache (i found it killed the speed). 15 disks in a pool should blow away 4 ssd drives in straight line speed.

I write 1TB/hour doing a tape restore.

copy.jpg

Link to post
Share on other sites
  • 0
2 hours ago, fluffybunnyuk said:

For me a pair of 12tb got me 400mb/sec which is decent enough for me to dump the ssd cache (i found it killed the speed). 15 disks in a pool should blow away 4 ssd drives in straight line speed.

I write 1TB/hour doing a tape restore.

copy.jpg

yes it should, and if I could get the 150MB/s windows gives me I would be happy. But I get 62 out of stablebit and that doesn't cut it. SSD has brought me up to wirespeed but hasn't fixed the transferring to drive issue.

Link to post
Share on other sites
  • 0
Posted (edited)

At 6TB/day regular workload i'd be recommending 6 or 8 drive hardware raid with nvme cache, with a jbod drive pool as mirror, if like me you want to avoid reaching for the tapes.

A 9460-8i is pricey (assuming using expander) but software pooling solutions dont really cut it with a medium to heavy load. Hardware raid is just fine so long as sata disks are avoided, sas disks have a good chance of correction if T10 is used. I've had the odd error once in a blue moon during a rebuild that'd have killed a sata array , whereas in the sas drive log it just goes in as a corrected read error, and carries on. I cant afford huge sas spinning rust, its very expensive.... so i have to compromise.... i run sas drive raid, and back it with large sata jbod in a pool. So its useful in a specific niche use case. In my case sustaining a tape drive write over several hours.

The big problem with drivepool is the balancing , and packet writes,  and fragmentation. All those millisecond delays add up fast.

What i've said isnt really any help. But what you wrote is exactly why i had to bite the bullet and suck it up , and go sas hardware raid.

Edited by fluffybunnyuk
got the model wrong
Link to post
Share on other sites
  • 0

It's just sad when my simple batch file can far outperform this product. I'm all SAS and was trying this out on a test bench before I moved it into production but it looks like it is not ready for prime time. There is no reason 15 drives with no duplication should not be able to keep up with a 1GbE!. one single SAS in my setup can do almost 150% of 1GbE but after drivepool I get about 50% of line rate.... redundancy is of no concern and a waste of drives for my current application which is why I am not going with standard RAID. balancing has to be multithreaded! If I were to put this in production on 10GbE with an NVME cache drive and an array of 120 drives my entire setup would come to a crawl waiting for it to transfer files one at a time at 60MB/s meanwhile the cache is filling up at 1100MB/s... I understand this isn't what this is intended for but my application do not want or need data redundancy.

Link to post
Share on other sites
  • 0

You might want to raise a ticket for support. AFAIK, this should not happen. When I copy a large file to my Pool (say 30GB), I get close to the 1GbE limit. Granted, it is a smaller Pool but also Hierarchical (so I would suffer any DP overhead roughly twice) and duplicated to two subpools, each 3 HDDs, no SSD Cache.

Link to post
Share on other sites
  • 0

8 drives in raid-6 gets me 1600mb/sec and saturates a 10gbe, it allows me to write that out again at 800mb/sec to my backing store (4 drives so less speed).

No write cache ssd in sight. Granted i'm not on the same scale as you, but i'm not far off with 24 bay boxes.

But then my network, and disk i/o is offloaded away from the cpu. With that many drives, ssd just sucks cos as soon as it bottlenecks it bottlenecks bad usually when it decides to do housekeeping.

I'd retest using 8 drives in hardware raid-0 (I/O offloaded), and use drivepool to balance to a pool of 4 drives. See what happens. I think the SSD with its constant cpu calling saying I/O is waiting is gimping your system.

Fun fact: A good way to hang a system is to daisy chain enough nvme to constantly keep the CPU in a permanent I/O wait state. :P

Link to post
Share on other sites
  • 0

yup, I do not want redundancy. waste of space for my workload. also I do not want the risk of losing too many drives to take out the whole array which is why I am here. I just want to be able to write to my drives as fast as windows natively can, not 40% the speed....

Link to post
Share on other sites
  • 0

Thats never going to happen with any software like drivepool. Theres a write read overhead(balancing) that multiplies with fragmentation.

My best recommendation is assuming your using large core xeons or threadrippers, is to open the settings.json. Change all the thread settings upto 1 per core, to allow all cores to scoop I/O from the SSD.

Give that a whirl. If that doesnt shift it. Try defragging the mft, and providing a good reserve so the balancing has a decent empty mft gap to write into.

My system reports the SSD balancer is slow as a dog which is why i uninstalled it. Its a cheap version of sas nvme support, except instead of  the controller handling the nvme I/O spam, the cpu does. (Which isnt a good idea the larger the system gets).

Have you tried using sym links, and using a pair of ssd per symlink so your folder structure is drives inside drives i.e. C:\mounts\mountdata(disk-ssd)\mountdocs(disk-ssd);C:\mounts\mountdata(disk-ssd)\mountarchive(disk-ssd)\;

So your data is transparently in one file structure but chunks are split over seperate drivepools. So the dump from the ssds are broadly in parallel?

I have to set aside 2 hours/day for housekeeping to keep drivepool in tip-top condition. Large writes rapidly degrade the system for me from the optimal 200mb/sec/drive+ at the moment its running at 80mb/sec/drive (which is the lowest i can afford to go) mostly because i've hammered the mft with writes, and theres a couple of tb i need to defragment.

I wish you all the best trying to resolve it.

Link to post
Share on other sites
  • 0

I'm already way above 1, also no duplication and no balancing (all disabled). changed about every setting I can in settings.json and the only thing i've managed to do is have balancing transfer at the boost rate when not boosted which is still 40% of the drives max. And yes my drives are mounted with mount points or symlinks. currently have 30 and I do not have the 60 ssd's needed to use two per symlink so that is out. also performance has not degraded from when this was a fresh setup on fresh drives so I wouldn't think the MFT would be the problem but I will give it a shot,

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...