DrivePool fast write speeds 450MB/s, but slower read speeds 250MB/s for multiple files/drives?

[techguy1]

Hi, new to drivepool, I recently setup an 8 bay DAS that is usb 3.0 (5Gbps) and I am using 5x 18TB Seagate Exos currently.

I tested out writing multiple files to my Pool Drive so that windows would have concurrent write streams going. I was able to max out at 450 MB/s which I was quite happy with.

However when reading multiple files from the drive pool, I only seem to get 250-270MB/s which is the read speed of 1 drive.

I would expect if I read file A from drive A and file B from drive B that I should cap out around ~450-500 MB/s just like I did with the write speed test.

To make sure, I also tested going into drive 1 and drive 2 directly via the hidden PoolPart folder to directly pull File A and File B off 2 different drives at the same time.

Windows shows 2 read operations going simultaneously and ... my speed caps out at 250-270 MB/s no matter how many different drives I read off of.

I am writing to an SN850X NVME SSD which has a write speed limit around 5000 MB/s (tested in CrystalDiskMark).

Any idea what I may be doing wrong? Initially I was only getting 50 MB/s from reading from the drive pool drive and realized smart scanner was trying to scan at the same time, reducing performance. After pausing smart scanner, I am in the scenario described above, where no matter how many different drives I try to read from that are in the drive pool, I can only read as fast as 250-270 MB/s.

My drive pool has bypass file system filters, read striping, and real time duplication enabled in performance setings. However I am not using file or folder duplication at all at the moment (I plan to use snap raid once I feel comfortable with drive pool).

Any help appreciated, thanks!

 

[techguy1]

Another Update:

I transferred drives from the SYBA to the Yotta to test out the Yotta speeds with all 5 bays occupied by Seagate Exos drives.

I am getting just slightly above 1000 MB/s when doing concurrent reads and writes to/from the drive pool! The theoretical max would be 1250 MB/s (10Gbps) on USB 3.1 so I am extremely happy with this.

[Shane]

4 hours ago, techguy1 said:

To make sure, I also tested going into drive 1 and drive 2 directly via the hidden PoolPart folder to directly pull File A and File B off 2 different drives at the same time.

Windows shows 2 read operations going simultaneously and ... my speed caps out at 250-270 MB/s no matter how many different drives I read off of.

If you have drives X and Y comprising pool Z, then directly accessing X:\PoolPart or Y:\PoolPart yourself rather than through Z:\ means you're not using DrivePool - so if you're still getting the same capped total speed then the problem isn't DrivePool.

You might need to look at the USB 3.x controller you're using, perhaps there is an issue there / a driver update available? What is the brand/model of the DAS?

[techguy1]

3 hours ago, Shane said:

If you have drives X and Y comprising pool Z, then directly accessing X:\PoolPart or Y:\PoolPart yourself rather than through Z:\ means you're not using DrivePool - so if you're still getting the same capped total speed then the problem isn't DrivePool.

You might need to look at the USB 3.x controller you're using, perhaps there is an issue there / a driver update available? What is the brand/model of the DAS?

That makes sense, so it could be an issue with the hdd enclosure -- I am using a Syba 8 bay hdd enclosure SY-ENC50119 (https://www.sybausa.com/index.php?route=product/product&product_id=1001) The specs say it has a "ASMedia ASM1074L and ASM1153E Chipset".

I tested going from an external USB-C nvme SSD to the internal SSD and I get about 750 MB/s.

The Syba DAS is connected via usb 3.0 --> to a usb type a 5Gbps port on the same computer.

Since Syba DAS --> usb type a port on pc gets 450 MB/s write speed, I assume this means that the pc usb type a port & controller is probably fine.

So then it must be the Syba DAS? Maybe the ASMedia ASM1074L and ASM1153E chipset has faster write speed than read speed, or only allows parallel drive access on writes but not reads? Seems weird, but is that the most likely problem here? I have ordered a different multi hdd enclosure with different chipset to test this out, but will be a few weeks before it gets here.

[VapechiK]

ok, just curious... first of all... (per the last paragraph of your first post), why are read striping and real time duplication enabled on your pool if duplication (as inferred) is not enabled?  seems to me your settings are complicating things.  kill them both (disable) for starters.

i too have a SYBA 8bay exactly like yours.  it is where my DrivePool lives.

i also have an Oyen Mobius Pro 5C where my backups live.  writing/reading speed from this enclosure can vary greatly (70MBps from the Oyen to the SYBA... 160MBps from the SYBA to the Oyen).  and they're both attached to the same USB3 adapter card.  *shrugs* go figure.

if you are actually seeing the speeds you mentioned, consider yourself lucky.  18+ TB and 7200 rpm speed = R/W speed i wish i had.  all of my drives max out at 8TB 128MB cache, so i have to be happy with what i have.

cheers

[Shane]

If the SSD was tested on the same port on the PC as the Syba then that would rule out the PC (at least in theory; in practice it could still be a combination of the interaction between the particular chipsets), leaving only the Syba as the common factor. Otherwise while I too would suspect the Syba here, we couldn't entirely rule out the PC.

[RetroG]

USB so-called DAS/JBOD/etc units usually internally use a SATA Port multiplier setup, and is likely the source of your issues.

A SATA Port multiplier is a way of connecting multiple SATA Devices to one root port, and due to the way that the ATA Protocol works, when I/O is performed it essentially takes the entire bus that is created from that root port hostage until the requested data is returned.
it also is important to know that write caching will skew any write benchmarks results if the enclosure uses UASP or you have explicitly enabled it.

these devices perform even worse with a striping filesystem (like raidz, btrfs raid 5/6 or mdraid 5/6), and having highly fragmented data (which will cause a bunch of seek commands that again, will hold the bus hostage until they complete, which with spinning media does create a substantial I/O burden)


honestly, your options are either accept the loss of performance (it is tolerable but noticeable on a 4 drive unit, no idea how it is on your 8 drive unit), or invest in something like a SAS JBOD which will actually have sane real world performance. USB isn't meant for more than a disk or two, and between things like this and hidden overhead (like the 8/10b modulation, root port and chip bottlenecks, general inability to pass SMART and other data, USB disconnects, and other issues that aren't worth getting into) it may be worth just using a more capable solution

[VapechiK]

yes i follow (kinda lol) ...

i for one hate that Windows will LIE TO YOUR FACE when/if write caching is enabled (easy enough to see, just open Task Manager and notice that whatever I/O operation is STILL continuing long after whatever transfer/progress box closes).

so i recommend killing write caching on ALL external USB enclosures.  better to see EXACTLY what is happening and WHEN than OOPS did i just delete that and it wasn't done yet?  yes i have learned from experience lol...

i have ZERO experience with SAS so i'll just keep quiet there...    thanks for the deets.

[everydayz]

Your setup sounds beefy, by the way! Nice choice on the SN850X NVME SSD. But yeah, your read speeds should definitely be better.

[techguy1]

On 1/24/2024 at 1:53 PM, everydayz said:

Your setup sounds beefy, by the way! Nice choice on the SN850X NVME SSD. But yeah, your read speeds should definitely be better.

Thanks 🙂 !

I just tested out a Yottamaster 5 bay FS5C3 with 2x Seagate Exos X18 18TB drives (pulled 2 drives out of the Syba enclosure to test out).

Write speeds achieved slightly above 500 MB/s which is excellent, as each individual drive I tend to get about 270 MB/s per CrystalDiskMark benchmarks.

I tested out read speeds by copying multiple files to this new drive pool and then reading off it with 2 separate windows read operations (basically, pull files off the drive pool in 2 distinct copies going on simultaneously.

534 MB/s maximum read speed achieved with 2 drives! Mostly oscillating between 510-534MB/s! SMART data can also be read from this enclosure.

I think the Yottamaster is a winner here 😄

I think I will slowly repurpose the SYBA for cheaper / slower drives I may acquire or for some of the parity drives for snap raid.

I will use the Yottamaster for my data drives for the speed boost.

Anyways I hope all this info helps someone else trying to figure out what DAS (disk attached storage) enclosure they might want to use with Windows 11 Pro / Stablebit DrivePool / snapRAID.

Now to figure out how to get WSL (Windows Subsystem for Linux) up and running and hopefully be able to use badblocks to thoroughly stress test the whole system 😅