Search the Community
Showing results for tags 'read striping'.
-
I am using StableBit DrivePool and noticed that I am getting file corruption that is somewhat reproducible. I particularly noticed this with FLAC files as I was attempting to verify my library of music using flac -t which checks the MD5 signature of the decoded WAV file. My setup is DrivePool with Folder Duplication enabled on specific folders, "Bypass filesystem Filters" checked, "Read striping" checked and "Real-time duplication checked". Note that it appears to be "Read striping" that is the culprit for this but I am not 100% sure. Particularly concerning to me is that this happens even with "Verify after copy" checked. Steps for me to reproduce are to download a FLAC to a DrivePool location that matches the above parameters, verify it with flac -t (ensure that it verifies OK), copy the file to a different location (doesn't even have to be a DrivePool location), verify this copied file with flac -t and see that the file does not verify anymore. Checking with a hex editor, I can see it's not even just 1 byte difference, usually something like the first 32kb is fine, then I get random jumbled up data for a 128kb or so then the remainder of the file is correct (and the file size is correct). Here's something I would never expect to see in a working filesystem: PS F:\test\> flac -t .\test.flac flac 1.3.2 Copyright (C) 2000-2009 Josh Coalson, 2011-2016 Xiph.Org Foundation flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. Type `flac' for details. test.flac: ok PS F:\test\> copy .\test.flac test2.flac PS F:\test\> flac -t .\test2.flac flac 1.3.2 Copyright (C) 2000-2009 Josh Coalson, 2011-2016 Xiph.Org Foundation flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. Type `flac' for details. test2.flac: *** Got error code 2:FLAC__STREAM_DECODER_ERROR_STATUS_FRAME_CRC_MISMATCH test2.flac: ERROR while decoding data state = FLAC__STREAM_DECODER_ABORTED Edit: It seems I am not the only one running into this problem, see this recent Reddit thread:
-
Hi I have 1x3TB WD RED + 1x 4TB WD RED, and with read striping enabled, drive pool insists on reading only from the slightly faster 4TB HDD. I'm copying a 6GB file to a RAMDisk, so that is not a bottleneck. I get around 160MB/S with striping enable/disabled, that's the max speed of the 4TB drive. During the transfer i can see for brief moments the 3TB hdd showing up in the performance tab, but it goes away almost instantly! I understand i wont get the total sum of read speed of both HDD, but i expected a improvement, even if only 10MB/s or something, at this time i don't see ANY improvement. OS Windows server 2016 Folder duplication is enabled 2x on the file i was testing.
-
Hi, I am currently evaluating DrivePool and mostly consider it because of its advertised read striping feature. However, there seems to be a bug or an existing bug (I have read related issues by other users dating back to 2015) has never been fixed. Here is my setup: I run on the latest Windows 10, 4 HDDs (WD Re 4TB, 170 Megabytes/second read performance for sequential data), which I pooled and chose 4x duplication. I copied several files and folders onto the pool and selected the following options: * Duplication -Enabled * Number Duplications: 4 ( I verified that the files and folders are in fact on each of the 4 hard drives - so DrivePool did its job at duplicating the data written into the pool) * Read Striping - Enabled * Realtime-Duplication - Enabled *Bypass file system filters - Enabled * File Protection - Pool File Duplication I ensured the data are in fact on each of the 4 drives. I then copied a single large file and also an entire directory, containing several multi GB large files to an NVME SSD drive (1.8 GB write performance so definitely no bottleneck here) with the same disappointing result: File transfer speed maxes out at around 150 Megabytes per second. Here some observations: * Initially I could see how only one drive was read from. But what points me to hint at this being bug (or at least a highly undesirable solution) is that DrivePool then jumped all over the place by reading for a second or two from 2 drives, then 2 others, then 3, then 1, then all four in a completely unpredictable manner. When >1 drives were read from the read performance on each individual drive most of the time dropped to <10 mb/second though 1 drive had larger throughput (not always the same drive). All in all, the read performance stayed relatively stable between 140-150 mb/second, but read striping is definitely not working as intended or desired on my end. With 4x duplication I expect at the very least 400 mb/sec throughput when reading identical data from 4 drives. Please note that all 4 HDDs are 100% identical brands/models/capacity. All are on 1 SATA 6 controller. I also eliminated the possibility of bottlenecks of my hardware components. I setup a Windows StorageSpaces Simple Virtual Disk with 4 columns (basically striping across all 4 HDDs) and I got a persistent read performance above 600 megabytes/second when reading from the virtual disk that comprised the 4 HDDs in Storage Space's Raid0 version. So, definitely no bottlenecks on my hardware/SATA controller/HDDs. What particularly worries me is the inconsistent jumping between different drives in the pool when reading in (read-striping enabled mode). Whether this is a bug in read-striping itself or a bug in the evaluation of the controller bandwidth or a an advanced setting that has to be enabled but is not by default I do not know, but I know that the current read-striping is not what is promised. Could some of you share whether this feature works for you and what throughput increase you see when you read from 2x, 3x, 4x, ... duplicated files in your pools? Also if the developer or support could chime in this would be awesome. I am urgently looking for a product that provides read-performance improvements and mirrored data resiliency at the same time. But I cannot consider this product for purchase with such currently abysmal read-striping performance. Any hints are welcome and maybe it is just me who overlooked a setting or feature that needs to be enabled but which I forgot to enable.
- 16 replies
-
- read striping
- parallelism
-
(and 1 more)
Tagged with: