Christopher (Drashna)

ok so I am redoing everything and shuffling stuff around. what has stayed is ...
Network... this is a beauty.. ive got $75 into.. HP Procurve 6400CL - 6 port CX4 port 10gb switch 5x ConnectX-1 CX4 port 10gb NIC running force fw to 2.9.1000 ConnectX-2 CX4 port 10gb NIC running Mellanox custom forced 2.10.xxxx fw!!!!! just got it and toying...I get that people say cx4 ports are old and dead but for $75 to be fully up for me is just the right price... then the hardware/software...
 
Case: Antec 900 
OS: Server 2019 Standard (Essentials role is gone.. im sad)
CPU: Intel i5-6600k
MoBo: Gigabyte GA-Z170XP-SLI 
RAM: 4x8gb ddr4 
GFX: Onboard Intel HD 530
PSU: Corsair HX 620W 
OS Drive: 128GB SSD, Samsung
Storage Controllers: 2x HP H220 SAS controllers flashed to current fw
Hot Swap Cages: ICY DOCK 6 x 2.5" SATA /SAS HDD/SSD Hot Swap 
Storage Pool1: SSD
Storage Pool2: Sata with 500GB SSD Cache
 
pics are garbage and I haven't moved it into my utility room...
 
 

There is ABSOLUTELY NO ISSUE using either the built in defragmentation software, or using 3rd party software.
 
Specifically, StableBit DrivePool stores the files on normal NTFS volumes. We don't do anything "weird" that would break defragmentation. At all.
 
In fact, I personally use PerfectDisk on my server (which has 12 HDDs), and have never had an issue with it.

Yeah i just figured it out myself, it didn't work even with registry imported,
This might take some time if i fill up all 60 drives someday..

thanks.. yeah I went back in and now I have it set for just I: as cache and rest are archive... so that now the drive is empty and seems to be functioning correctly where it is a straight copy without the dwindling speeds.... I added the SSD as a cache as you see I'm having copying file issues. Now that I have set this, I'm still having an issue but I believe it is my machine itself.
before you see slowness, now it copies at a full 450MB/s but another machine I have (plex) copies at 750MB/s. While it is totally faster from my plex box and funny how that works as the computer not copying as fast is the main rig that edits videos, photos, large iso copies, etc... so id want it faster there...
but still 450MB/s on 10gb is still faster than 120MB/s on my 1gb network!!! so while 4x faster.. not full speed. ive got a system issue.. because..
iperf shows super fast across the 10gb (and think iperf does memory to memory omitting hardware) so network is good.
my machine has 2x nvme on a quad pci-e 16x card that copying across each of them, they get 1.35GB/s.. its just exiting this machine... so more for me to test when I get time.

Wow yall got awesome setups!
I don't have a rack, nor do I want the sound of the rack servers.
what I have started using was a Lenovo ThinkStation Towers - dual xeon - 16 slots for memory!!!!! and now Lenovo P700 and P710's.
they are all quiet and can be pumped up on drives and ram and dual xeon's
ESXI 6.7 Machine 1 - 2x Xeon E5-2620 v4 @ 2.10GHz  - 64gb ram
ESXI 6.7 Machine 2 - 2x Xeon E5-2620 v0 @ 2.0GHZ - 128 GB Ram
ESXI 6.7 Machine 3 - 2x Xeon E5-2620 v4 @ 2.10GHz  - 64gb ram
FreeNAS 11.1 - 1x Xeon E5-2620 V3 - 24gb ram - 6x 2tb wd black (yeah I know reds not back but ive got them and they work.. hah)
Server 2016 / stablebit drive pool - HP Z420 - OS-128gb SSD / pool - 3x 2tb wd black + 2x 4tb wd black + 512gb ssd crucial for SSD Optimizer
Server 2016 is getting ready to gain 2 ( 6x2.5" hot swap bays) and filled with 12x 512gb crucial ssd running off 2x HP 220 SAS controllers
 
Network... this is a beauty.. ive got $75 into..
HP Procurve 6400CL - 6 port CX4 port 10gb switch
5x ConnectX-1 CX4 port 10gb NIC running HP FW 2.8
1x ConnectX-2 CX4 port 10gb NIC running Mellanox custom forced 2.10.xxxx fw!!!!! just got it and toying...
I get that people say cx4 ports are old and dead but for $75 to be fully up for me is just the right price...

@TeleFragger From the image, it looks like it's writing to the G:\ drive, which is not an SSD.  
So my guess is your settings are not configured correctly. 
If you could, open a ticket at https://stablebit.com/Contact

I just did, thank you!

Whatever you all do, don't wait 3 years and 8,000 movies (taking up 50TB), later to decide duplication would be a good idea. When I noticed my pool was getting full, it finally dawned on me I'd have a miserable time replacing lost movies if even one of the 15 WD40EFRX 4TB drives went south. Not only did it blast a hole in my wallet this week, to fill the remainder of my RPC-4224 case with 8x new WD80EFAX 8TB and 1x new WD100EFAX 10TB drive (experimental), it appears it will take a month of Sundays to get the job done. I probably doesn't help than I'm doing this on an old WHS2011 machine with 3x AOC-SASLP2-MV8 controllers, one of which is running in a 4x slot. I just hope I don't kill something in the process. I honestly didn't think the 10TB drive would work. I had it initialize, partition and format it on a newer PC for some reason. So I'm still not 100% sure how reliable its going to be.
After 4 hours, it actually looks like its copying about 500GB per hour. So maybe it won't a full month of Sundays... 

Hi everyone,
First, I would like to share that I am very satisfied with DP&Scanner. This IS a "State of the art" software.
Second, I have personally experienced 4 HDD drives fail, burned by the PSU,(99% data was professionally $$$$ recovered) and a content information, would have been comfortable, just to rapid compare and have a status overview.
I also asked myself, how to catalog the pooled drives content, logging/versioning, just to know, if a pooled drive will die, if professional recovery make sense (again), but also, to check the duplication algorithm is working as advertised.
Being a fan of "as simple as it get's", I have found a simple free File lister, command line capable.
https://www.jam-software.com/filelist/
I have build up a .cmd file to export Drive letter (eg: %Drive_letter_%Label%_YYYYMMDDSS.txt), for each pooled drives. Then I scheduled a job to run every 3hours, and before running, just pack all previous .txt's into an archive, for versioning purposes. 
I get for each 10*2TB, 60% filled pooled HDD's, around 15-20MB .txt file (with excluding content filter option) in ~20minute time. An zipped archive, with all files inside, is getting 20MB per archive. For checking, I just use Notepad++ "Find in Files" function, point down to the desired .txt's folder path, and I get what I'm looking for, on each file per drive.
I would love to see such options for finding the file on each drive, built up in DP interface.
Hopefully good info, and not a long post.
Good luck!
 

Ah, it's still early here. I'm not reading terribly thoroughly, yet.
found.000 is where chkdisk throws "lost" files it finds on scan. For whatever reason the allocation table lost track of those files.I've never seen such a large number of them, but that's not to say it's not possible.
Review the folder delete whatever you don't need, move everything else to a Recovered folder within your pool and forget about it. If it happens again, then get worried.
If the condition of the disk is bugging you Stablebit Scanner is a good tool Stablebit sells, but there are free options that are just as good (but not automated) such as CrystalDiskInfo (my favorite tool). https://crystalmark.info/en/software/crystaldiskinfo

I can answer this question.  But you won't like the answer.
Everything REQUIRES a volume that is capable of JOURNALING in order to work its magic.  The volume provided by DRIVEPOOL does not support that.  I know, bummer, but these are the facts.  I've already gone over this before a couple of years ago and there should already be a forum thread that talks about this situation.
Now for a small bit of good news...you CAN get Everything to work with a DRIVEPOOL volume BUT it works slower.  You have to fiddle with the settings and ADD ALL DRIVEPOOL drives to a special list in Everything.  Do that, and then Everything will work, BUT, as I said, because Everything uses the JOURNALING info to work its high speed magic, you won't get the blazing speed you've come to expect...but at least the program will function.
Look at the following screencap.  You have to add any DRIVEPOOL drive you want to index as a special INDEXED FOLDER (like my S:\ drive, which is the root of my filesystem on my DRIVEPOOL volume).

To get this started apparently:
 
My server was kind of piecemeal constructed.
 
 
I recently purchased a 42U HP Rack from a local company (via Craigslist), for super cheap ($50, so literally couldn't pass it up)
 
Sophos UTM (Home): 
Case: Antec ISK 110 VESA case,
Mobo (SoC): ASRock RACK J1900D2Y
RAM: 4GB of non-ECC RAM
OS Drive: Samsung 850 Pro 120GB SSD 
 
 
Storage Server:
Case: SuperMicro 847E26-R1K28LPB
OS: Windows Server 2012R2 Essentials
CPU: AMD FX-8120  Intel Xeon E3 1245v3 (link)
MoBo: ASRock 990FX Extreme3  Supermicro MBD-X10SAT-O (link)
RAM: 2x8GB Crucial ECC
GFX: nVidia geForce 9400  Intel HD 4600 (on processor GFX)
PSU: Built in, 2x redundant power supplies (1280W 80+ Gold) 
OS Drive: Crucial MX200 256GB SSD
Storage Pool: 146TB:  4x 4TB (Seagate NAS ST4000VN000) + 8x 4TB (WD40EFRX) + 12x 8TB Seagate Archive (ST8000AS0002), 2x 8TB Seagate  Barracudas (ST8000DM004), 2x 128GB OCZ Vertex 4s
Misc Storage: 500GB, used for temp files (downloads)
HDD Controller card: IBM ServeRAID M1015, cross flashed to "IR Mode" (RAID options, used to pass through disks only), plus an Intel SAS Expander card 
USB: 2TB Seagate Backup Plus for Server Backup (system drive, and system files) using a WD Green EARS 
 
 
NVR (Network Video Record, aka IP camera box) via BlueIris:
Case: Norco ITX-S4 
OS: Windows 10
CPU: Intel Core i3-4130T
MoBo: ASRock Rack E3C226D2I 
RAM: 2x8GB G.Skill 
GFX: ASPEED 2300
PSU: 450W 1U 
OS Drive: 128GB SSD, Crucial M550
Storage Pool: 2x4TB Toshiba HDD
 
HyperV VM Lab:
Case: Supermicro SYS-6016T-NTF (1U case) 
OS: HyperV Server 2012R2
CPU: Intel Xeon 5560  (x2, hyperthreading disabled)
MoBo: Supermicro X8DTU 
RAM: 64GBs (8x8GB) Hynix Registered ECC (DDR3-1333)
GFX: ASPEED 2300
PSU: 560W 1U 
OS Drive: 160GB HDD 
Storage: 500GB Crucial MX200 SSD, using Data Deduplication for VMs
 
 
Emby Server: 
Case: Unknown (1U case) 
OS: Windows 10 Pro x64
CPU: Dual Intel Xeon x5660's (hardware fairy swung by)
MoBo: Supermicro X8DTi 
RAM: 20GB (5x4GB) Samsung Registered ECC
GFX: Matrox (Onboard)
PSU: 560W 1U 
OS Drive: 64GB SSD,
Storage: 128GB  (cache, metadata, transcoding temp) 
 
 
Netgear GS724T Smart Switch
24 port, Gigabit, Managed Switch (one port is burned out already, but it was used). 
 
 
Dell 17" keyboard and monitor tray (used, damaged, propped up). 
 
Images here: http://imgur.com/a/WRhZf
 
 

Here is my network hardware.  Not a great image, but that's the 24 port, managed switch, a punchout block, waaay too long cables, cable modem and Sophos UTM box.
 

Misc drawers and unused spares. 
 

 
And my servers. HyperV system in the 1U, and my storage server in the 4U. And the Cyberpower UPS at the bottom. 
 
What you don't see is the NVR box, as it's been having issues, and I've been troubleshooting those issues. 

Have you tried resetting the status in StableBit Scanner?
 
To do so, open the UI, double click on the drive in question, click on the ">" button next to "File system health" at the bottom, and click on the button with the green circle, and select "mark as unchecked" (or to that affect). 
 
If it continues to come up as "damaged", enable logging, and reset the status:
Click on the Settings button in the toolbar and select Scanner Settings. Enable the Show advanced settings and information option, and hit "OK". You will only need to do this once, from now on, it will always display this option. Click on Settings, and select the new Advanced Settings and information option. This will open to the logging tab, by deafult. Find the "Chkdsk" option, click on it and select "Verbose".   Once you've done this, let it rescan the disk, and if continues to error out, grab the contents of "C:\ProgramData\StableBit Scanner", right click on the "Service" folder and select "Send To" > "Compressed Folder".   Then upload the compressed folder to us at https://stablebit.com/Contact

As @srcrist has said.  Change the settings to: 
Download Threads: 10
Minimum Download Size: 20MB
Prefetch Trigger: 5MB
Prefetch Forward: 150MB
Prefetch Time Window: 30s
 
Also, feel free to play with these values.  Though, don't increase the time window (this is how long you have to read the "trigger" amount, so shorter is better), and the forward should be 75-85% of "Download Threads" x "Min Download Size" (this way, it should grab a lot of data in parallel, but not so much that it bottleneck's CloudDrive from doing anything else for that drive). 

Changed it to 20 threads and now i am getting 300 mbps and hitting the 750 GB limit, so I think i am good now.

Hi,
I assume you mean to say that you have data in root folders on HDDs that are already added to the Pool and you want to move the data in the root folder(s) to the Pool, quickly.
This is what you are looking for: http://wiki.covecube.com/StableBit_DrivePool_Q4142489 I think.

Issue resolved by updating DrivePool. My version was fairly out of date, and using the latest public stable build fixed everything.

will do thanks!

Thanks, Christopher.  I agree.  Currently, I have a SMART warning just for LCC, so I did permanently ignore it. 

It could be related, yes.  But it's hard to know for sure.
If you could, upgrade to the latest beta and see if that fixes the issue. 
http://dl.covecube.com/DrivePoolWindows/beta/download/StableBit.DrivePool_2.2.3.948_x64_BETA.exe
If not, then try running a CHKDSK pass on all of the pooled disks (not the pool itself).
If that doesn't help, then I'd recommend opening a ticket at https://stablebit.com/Contact 

@Mick Mickle Unfortunately, there isn't a way to handle multiple settings at once.  
But honestly, if you are seeing multiple SMART warnings at once, you should replace the drive.  LCC by itself is fine, but with other errors.... it's time to replace. 

It's probably related.  Though, this should hopefully be resolved in the 3220 beta:
http://dl.covecube.com/ScannerWindows/beta/download/StableBit.Scanner_2.5.4.3220_BETA.exe

I assume you have no duplication. I would, provided I have enough ports:
Physically remove the faulty HDD (you have done this already) Remove it through the DP UI -> This should stop DP complaining about a missing disk and unlock the Pool. Add the two new HDD to the Pool Remove the two old HDDs from the Pool through the UI -> This will move all files to the new HDDs Remove the two old HDDs physically from the Server Then see what you can recover from the faulty HDD and copy that back to the Pool. I would consider to keep the two performing old HDDs in the Pool and use x2 duplication.

Yup.  It's a lifetime license that belongs to you, and includes all upgrades, and can be moved to a new system/OS.
I'm glad to hear it! 
And yeah, the automatic rebuild of the pool is very nice!

So, removing StableBit Scanner from the system fixed the issue with the drives? 
If so.... well, uninstalling the drivers for the controller (or installing them, if they weren't) may help out here. 
Otherwise, it depends on your budget.  I personally highly recommend the LSI SAS 9211 or 9207 cards, but they're on the higher end of consumer/prosumer.