Jaga
-
Posts
413 -
Joined
-
Last visited
-
Days Won
27
Posts posted by Jaga
-
-
I've used SiSoft Sandra many years ago, so that's one option. Don't know if it's sector/drive or file cached, but a little investigation and testing might tell. Never used IOzone, but it might be interesting to run too. And I'm fairly certain Christopher and Alex would like to see the results, if for no other reason than morbid curiosity.
My Windows 7 Ultimate 64-bit server (the one in the tests) has 32GB of RAM, and with just the Windows cache it ran abysmally slow. It almost got to the point of irritation, so I broke down and bought a copy of Primocache for it and it was night-and-day. I don't know how Windows 7 32-bit could address a 1TB cache, perhaps that's the x64 window you're seeing. Mine with only the Windows cache never used up more than ~2GB total RAM, even after leaving it on for weeks serving up Plex music/movies. So it's not nearly as aggressive as it should be.
If you get around to installing Primocache, let us know your results. And/or ask for recommended settings over there - if you use the same username I'll see and reply.
-
Also, as far as I know, you can't "set Windows' system cache to allow it to expand well past normal limits" - In Win7 at least, it will used-up ALL available RAM by default (at least it does on my 32GB system, and I haven't touched any cache parameters.)
Sure you can. Windows 32-bit program here. 64-bit here. Just don't change anything using the 32-bit interface after setting limits using the 64-bit program, or it might fubar the settings.
I'd use PrimoCache almost immediately if I could tell it to do like Windows does - Grow to use all available RAM, then shrink when something needs the RAM. I'm not too keen having to specify in advance how big the cache will be. What happens if we use BOTH PrimoCache and Windows Cache? I might be interested in a small PrimoCache, below the native cache (but really, what I want is a cache that grows and shrinks, and that does write-back...)
Primocache can't do that, but their other product Primo RAMdisk can. You set a maximum limit for it, and it only takes as much RAM as the RAMdisk is currently using (I have it installed and working that way myself).
It's a good suggestion for Primocache however - I may toss it at the main developer over there and see what they think of a min/max limit on the cache so it can dynamically adjust.
Primocache exists just fine right alongside the Windows cache. Since Windows isn't aggressive in file caching (at all), it typically uses very little RAM for files it reads. And it doesn't have a write cache (I've never heard of anyone seriously using the write-back feature and getting good results). That's where Primocache picks up - it has a much more robust kernel-level cache that will pick up where Windows leaves off, they exist in perfect harmony. If a file is in Windows' cache, Primocache doesn't take any action, it just lets Windows deliver the content. If it's not in Windows' cache, Primocache caches the blocks being requested (it's block level, not file) and delivers the content. Since it is kernel level, it can also be told to pre-fetch the contents of the cache at last shutdown during boot time, to speed up the system before Windows has even loaded. My boot times (with the RAMdisk turned off) are around 6 seconds, and I'm not using a NVMe.
Primocache's write cache is quite robust as well, and highly recommended with a UPS on the machine. Both of my benchmarks above are with the write cache on, and when using a large L1 (RAM) cache for both read/write with large block size.
Even if you only have 1GB to throw at a L1 cache (or a spare SSD to use as a L2 cache against spinner drives), it can dramatically speed up almost any system.
I rebooted my server just this morning, so the cache hit rate isn't as high as it would normally be (around 93%), but this gives you a good look at how I have my boot volume cache configured, and at the Primocache interface on a live machine:
If you know of a benchmark program that works at the file level (and not sector or drive level), let me know what it is and I'll re-run the benchmarks at whatever file size you think is good. It'll be interesting to see how Drivepool responds in both cases.
p.s. Thanks @Christopher (Drashna) & @Alex for allowing us to discuss non-Covecube products on the forums here!
-
You could put multiple volumes on the physical drive, size them how you wanted, and then add the volumes to different pools. That would also allow separate security permissions, features like encryption, etc on the separate volumes.
As far as I know Drivepool doesn't have the functionality built in.
-
Just a thought - you said you "recently installed" the enclosure. If *any* part of Windows is reporting the drive size/features incorrectly, you could attempt to either return or exchange it, OR file it under warranty as defective.
-
As a side note: I've been using Primocache for around 6-7 years, since it was called Fancycache. Windows' cache can't even begin to compare. How do I know this for certain? The server that Drivepool is being benchmarked on in the above examples has 32gb of RAM, running Windows 7, with a 5400 RPM system drive. I've explicitly set Windows' system cache to allow it to expand well past normal limits. Booting the system and running things like Plex take absolutely forever, but doing the same thing with Primocache installed takes *far* less time. It almost runs at the same speed my 4.7ghz workstation with a 512GB Samsung Pro SSD runs. That's how good Primocache is.
But hey - I'm not here to pimp Primocache, just to let Drivepool users know they have an option if they want to speed up their pools. If you don't believe my numbers and opinions, do as I suggested and test it out yourself.
-
I didn't really do it to compare speeds, but rather to show that a popular, robust, and established caching program works with DrivePool. And it's one I highly recommend (I run it on all my workstations, laptops, and server). The only caveat in the case of using it with a Drivepool, is that usually pool drives are very large, so you need a lot of RAM to get even a small hitrate on the cache. But for writes, most recently accessed files and frequent files, it's quite effective.
I ran a similar test with Anvil Pro, despite it being geared more against SSD testing (which really is how you should benchmark Primocache). Here's the test with Primocache paused:
And the test with Primocache re-enabled:
It shows a ridiculous increase in speed, as you'd expect. The read/write speeds for both tests are exactly where I'd expect to see them. And since Windows' cache isn't all that aggressive (nor does it auto-scale well), I would not expect to see any difference using a benchmark that was intended to use it. Anvil Pro may - I don't know. Certainly the Windows cache wouldn't have much of an impact once you start scaling the test over 1GB.
Feel free to run the test yourself, with whatever benchmark software you want. Primocache has a free 60-day trial license with full features so people can test it themselves.
-
Possibly - I did a search on the title, but the word "adapter" brought up too many results to be useful. Thanks for flagging it Christopher.
-
I really, really want one of these 115 PB drives.
Definitely sounds like a controller/backplane issue though.
-
Just a suggestion for those of us using VPN on machines with CloudDrive installed. It would be nice to be able to bypass the VPN without implementing static routes in the table. Just a config option somewhere that polls network adapters, and allows the user to bind CloudDrive (and it's service) to a specific adapter, forcing all traffic in/out on it.
My VPN slows traffic appreciably, and being able to use secure communication on the primary adapter would significantly speed up transfers and lower connect times.
-
And uncheck the "Wake up to Scan" option.
With a sleeping server, wouldn't you want to wake it up for the schedule to keep on track? That's how I used to have mine set. Otherwise it sleeps, and sleeps more, and keeps on sleeping...
-
Scanner also pings the SMART status of each drive on a short timer. You will want to go into Settings, and to the S.M.A.R.T. tab to enable "Throttle queries". It's your choice how long to set it, and you also have the option to limit queries to the work window (active disk scanning) only, or to not query the drive if it has spun down (sleep mode on the drive).
In your case, I would simply throttle queries to every 5-10 minutes, and enable "Only query during the work window or if scanning". Then under the General tab set the hours it's allowed to access the drives to perform scans, and check "Wake up to scan". It should limit drive activity and allow the machine to sleep normally.
-
Definitely a false-positive. Covecube's releases are clean. Disable Avira and re-install.
-
As long as you don't overfill the drives (forcing a re-balance), or use the Disk Space Equalizer plugin (which also forces an immediate re-balance) it shouldn't ever touch those files. Even adding a new drive to the pool shouldn't do it, since Drivepool won't auto-migrate. It will just keep adding new files to the disk with the most free space.
If you setup odd balancing rules however, all bets are off. You'll have to approach that with care, if you ever do.
-
Second test on this with a small 1GB Cloud drive seems to be behaving better. No errors being thrown by SnapRAID's sync/newscrub cycle. I think perhaps there's some activity still transpiring even after the service is shut down (very short-term afterwards) which was giving it issues.
For reference, I am -not- stopping the service for these recent SnapRAID runs. The Cloud drive simply has nothing on it, and nothing to do at that time of the night.
Going to re-make a larger 100GB Cloud drive and duplicate some data onto it, and see what the cycle does in the morning.
-
1 minute ago, Bryan Wall said:
It would be great if Drivepool just had an "eject" button of its own, which just closed all files it had open on the selected pool/drive. It's impossible to really be sure that nothing is writing to the drive at the moment you pull the power or set it to offline.
Something similar to CloudDrive's "Pause uploads" menu option. It would flush any pending data transactions and put the pool in to an offline mode. Sounds handy for some situations.
-
Just adding information in the hope it will be useful: using DP v2.2.0.906 I don't see any memory leaks occurring.
-
This topic might be of help to you using Ocuclus. Christopher's suggestion to install directly to the poolpart (hidden folder) on one of your pool drives may be a solution.
-
I was going to suggest using Hierarchical Pooling to make a top-level pool consisting of your normal data pool, and a new 'external pool' made up of your USB drives. However I'm unsure of how Drivepool treats removing sub-pools that have duplicated data on them. Normally when you remove a standard drive from a pool, DP tries to migrate files off that drive, and then copy duplicate files it's holding to another pool drive (which in this case would be another sub-pool, that doesn't exist). I'm thinking that DP would flag the duplicated files on your primary pool as read-only if you just unplugged the external drives, which isn't helpful either. There are more command-line options for DP that might accomplish what you're looking for. Someone more knowledgeable might have to help, as I don't have a concrete answer.
Backup software like Macrium might be more efficient however, since it can do compression (which affects both space used and throughput). It would end up giving you a true backup.
The other option would be to setup your external drives as a new, separate pool, and then use Sync software to mirror both pools from a file-level (with the data pool being the master). It would really be the easiest and cleanest option, in my opinion.
-
Try using Process Explorer to poke around and see what's using up RAM/CPU and writing to disk.
-
Good to hear.
-
9 hours ago, Christopher (Drashna) said:
In Server 2012R2, and up, I'd say it's pretty much ready.
The biggest issue is that most disk tools are NOT compatible with ReFS yet. Which is the real problem.
That saves you cost up front, and dumps the cost on the warranty. Eg, you have none. So if the drive fails, you may be SOL, if you didn't save the enclosures.
But seriously, a 3-5 year warranty is worth the difference, IMO.
I haven't messed with the server implementation of ReFS, though I assumed it used the same core. I ditched it ~2 years ago after having some issues working on the drives with utilities. Just wasn't worth the headache. I never had actual problems with data on the volume, but just felt unsafe being that "out there" without utilities I normally relied on. When the utilities catch back up, I'd say it's probably safe to go with it, for a home enthusiast. Just my .02 - I'm not a ReFS expert.
Shucking has positives and negatives, to be sure. There's one 8TB drive widely available in the US that normally retails for $300, and is on sale regularly for $169. For a reduction in warranty (knowing it's the same exact hardware in the case), I'm more than happy to save 44% per drive if all I need to do is shuck it. They usually die at the beginning or end of their lifespan anyway, so you know fairly early on if it's going to have issues. That's my plan for the new array this July/Aug - shuck 6-10 drives and put them through their paces early, in case any are weak.
6 hours ago, Dave Hobson said:Now I'm even more curious as I have just read a post on /r/snapraid suggesting that its possible to raid 0 a couple of pairs of 4TB drives and use them as 8TB parity drives. Though the parity would possibly less stable it would give me parity (even though it's not priority) and would allow for data scrubbing (my main aim) and mean that those 4TB drives wouldn't sit in a drawer gathering dust. So if any of you Snapraid users have any thoughts on this, I would be glad for any feedback/input..
No need to RAID them just for SnapRAID's parity. It fully supports split parity across smaller drives - you can have a single "parity set" on multiple drives. You just have to configure it using commas in the parity list in the config. There's documentation showing how to do it. I am also doing that with my old 4TB WD Reds when I add new 8TB data drives. I'll split parity across 2 Reds, so that my 4 total Reds cover the necessary 2 parity "drives". It'll save me having to fork out for another 2 8TB's, which is great.
-
Well, anything inside the cloudpart directory honestly. I was a bit stunned to see anything in that directory change after the CloudDrive service was stopped. It put a monkey wrench in SnapRAID's parity calculations, since the changes happened in-between the sync pass and the scrub pass (each typically takes 12+ minutes with normal daily changes to my pool on the same drives).
I am more than happy to try it again (adding a Cloud drive, stopping the service, running a sync and then a new-scrub) if you think nothing in there is -supposed- to change while the service is stopped. But I don't know how else bits would have been altered, unless it's something Windows 10 is doing to that folder/file structure on it's own. My assumption is that when the CloudDrive service is stopped, all information is flushed to disk, the disk is unmounted, and no further activity happens to the cloudpart folder.
-
Yep, you'd want at least 2 parity drives for 13 data. If cost is an issue, look into "shucking drives" to save some money.
-
1 hour ago, Dave Hobson said:
So, in a nutshell, would you recommend ReFS (with 64K clusters) and just disable monitoring in PerfectDisk to eliminate my OCD
I'm thinking 64K blusters because on these drives the files sizes are 5GB minimum up to probably 80GB. More than happy with a yes or no.
No. ReFS isn't ready for primetime yet. Tried it for a year on a pair of drives in software RAID 0. Did a lot of testing, and a lot of reading. Too many people running into issues on ReFS for me to fully trust it. And many of the usual utilities that work well with NTFS aren't updated for ReFS yet. I combat bit rot by using SnapRAID and frequent syncs/scrubs on my data. I'd rather have full control, but that's me.
Use the same disks for multiple pools
in General
Posted
Yep, there are limitations with that method, but it would work.
With a drive partitioned into volumes like I suggested, you'd just have to over-provision space. Or you could divide each disk's volumes like some storage/web providers do - a single disk consists of perhaps 2 large volumes, 4 medium volumes, and 8 small volumes. That allows you to move a user around (upgrade or downgrade their storage) when they no longer fit their old volume. The larger the physical disk you have to work with (i.e 12 TB) the more volumes and size options you could come up with. And you could mix-and-match different size volumes from different disks to make a single user's pool, allowing some flexibility, provided you held some volumes in reserve for that purpose.