Performance with (b)millions of files

[kts]

Hi,

 

we're testing drivepool to determine if it might be an suitable solution to keep a second online copy and backup.

 

We're dealing with huge numbers of small files (500kb - 25mb) in several "main" folders each with 4096 subfolders. Now we got quite a few of those main folders with between 10 and 50 million files each and adding 25-100k files per day. Additionally we have to import other systems which may add 10-30 million files at once. We're planning for 24 or 36 drive systems with drivepool and can see the 1 billion file barrier break on single systems.

 

I've set up a test machine (i7, 32gig and 6x4tb 7.2k drives exclusively for storage) and gave it a taste of around 50-60 million files with duplication (2x). It did not to bad until I restarted the system. It is working but the dashboard is calculating for a couple of hours now and claiming xxx GB are not duplicated (while they are, number is increasing, I assume it has to check _every_ file on disk?). Will give it a couple of hours more to get organized.

 

Is if drivepool is suitable for those large number of files? I assume there is some kind of database keeping records where which files are placed. Is there a maximum database size or overall file limit per pool? Do those numbers have a known impact on performance? Does this recalculation occur after every restart?

Anyone having experience with similar file numbers?

 

Thanks!

[Christopher (Drashna)]

Well, it depends on how the files were imported.
Were they copied to the pool directly? 

 

Also, what version of StableBit DrivePool are you using?

And of Windows?

 

 

And, no we don't maintain a database of the individual files. We measure the files, and then keep track of changes that happen after the fact (through the pool drive, or anything done by duplication or balancing).

 

And no, this recalculation shouldn't happen at every restart. In fact, I believe it shouldn't happen unless manually initiated, or a disk has been removed or is missing, or added.

 

And yes, it does check all the files in the Pool.

 

And since you're using a LOT of small files here, it may be a good idea to increase the background task priority, so that if this does happen, that it will complete much faster. 

To do so, use the following link:

http://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings

And set "DrivePool_BackgroundTasksPriority" to something higher (like 6).

[kts]

Version 2.1.1.561

Windows 8.1

 

I restarted again, and the process started from the beginning.

 

After reading your reply I restarted the service, it kept the statistics. After changing the priority and restarting the service it lost them again, and is now indexing quickly. Strangely it counts to a 2 or 3 figure number and then drops down to close to zero again (I must say that I'm continuing to add data and assume this is the problem). I'll give it a while to calm down. If it will finish I may restart again to see if that is a permanent problem.

 

Do incomplete statistics impair or slow drivepool down? As far as I can see (and hope) balancing and duplication is still working.

[Christopher (Drashna)]

If the system hasn't fully calculated the size, then if you reboot, it will restart the count. And if the system crashes, then it will cause it to have to recount as well.

 

So if you wait for it to finish counting and reboot.... if it starts recounting again, then something is going wrong during shutdown.

Also, you're using USB or external drives, if they're taking too long during boot, it can cause issues as well.

Either of these issues can be identified by using the event viewer's "System" section.

[kts]

Just wanted to let you know it's working now. Checking takes several days but everything seems to be doing good so far. Performance is good even with many small files.

[Christopher (Drashna)]

I'm glad to hear that!

 

Especially with the small files.

[kts]

We're beginning to run into a problem with the measuring process. Apparently it takes too long to finish (takes several days up to a week?!?) and with about 2/3 of the drives measured it starts from scratch. Is this behaviour normal?

 

I resetted the settings already and also increased CoveFs_AsyncCriticalThreads and CoveFs_AsyncNormalThreads. Although done, there is not much activity during measuring. Also got the BackgroundTaskPriority up to 12 meanwhile.

 

Is there a way to increase the time in between measuring and especially speed up the measuring process itself? DP service uses like 0,1% of CPU and the measured HDD is at around 100kb/s.

[Christopher (Drashna)]

IIRC, the measuring occurs in the kernel drive, so I don't think that those settings will help.

 

However, could you install the latest version?

http://dl.covecube.com/DrivePoolWindows/beta/download/

 

There are some fixes to the measuring code, and additional logging. If it's still having issues with this version, then enable file system logging and remeasure the pool.

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

[RobbieH]

I have a question. In my experience, Windows operating systems, even Windows Server OS installs, don't like very large numbers of files within a folder. All sorts of bad performance issues crop up, mainly huge performance issues when trying to do any sort of mass file stuff - example would be backing up the folder. Christopher, does this issue affect the performance of DrivePool?

[Christopher (Drashna)]

I have a question. In my experience, Windows operating systems, even Windows Server OS installs, don't like very large numbers of files within a folder. All sorts of bad performance issues crop up, mainly huge performance issues when trying to do any sort of mass file stuff - example would be backing up the folder. Christopher, does this issue affect the performance of DrivePool?

Yes, it would.  Specifically, the Pool driver acts as a sort of file system proxy. If the drive takes a long time to enumerate data, then that delay is seen by DrivePool.  This applies to measurement, listing folders, etc.  The Pool is completely "at the mercy" of the underlying disks.

 

 

However, we do run a number of optimizations to help expedite everything.  And that includes taking advantage of NTFS's caching.  Additionally, the actual measurement takes place in the driver (a kernel driver, specifically), so it should be VERY fast, and should have no issues with NTFS permissions.