gj80

The dpcmd log file is only used to produce the csv. I could just set the script to auto-delete the log file, I guess. I just figured that it wouldn't use any significant amount of space as a single file being left sitting around (since, again, there's only ever the one log file - those aren't retained in the zips). Converting to ascii would cause filenames/paths to be incorrect if any extended characters were used (kanji, etc presumably). I'll set the script up to auto-delete the .log and .csv after the zip process in a later revision. For now, we're still testing (and the auto-zip isn't working for you), so doing it now would make it harder to test. Wow! Can you open it and filter by the filepath column in excel? How much memory does excel use when doing so? At 1.4m files, that should be as good a pretty good test of whether a spreadsheet will work in all cases. Your .NET is newer than mine, and we're on the same OS and PS version...it's odd that it's not working for you. Hmm. It should work regardless, but can you maybe try making a root-level folder on your C: drive, without spaces, like c:\dplogtest or something, setting the $auditLocation to that, and then granting "Everyone" full access to that folder in the NTFS security settings? Also, is the language locale in Windows on that computer English, or perhaps something else? Oh, and one other thing to check... the zip files it's producing that are 1kb... can you manually double click in Explorer to enter those archives as a folder (and paste a file into them), or does it give you a message about the .zip file being damaged/invalid?

The log file is produced by the "dpcmd" utility, which is part of DrivePool itself, and not something I wrote - so I can't control whether it writes out as unicode or not. It makes sense for it to be in unicode, though, since otherwise some filenames with non-ascii text wouldn't be logged properly. The .log isn't zipped, though - only the CSV. So, there will only ever be the "current" .log file after each run, and none in the zip files. As for the CSV which the script is producing from the log file, I'm opening the file for writing with ASCII text encoding. Actually, I guess I probably should have gone with unicode for that as well... hmmm.

Aaaand added V1.4 just now. I realized it wasn't properly zipping files on my side either, and it was due to something I had added after I tested the zip functionality originally. That has been fixed, and it's zipping files again now for me. Let me know if it doesn't work for you now with v1.4, thanks

Okay, uploaded new V1.3 that fixes the bug where it wasn't recording all the duplicated entries for each file. In my test pool I've now got some 2x duplicated files (1 spare copy per file) and those are all being logged properly. If anyone has a pool with 3x or more, I'd appreciate feedback... it should scale, but I couldn't test it. Thanks!

The idea is that you'd sort by the path column in Excel or another spreadsheet program, and then you'd see all the disks that a particular file reside on. I did test this on a pool without duplication, though... I just thought the "dpcmd" output was going to work like that, since it listed specific disks associated with each file line by line. Did you sort by that column in excel, and still only saw one copy of each file listed??? Edit: Oh! My bad... I just looked at the example output you posted earlier and saw how it's handling duplication. I'll have to revise the script. Thanks for catching that. You mean that the path doesn't start with the drive letter? I thought this might not work for some... can you let me know: * What version of Windows you're using * What version of powershell you're using (paste $PSVersionTable.PSVersion in a powershell window) * What's the newest version of .NET you have installed? You can find this either by looking in the registry or running a script. Thanks!

Thanks!

Odd - I watched for any memory issues on my end, but it only seemed to go up very minorly as the array in memory increased. Maybe powershell is handling memory allocation differently between our powershell versions. Anyway, I switched it over to use a streamwriter method instead. I guess I should have done that to begin with. Now that I'm avoiding constructing objects for each line in an array, it's gone from 50 lines per second to over 700. No memory buildup will happen as well. Plus, the .CSV file can now be read while it's being written. I've updated the attachment above to V1.2

The script takes the trailing "[Device XX]" from the dpcmd output, matches it up to the "DeviceId" property from "Get-PhysicalDisk" in powershell, and then pulls the disk's serial numbers and models from other corresponding properties. So those are added to the CSV (which the script produces once the dpcmd output is finished to the .LOG file), rather than being logged directly by dpcmd. You'll know the dpcmd portion is finished when you don't see it running any longer in task manager. At that point, you could stop the task, and you should have an incomplete .CSV to examine (if you don't want to wait until it's completely finished to check the format/etc).

Since you initiated it via task scheduler, it shouldn't run a second time as long if it's still running. I've done some work with SQLite via powershell before. It definitely allows for more flexibility and performance, and I'm comfortable running SQL queries, but without a UI, I think not being able to just open a CSV and filter in excel would limit the usefulness for a lot of people... it certainly might be a good alternate "mode" to have, though. Worst case, there's probably an easy way to convert a CSV over to a SQLite DB with one table. It will definitely be interesting to see how this turns out for you, since it seems like you have an extremely large pool. If you get the resulting CSV open in excel when it's done, please jump to the end and let us know the number of rows, and how long it took to produce it (the difference between the timestamp on the "DPAuditLog-Current.log" and "DPAuditLog-Current.csv") along with the CSV file size (and the .ZIP file size). Thanks! It doesn't rely on having any of the drives mounted. The path it records in the CSV is relative to the base of the pool, like: "BACKUPS\365\CLIENT\data\filename.file" It then records, per line, the disk number (what you see in disk management), along with the disk model and serial number (to help physically identify it) and the file size in bytes. ...I guess one way of cutting down on the size would be to just record the disk number, and then record the associated disk models and serial numbers for each number as a separate file that is zipped up together with the CSV.

It certainly sounds good. I'm not sure if it'll be worth us to put in the time required as end-users, though... If it was built into the application itself it'd provide more utility to the entire userbase, but as end-users writing an accessory utility, only a small portion of the overall userbase will end up using it. And, being honest, I'm sure not everyone is even as paranoid about all of this as we are The only UI work I've done is in AutoIT, but from my limited experience with it, doing those first two tabs would be a lot of work. UIs, at least in AutoIT, are kind of a pain. Or, maybe I'm just less comfortable/experienced with it. Opening the CSVs directly and filtering on the headers with a spreadsheet program are probably things that anyone who would be using an accessory utility like this would be comfortable doing. I could be wrong though... The third tab (and a UI for an installer which would put the file in a fixed path, run the task scheduler command, etc) would probably bring more utility to people for the time invested, though - I know some won't feel comfortable editing a script file, even to just modify a few variables. Even that would still probably be a fair amount of work, though... I work with powershell, regex stuff, etc a fair amount, so that was pretty easy. I'm not comfortable with trying to tackle a UI for it personally. If you feel inspired, though, then go for it! The idea about retention is a good one - I think I'll update the script to add that as a variable and a routine to clean up old files. Edit - I added a day-based retention setting and added the updated "V1.1" attachment to the post above.

Hi, I really liked the idea of doing this, but as others had also mentioned, I had issues with the maximum path limitation. I also wanted to log other information, like the drive model, serial number, etc. I also wanted to schedule it as a daily task, and to have it automatically compress the resulting files to save disk space. I wrote a powershell script to do all of that. It relies on the dpcmd utility's output (for which you will need a recent version). DrivePool itself isn't limited by the maximum path limitation, and thus the dpcmd log output also isn't constrained by the path limitation. The script takes this log output, parses it with regex matches, retrieves associated disk information, and then writes out to a CSV. It then compresses it. The header of the file has a command you can paste into a CMD prompt to automatically schedule it to run every day at 1AM. Please edit the file. Two variables need to be customized before using it, and the file describes requirements, where to put it, how to schedule it, what it logs, etc. If you want to do a test run, you can just edit the two variables and then copy/paste the entire file into an elevated powershell window. The .CSV (generated once the .LOG is finished) can be viewed as it's being produced. Also, you might want to hold off playing with this if you're not familiar with powershell scripting/programming in general/etc until a few other people report that they're making use of it without any issues. @Christopher - If you want me to make this a separate post, just let me know. Thanks DrivePool-Generate-Log-V1.51.zip

Hi, I would like to enable the "DrivePool_VerifyAfterCopy" option. I was wondering if it's possible to edit a copy of the "DrivePool.Service.exe.default.config" XML file to only contain the option I want to override. That is, if I did this (removed all the other "<setting ...> ... </setting>" pairs), would the missing ones be inherited, or would it cause errors? I'm asking because I'm a bit leery of accidentally forgetting that I had every single parameter explicitly defined in the event of a drivepool program update, and forgetting to recopy the (new) default XML and make the change again. Thanks!

I'm so glad I noticed this topic! The character limit has plagued me since I first started using Windows computers... I rely on folder structure a lot at home and at work, and I bump into it constantly. My clients bump into it quite often as well, and they end up losing data inadvertently - it's a nightmare. I understand it could cause application compatibility issues when directly referencing those locations, but at least Explorer can support addressing these files now. Without explorer itself supporting it, any under-the-hood changes before have been kind of moot for most applications. I was relieved to know that DrivePool itself has supported extended paths previously, though - I had opened a topic about that question since I knew it extended the path length for existing data on each disk since it's stored in specially-named folders.

Hi, I'm doing some research regarding possibly deploying this for some of our serverless clients that want a mobile shared document repository. I had one question though... My understanding from research I did a few months back was that onedrive for business has a "20,000 item sync limit". And sharepoint team libraries have a 5,000 item limit. This is an issue for a lot of our users that have more files than that. Does clouddrive inherit the same limitation, or is the way files are stored (by chunk?) block-based and thus doesn't run into that? Oh, a few more questions... Backups - when using clouddrive, is running a backup from a computer the most practical solution for backups, or is one of the cloud-based backup services for 365/sharepoint also viable? Does clouddrive support VSS? Change management - when multiple people edit a file, how is it determined which version is committed? The last one committed takes precedence? Edit: on second thought, I think I may have gotten the wrong idea about this - this is entirely a personal drive, right? No sharing? Still curious about the file limitation/backups though. Thanks!

Thanks. I bought another bundle, and cloud (great product idea) came along with StableBit and Drivepool in the bundle, so I thought I'd try it out. Not a big deal, but I've noticed unreported-by-disk bit flips corrected and reported in ZFS pools I manage, so checksumming always makes me feel better - though of course it's less important for my own personal stuff. Thanks for following up on it.

I added a local DrivePool and attempted to create a cloud drive, but I noticed that "use checksum verification" is disabled. I don't really need cloud storage, but I've been looking forward to some sort of checksum solution for use with DrivePool, which is why I was going to try it out. Is the feature disabled for local storage sources?