fly

Just an update, I got a response from @Christopher (Drashna). According to him, this is a false positive. I sincerely hope that is the case, and if so, how do we get it cleared on all these sites?

I'm also seeing this error from Windows Defender. My CloudDrive was recently updated. Can we confirm that no malware got into this?

Since the C&P here was a little wonky, I uploaded it to Github. There have probably been some slight moditifcations to it since then as well. https://gist.github.com/flyize/ff78a44fa502c6b4cd10dd0e25212fb9

Hopefully they can update the balancer, but in the meantime, my PowerShell script might help.

If you'd like to see the genesis of this script, check out my original thread here Since I finally got my PowerShell script running, and I thought I'd post it here in case anyone else might find it helpful. SYNOPSIS: Script will move files from one DrivePool to another according to FIFO policy REQUIRED INFRASTRUCTURE: The expected layout is a DrivePool consisting of two DrivePools, one magnetic and one solid state. The main variables are pretty obviously documented. I added the file archive limit for people like me who also run SnapRAID Helper. That way the script doesn't trip the 'deleted' file limit (I'm assuming moves would trip it, but I didn't actually test it). Warning, I've obviously only tested this on my system. Please test this extensively on your system after you have ensured good backups. I certainly don't expect anything to go wrong, but that doesn't mean that it can't. The code is full of on-screen debugging output. I'm not a great coder, so if I've done anything wrong, please let me know. I've posted the code here so that you can't C&P it into a script of your own, since Windows can be annoying about downloaded scripts. Please let me know if you have any questions. Set-StrictMode -Version 1 # Script drivePoolMoves.ps1 <# .SYNOPSIS Script will move files from one DrivePool to another according to FIFO policy .DESCRIPTION The script can be set to run as often as desired. The expected layout is a DrivePool consisting of two DrivePools, one magnetic and one solid state. .NOTES Author : fly (Zac) #> # Number of files to move before rechecking SSD space $moveCount = 1 # Path to PoolPart folder on magnetic DrivePool drive $archiveDrive = "E:\PoolPart.xxxxx\Shares\" # Path to PoolPart folder on SSD DrivePool drive $ssdSearchPath = "F:\PoolPart.xxxxx\Shares\" # Minimum SSD drive use percent. Below this amount, stop archiving files. $ssdMinUsedPercent = 50 # Maximum SSD drive use percent. Above this amount, start archiving files. $ssdMaxUsedPercent = 80 # Do not move more than this many files $fileArchiveLimit = 200 # Exclude these file/folder names [System.Collections.ArrayList]$excludeList = @('*.covefs*', '*ANYTHING.YOU.WANT*') # Other stuff $ssdDriveLetter = "" $global:ssdCurrentUsedPercent = 0 $fileNames = @() $global:fileCount = 0 $errors = @() Write-Output "Starting script..." function CheckSSDAbove($percent) { $ssdDriveLetter = $ssdSearchPath.Substring(0, 2) Get-WmiObject Win32_Volume | Where-object {$ssdDriveLetter -contains $_.DriveLetter} | ForEach { $global:ssdUsedPercent = (($_.Capacity - $_.FreeSpace) * 100) / $_.Capacity $global:ssdUsedPercent = [math]::Round($ssdUsedPercent, 2) } If ($ssdUsedPercent -ge $percent) { Return $true } Else { Return $false } } function MoveOldestFiles { $fileNames = Get-ChildItem -Path $ssdSearchPath -Recurse -File -Exclude $excludeList | Sort-Object CreationTime | Select-Object -First $moveCount If (!$fileNames) { Write-Output "No files found to archive!" Exit } ForEach ($fileName in $fileNames) { Write-Output "Moving from: " Write-Output $fileName.FullName $destFilePath = $fileName.FullName.Replace($ssdSearchPath, $archiveDrive) Write-Output "Moving to: " Write-Output $destFilePath New-Item -ItemType File -Path $destFilePath -Force Move-Item -Path $fileName.FullName -Destination $destFilePath -Force -ErrorAction SilentlyContinue -ErrorVariable errors If ($errors) { ForEach($error in $errors) { if ($error.Exception -ne $null) { Write-Host -ForegroundColor Red "Exception: $($error.Exception)" } Write-Host -ForegroundColor Red "Error: An error occurred during move operation." Remove-Item -Path $destFilePath -Force $excludeList.Add("*$($fileName.Name)") } } Else { Write-Output "Move complete." $global:fileCount++ # Increment file count, then check if max is hit If ($global:fileCount -ge $fileArchiveLimit) { Write-Output "Archive max file moves limit reached." Write-Output "Done." Exit } Else { Write-Output "That was file number: $global:fileCount" } } Write-Output "`n" } } If (CheckSSDAbove($ssdMaxUsedPercent)) { While (CheckSSDAbove($ssdMinUsedPercent)) { Write-Output "---------------------------------------" Write-Output "SSD is at $global:ssdUsedPercent%." Write-Output "Max is $ssdMaxUsedPercent%." Write-Output "Archiving files." MoveOldestFiles Write-Output "---------------------------------------" } } Else { Write-Output "Drive not above max used." } Write-Output "Done." Exit

Figured it out. For whatever reason, when using mount points, the PoolPart folders don't get created until something is written to it. That doesn't seem to be the case for drive letters. Is that possible or am I missing something?

Thanks. I did see that KB, but I'm not sure I follow. Where will the Poolpart folders be for the new top level drive? Hopefully this makes sense: D: - Top level drive pool |___ E: Magnetic drive pool |____ C:\mount\1\PoolPart |____ C:\mount\2\PoolPart |___ F: Sold state drive pool |___ R:\ edit: Just created the above DrivePool hierarchy. The top level drive was empty and I only saw one PoolPart folder in C:\mount\1. I'm clearly missing something.

Am I asking really dumb questions? (Other than this one?)

And to follow that up... If I create a new top level DP drive hierarchy, how do I prime the new top level drive with the contents of DP drive below it?

Okay, so I've been thinking more about this again. Here's my plan to have a FIFO SSD cache. Create a DrivePool of magnetic drives (E:) Create a DrivePool of solid state drives (F:) Create a DrivePool containing E: and F: (D:) I would create placement rules for D: that tell DP to only place files on F:. Then I'd simply write a PS script that would check free space on F: and move files to E: until there was XXXGB of free space. If it matters, the only reason that E: and F: would exist is so that I don't have to find the underlying disk with the most free space on E: or find the SSD under F: that needs files archived off. I just have my script move things and let DP deal with file placement. Would this work?

I actually assumed that's how they would work, but apparently not. I have an SSD that I want to add to the pool, but only have new files placed there. As soon as I create the '* > SSD' rule, DP starts filling up the SSD. What am I not understanding?

No. I asked about this a few months ago as well. At the time, I started working on a PowerShell script that would move stuff in FIFO fashion from SSD to platter. At that time, I failed. But I've started working on it again.

This is all a good point. Not sure why I didn't think of it before. Thank you. edit: Can I 'backup' the same data from the same CloudDrive to multiple storage providers?

I'm confused and concerned. I use CloudDrive to backup important client data for my business, as I assumed it was the perfect backup solution. Are people actually losing data?

I often travel for work with simply my ChromeBook. Is it possible for me to access my encrypted files from it?

Just checking in to see if any movement was possible on this?

While I would guess that there's no real correlation, that was the easy part in PowerShell. This recursively (and surprisingly quickly) searches a directory and grabs the two oldest files: $fileNames = Get-ChildItem -Path $searchPath -Force -Rec | Where-Object { -not $_.PsIsContainer } | Sort-Object CreationTime | Select-Object -Property FullName -First 2

Pretty please? Just to reiterate my use case: My media server has quite a few users. As new media comes in, it's likely to have a lot of viewers and therefore a lot of I/O. As files get older, fewer and fewer people access them. So I'd love to have an SSD landing zone for all new files, which then get archived off to platters as space requires. It would be fantastic if I could say, once the SSD drive(s) reaches 80% full, archive files down to 50% using FIFO.

This is turning out much more difficult than anticipated with my limited PowerShell skillset. How would I go about putting in a feature request?

Ah, that's probably it! The link that @Jaga posted above might need to be updated. http://wiki.covecube.com/StableBit_DrivePool_Utilities#DrivePool_Command_Utility

So I'm finally getting around to writing this... I'm playing with dp-cmd right now and ListPoolParts doesn't seem to work correctly. Does it need to be in a certain directory to work? c:\Users\Administrator\Desktop>dp-cmd.exe ListPoolParts Error: Method not found: 'CoveUtil.OmniStore.Store CoveUtil.OmniStore.Store.Create(Boolean, Boolean, System.IO.DirectoryInfo, Boolean, CustomSerializer)'. Details: System.MissingMethodException: Method not found: 'CoveUtil.OmniStore.Store CoveUtil.OmniStore.Store.Create(Boolean, Boolean, System.IO.DirectoryInfo, Boolean, CustomSerializer)'. at DrivePoolCmd.DrivePoolTasks.ListPoolParts() at DrivePoolCmd.Program.Main(String[] args)

Well then, seems I have a weekend project. Will report back with script in case anyone else would like to use it.

If I ran my script nightly, is there a way for it to tell DP to examine the pool again? Otherwise, if I'm understanding you correctly, it sounds like the files would appear unavailable.

If I create the PowerShell script, and it moves files to different drives, are you saying that DP is alerted to the change instantly and will update it's physical location on the virtual drive? If so, I'd think it would be relatively trivial for my script to determine the drive with the most space and dump it there! And yes, ideally that would be great. Probably choose a max used and a min used. Then once max is hit, dump down to min, based on FIFO. That would be perfect for my use case.

I'd like for newest content to be on SSDs, since it's the content most likely to be accessed. If I could get FIFO, then it wouldn't be that upsetting to SnapRAID. What if I just did this 'manually' with PowerShell? I would assume that DrivePool wouldn't be very happy if I just placed a file on one of the other drives PoolPart folder, but it's worth asking. I suppose a RAID 1 would work, but somewhat less than ideal since it's basically a 50% storage penalty.