fly
-
Posts
32 -
Joined
-
Last visited
-
Days Won
3
Posts posted by fly
-
-
I'm also seeing this error from Windows Defender. My CloudDrive was recently updated. Can we confirm that no malware got into this?
-
Since the C&P here was a little wonky, I uploaded it to Github. There have probably been some slight moditifcations to it since then as well.
https://gist.github.com/flyize/ff78a44fa502c6b4cd10dd0e25212fb9
-
Hopefully they can update the balancer, but in the meantime, my PowerShell script might help.
-
If you'd like to see the genesis of this script, check out my original thread here
Since I finally got my PowerShell script running, and I thought I'd post it here in case anyone else might find it helpful.
SYNOPSIS: Script will move files from one DrivePool to another according to FIFO policy
REQUIRED INFRASTRUCTURE: The expected layout is a DrivePool consisting of two DrivePools, one magnetic and one solid state.
The main variables are pretty obviously documented. I added the file archive limit for people like me who also run SnapRAID Helper. That way the script doesn't trip the 'deleted' file limit (I'm assuming moves would trip it, but I didn't actually test it).
Warning, I've obviously only tested this on my system. Please test this extensively on your system after you have ensured good backups. I certainly don't expect anything to go wrong, but that doesn't mean that it can't.
The code is full of on-screen debugging output. I'm not a great coder, so if I've done anything wrong, please let me know. I've posted the code here so that you can't C&P it into a script of your own, since Windows can be annoying about downloaded scripts. Please let me know if you have any questions.
Set-StrictMode -Version 1 # Script drivePoolMoves.ps1 <# .SYNOPSIS Script will move files from one DrivePool to another according to FIFO policy .DESCRIPTION The script can be set to run as often as desired. The expected layout is a DrivePool consisting of two DrivePools, one magnetic and one solid state. .NOTES Author : fly (Zac) #> # Number of files to move before rechecking SSD space $moveCount = 1 # Path to PoolPart folder on magnetic DrivePool drive $archiveDrive = "E:\PoolPart.xxxxx\Shares\" # Path to PoolPart folder on SSD DrivePool drive $ssdSearchPath = "F:\PoolPart.xxxxx\Shares\" # Minimum SSD drive use percent. Below this amount, stop archiving files. $ssdMinUsedPercent = 50 # Maximum SSD drive use percent. Above this amount, start archiving files. $ssdMaxUsedPercent = 80 # Do not move more than this many files $fileArchiveLimit = 200 # Exclude these file/folder names [System.Collections.ArrayList]$excludeList = @('*.covefs*', '*ANYTHING.YOU.WANT*') # Other stuff $ssdDriveLetter = "" $global:ssdCurrentUsedPercent = 0 $fileNames = @() $global:fileCount = 0 $errors = @() Write-Output "Starting script..." function CheckSSDAbove($percent) { $ssdDriveLetter = $ssdSearchPath.Substring(0, 2) Get-WmiObject Win32_Volume | Where-object {$ssdDriveLetter -contains $_.DriveLetter} | ForEach { $global:ssdUsedPercent = (($_.Capacity - $_.FreeSpace) * 100) / $_.Capacity $global:ssdUsedPercent = [math]::Round($ssdUsedPercent, 2) } If ($ssdUsedPercent -ge $percent) { Return $true } Else { Return $false } } function MoveOldestFiles { $fileNames = Get-ChildItem -Path $ssdSearchPath -Recurse -File -Exclude $excludeList | Sort-Object CreationTime | Select-Object -First $moveCount If (!$fileNames) { Write-Output "No files found to archive!" Exit } ForEach ($fileName in $fileNames) { Write-Output "Moving from: " Write-Output $fileName.FullName $destFilePath = $fileName.FullName.Replace($ssdSearchPath, $archiveDrive) Write-Output "Moving to: " Write-Output $destFilePath New-Item -ItemType File -Path $destFilePath -Force Move-Item -Path $fileName.FullName -Destination $destFilePath -Force -ErrorAction SilentlyContinue -ErrorVariable errors If ($errors) { ForEach($error in $errors) { if ($error.Exception -ne $null) { Write-Host -ForegroundColor Red "Exception: $($error.Exception)" } Write-Host -ForegroundColor Red "Error: An error occurred during move operation." Remove-Item -Path $destFilePath -Force $excludeList.Add("*$($fileName.Name)") } } Else { Write-Output "Move complete." $global:fileCount++ # Increment file count, then check if max is hit If ($global:fileCount -ge $fileArchiveLimit) { Write-Output "Archive max file moves limit reached." Write-Output "Done." Exit } Else { Write-Output "That was file number: $global:fileCount" } } Write-Output "`n" } } If (CheckSSDAbove($ssdMaxUsedPercent)) { While (CheckSSDAbove($ssdMinUsedPercent)) { Write-Output "---------------------------------------" Write-Output "SSD is at $global:ssdUsedPercent%." Write-Output "Max is $ssdMaxUsedPercent%." Write-Output "Archiving files." MoveOldestFiles Write-Output "---------------------------------------" } } Else { Write-Output "Drive not above max used." } Write-Output "Done." Exit -
Figured it out. For whatever reason, when using mount points, the PoolPart folders don't get created until something is written to it. That doesn't seem to be the case for drive letters. Is that possible or am I missing something?
-
Thanks.
I did see that KB, but I'm not sure I follow. Where will the Poolpart folders be for the new top level drive?
Hopefully this makes sense:
D: - Top level drive pool |___ E: Magnetic drive pool |____ C:\mount\1\PoolPart |____ C:\mount\2\PoolPart |___ F: Sold state drive pool |___ R:\
edit: Just created the above DrivePool hierarchy. The top level drive was empty and I only saw one PoolPart folder in C:\mount\1. I'm clearly missing something.
-
Am I asking really dumb questions? (Other than this one?)
-
And to follow that up...
If I create a new top level DP drive hierarchy, how do I prime the new top level drive with the contents of DP drive below it?
-
Okay, so I've been thinking more about this again. Here's my plan to have a FIFO SSD cache.
Create a DrivePool of magnetic drives (E:)
Create a DrivePool of solid state drives (F:)
Create a DrivePool containing E: and F: (D:)
I would create placement rules for D: that tell DP to only place files on F:. Then I'd simply write a PS script that would check free space on F: and move files to E: until there was XXXGB of free space.
If it matters, the only reason that E: and F: would exist is so that I don't have to find the underlying disk with the most free space on E: or find the SSD under F: that needs files archived off. I just have my script move things and let DP deal with file placement.
Would this work?
-
I actually assumed that's how they would work, but apparently not.
I have an SSD that I want to add to the pool, but only have new files placed there. As soon as I create the '* > SSD' rule, DP starts filling up the SSD. What am I not understanding?
-
No. I asked about this a few months ago as well.
At the time, I started working on a PowerShell script that would move stuff in FIFO fashion from SSD to platter. At that time, I failed. But I've started working on it again.
-
On 4/17/2019 at 5:44 PM, srcrist said:
I mean, this assumption is faulty, and important data should never be trusted exclusively to a single backup solution--cloud or otherwise. There is no such thing as the perfect backup solution. There is a reason that the 3-2-1 backup rule exists. 3 copies, 2 different types of media, at least 1 offsite location.
If you're relying exclusively on CloudDrive or any other storage solution as your only backup option, you're going to have a bad time. Google rolled back data and corrupted many CloudDrive volumes (we think), and they may do the same in the future. Google Drive is a consumer solution, after all. It is not intended to meet the needs of sensitive business data. CloudDrive is a remarkable product and takes a great number of steps to oversee the integrity of your data, but it can't work miracles. Cloud storage problems exist, and should not be relied upon exclusively.
Using CloudDrive with an enterprise provider like Google Cloud Storage, Amazon S3, or Microsoft Azure should be the first step, at a minimum, to store sensitive and important business data. Not a consumer provider like Google Drive, which is where we saw corruption last month. Linux ISOs and media files are one thing, but I wouldn't even use CloudDrive with Google Drive to store copies of family photographs that I did not have another backup for.
This is all a good point. Not sure why I didn't think of it before. Thank you.
edit: Can I 'backup' the same data from the same CloudDrive to multiple storage providers?
-
I'm confused and concerned. I use CloudDrive to backup important client data for my business, as I assumed it was the perfect backup solution. Are people actually losing data?
-
I often travel for work with simply my ChromeBook. Is it possible for me to access my encrypted files from it?
-
Just checking in to see if any movement was possible on this?
-
While I would guess that there's no real correlation, that was the easy part in PowerShell. This recursively (and surprisingly quickly) searches a directory and grabs the two oldest files:
$fileNames = Get-ChildItem -Path $searchPath -Force -Rec | Where-Object { -not $_.PsIsContainer } | Sort-Object CreationTime | Select-Object -Property FullName -First 2 -
Pretty please?
Just to reiterate my use case: My media server has quite a few users. As new media comes in, it's likely to have a lot of viewers and therefore a lot of I/O. As files get older, fewer and fewer people access them. So I'd love to have an SSD landing zone for all new files, which then get archived off to platters as space requires. It would be fantastic if I could say, once the SSD drive(s) reaches 80% full, archive files down to 50% using FIFO.
-
On 6/21/2018 at 11:32 AM, Jaga said:
Perhaps it's a good feature request for DP's SSD cache too - a FIFO-like system where not all the files come off at once. Just one checkbox and a slider to determine how "full" you want the SSD to stay.
This is turning out much more difficult than anticipated with my limited PowerShell skillset. How would I go about putting in a feature request?
-
20 hours ago, Christopher (Drashna) said:
it should be "dpcmd" specifically.
If you downloaded this, then that would be why it's having issues, since it would be a much older version.
When installing the software, it actually installs dpcmd.exe to c:\Windows\system32, so you can access it from anywhere.
And the command should be "dpcmd ListPoolParts X:", where X: is the drive letter of the pool.
Ah, that's probably it! The link that @Jaga posted above might need to be updated.
http://wiki.covecube.com/StableBit_DrivePool_Utilities#DrivePool_Command_Utility
-
So I'm finally getting around to writing this...
I'm playing with dp-cmd right now and ListPoolParts doesn't seem to work correctly. Does it need to be in a certain directory to work?
c:\Users\Administrator\Desktop>dp-cmd.exe ListPoolParts Error: Method not found: 'CoveUtil.OmniStore.Store CoveUtil.OmniStore.Store.Create(Boolean, Boolean, System.IO.DirectoryInfo, Boolean, CustomSerializer)'. Details: System.MissingMethodException: Method not found: 'CoveUtil.OmniStore.Store CoveUtil.OmniStore.Store.Create(Boolean, Boolean, System.IO.DirectoryInfo, Boolean, CustomSerializer)'. at DrivePoolCmd.DrivePoolTasks.ListPoolParts() at DrivePoolCmd.Program.Main(String[] args)
-
1 hour ago, Jaga said:
DP just passes drive operations to the underlying drives. The files would still be there, and still show in their folders when any app told DP to "go look and see what's there". Until then, it wouldn't know they were part of the pool, and the pool measurement (for balancing and space used) would appear incorrect. But you'd still be able to access/stream the files just fine.
I open Scanner and Drivepool daily to check on status and re-measure the pool, and read SnapRAID logs. Takes all of 3-5 minutes, and gives me peace of mind. If you had powershell scripts moving files to physical drives nightly, just open Drivepool every now and then and tell it to re-measure.
DP does have a command line utility, but I'm unsure if it's compatible with current version(s), or if it allows re-measuring.
Well then, seems I have a weekend project.
Will report back with script in case anyone else would like to use it.
-
21 minutes ago, Jaga said:
DP wouldn't know until you told it to measure pool space. But it wouldn't care either, especially if you (mostly) tried to keep space balanced with the PS scripts. Just re-measure the pool manually once in a while and it'll be happy.
I suspect the FIFO feature as a request wouldn't be hard to implement. Can always keep fingers crossed.
If I ran my script nightly, is there a way for it to tell DP to examine the pool again? Otherwise, if I'm understanding you correctly, it sounds like the files would appear unavailable.
-
If I create the PowerShell script, and it moves files to different drives, are you saying that DP is alerted to the change instantly and will update it's physical location on the virtual drive? If so, I'd think it would be relatively trivial for my script to determine the drive with the most space and dump it there!
And yes, ideally that would be great. Probably choose a max used and a min used. Then once max is hit, dump down to min, based on FIFO. That would be perfect for my use case.
-
10 minutes ago, Jaga said:
Don't calculate parity on the SSD drive. Or.. be prepared to have SnapRAID re-calculate a large amount of it's parity every time your SSD empties out. It's a volatile volume that you're trying to do parity on, which is naturally going to lead to a lot of activity and re-calcs.
If you really want to do it right, make a RAID 1 mirror with multiple SSDs, then use that as the Drivepool SSD cache, and don't do parity on it with SnapRAID. You'll still have temporary redundancy in the mirror.
Other options: speed up your pool by getting faster drives. Install a fast SSD or NVMe along with Primocache as a L2 caching option against the Pool. Or manually manage what you keep on the SSD and what gets moved to the pool.
Lots of ways to handle it. Drivepool wasn't setup to do what you're trying to do with SnapRAID, as SnapRAID likes mostly static archives (or fast disks to work with when there are large numbers of changes).
I'm still unsure why even a 5400 RPM spindle drive can't deliver information fast enough to stream a 4k UHD movie, for example. My WD Reds @ 5400RPM can easily read/write fast enough to saturate my Gigabit network. Is there a reason you simply have to have the latest content on a faster SSD?
I'd like for newest content to be on SSDs, since it's the content most likely to be accessed. If I could get FIFO, then it wouldn't be that upsetting to SnapRAID. What if I just did this 'manually' with PowerShell? I would assume that DrivePool wouldn't be very happy if I just placed a file on one of the other drives PoolPart folder, but it's worth asking.
I suppose a RAID 1 would work, but somewhat less than ideal since it's basically a 50% storage penalty.
CloudDrive.UI.exe detected as malware by multiple antivirus engines
in General
Posted
Just an update, I got a response from @Christopher (Drashna). According to him, this is a false positive. I sincerely hope that is the case, and if so, how do we get it cleared on all these sites?