Unable to remove clouddrive(s) from a drivepool because I lost write access

[dan66215]

I have a drivepool made up of 3 clouddrives. Two of them are connected to 2 separate Onedrive accounts and the 3rd is with a different provider. There is no duplication on this drivepool.

The 2 drives on Onedrive had their quotas reduced while I wasn't looking, so they both are over-quota and as such are limited to read-only. I added the 3rd drive to the pool expecting that I could remove the 2 Onedrive clouddrives and eveything would move to the 3rd clouddrive.  Unfortunately, both Oneddrive clouddrives have 100's of GBs on the cache drive pending upload. This makes it so I can't remove either of the drives because drivepool wants to write the removed files to the other Onedrive clouddrive. I thought clicking the pause button on the drive would force the files to the new drive, but the pause doesn't work - it lights up for a split second then goes away.
I've also tried detaching the cloud drives, but get a message that "they'll detach as soon as we upload that 100 GB that's pending upload" or something like that.

I'm kind of stuck. Any thoughts on a way to get these read-only drives out of the pool so that the pool is writeable again?

 

[Shane]

Using the Ordered File Placement balancer, to require each removed drive's content goes only to the new drive, might work but I'm not sure in this case so I think I'd suggest skipping to the bigger hammer in the toolbox:

1. Open a command prompt run as administrator; use the "dpcmd ignore-poolpart" command to disconnect each of the old poolparts from the pool (from the pool's perspective it's like you've just manually unplugged a physical drive). It also marks them with an ignore tag so they can't return to the pool unless the "dpcmd unignore-poolpart" command is used on them, but that's moot since we'll be removing them.
   • usage: dpcmd ignore-poolpart p:\ foldername
   • where p:\ is the pool drive root and foldername is the hidden poolpart folder on/representing the drive you want to disconnect from the pool.
2. Use the Remove option in the GUI to remove the "missing" disks from the pool so that the pool is writable.
3. Manually copy the content of the ignored poolparts on the old cloud drives into the pool (and thus to the new cloud drive).

(note for future readers: if you have nested pools or duplication, it's a little different, see below)

[dan66215]

Just so we're using the same terms, please see attached diagram of my setup.  Given my specifics, 
for #1 I believe the commands should be:
  dpcmd ignore-poolpart L:\PoolPart.a5d0f5ec-ca52-48e9-a975-af691ace6a16
  dpcmd ignore-poolpart L:\PoolPart.4d2b0ebb-14f2-4d9d-a192-641de224b2cb

Is this correct?

 

 

[Shane]

Could you please post the output of the following admin prompt commands?

• dpcmd list-poolparts L:\
• dir /ah /b /s L:\poolpart.*
• dir /ah /b /s M:\poolpart.*
• dir /ah /b /s K:\poolpart.*
• dir /ah /b /s F:\poolpart.*

And a screenshot of your DrivePool GUI with the L:\ pool selected, showing its Pooled Disks?

[dan66215]

C:\Users\Administrator>dpcmd list-poolparts L:\

dpcmd - StableBit DrivePool command line interface

Version 2.3.7.1570

 + Pool ID 'c9212568-6742-460a-aff1-705893e40c16':
  - '\\?\GLOBALROOT\Device\HarddiskVolume40\PoolPart.a5d0f5ec-ca52-48e9-a975-af691ace6a16' [Device 13]
  - '\\?\GLOBALROOT\Device\HarddiskVolume42\PoolPart.3ac52a32-09ed-42bd-bc40-3e08f315b9b0' [Device 14]
  - '\\?\GLOBALROOT\Device\HarddiskVolume46\PoolPart.4d2b0ebb-14f2-4d9d-a192-641de224b2cb' [Device 16]

C:\Users\Administrator>dir /ah /b /s L:\poolpart.*
L:\PoolPart.2fa37324-d52b-431d-8eaa-3d7175d11cd4

C:\Users\Administrator>dir /ah /b /s M:\poolpart.*
M:\PoolPart.a5d0f5ec-ca52-48e9-a975-af691ace6a16
M:\PoolPart.a5d0f5ec-ca52-48e9-a975-af691ace6a16\PoolPart.2fa37324-d52b-431d-8eaa-3d7175d11cd4

C:\Users\Administrator>dir /ah /b /s K:\poolpart.*
K:\PoolPart.4d2b0ebb-14f2-4d9d-a192-641de224b2cb
K:\PoolPart.4d2b0ebb-14f2-4d9d-a192-641de224b2cb\PoolPart.2fa37324-d52b-431d-8eaa-3d7175d11cd4

C:\Users\Administrator>dir /ah /b /s F:\poolpart.*
F:\PoolPart.3ac52a32-09ed-42bd-bc40-3e08f315b9b0
F:\PoolPart.3ac52a32-09ed-42bd-bc40-3e08f315b9b0\PoolPart.2fa37324-d52b-431d-8eaa-3d7175d11cd4

C:\Users\Administrator>

 

 

[Shane]

Ah, okay, L is itself being used as a drive for another pool. Going to do some testing and see if I avoid kicking the can down the road, so to speak, or need to rewrite my 1-2-3.

Question: I understand the onedrive accounts are over-quota and thus in a read-only state, but are the M and K drives still technically writable (in the sense that they can still accept files into whatever room remains in their upload caches) or are they (and thus the pool) also read-only?

[dan66215]

Yes, the overall structure is that I have a main pool (P:) that is duplicated 2x and made up of 2 nested pools - a cloud pool (L) and a local pool (Q) - which have no duplication.

The only reason I noticed that the Onedrives were over quota and read-only was because all of a sudden I wasn't able to write large files to P: (that's a separate topic that I will ask about when this is sorted.)

To answer your question, I just created a 1kb txt file on each of the drives, M:, K:, L:, P: which succeeded, but was very slow to write. If I try a larger file it just times out.

 

 

 

[Shane]

P being duplicated across L and Q is great, as presuming P's duplication is up to date and Q has no issues of its own you should be able to proceed with:

• dpcmd ignore-poolpart L:\ PoolPart.a5d0f5ec-ca52-48e9-a975-af691ace6a16
• dpcmd ignore-poolpart L:\ PoolPart.4d2b0ebb-14f2-4d9d-a192-641de224b2cb

(So what you had but with a space between the L:\ and the Poolpart).

And then step 2 is unchanged; removing the "missing" drives from pool L, which should make L usable again.

And for step 3, if all your content is in P with nothing other than the nested poolpart (and any system folders) in L, M and K then you should just be able to select pool P and Cog icon -> Troubleshooting -> Recheck Duplication and it should proceed to fill back in L (and thus F) from Q rather than you having to manually copy your content from M and K's poolparts (which can get a little fiddly with nesting).

[dan66215]

Great. Thanks for the very clear instructions.
I completed step 1 and 2 without issue.

As for step 3, the GUI for P: said that the "duplication was inconsistent", but I suspect that was due to the issues with the cloud pool (L:).  I started a "recheck duplication" and it's currently duplicating.  Unfortunately F: - the remaining cloud drive in the cloud pool (L:) - is being I/O throttled because it shares the cache drive with M: & K:, which have filled it up.

Is there any danger in deleting the cache for M: & K:? And how would I go about that?

I'll answer my own question about the danger - which would be if anything from P: was written to L: without being duplicated to Q, after the Onedrives became quota-restricted, then the only place the files may be would be the cache drive. Correct? And in that case, can I manually move the cache files for M: & K: to another drive to give F: some headroom?

 

[Shane]

"I'll answer my own question about the danger - which would be if anything from P: was written to L: without being duplicated to Q, after the Onedrives became quota-restricted, then the only place the files may be would be the cache drive. Correct?"

Correct. In theory if you're using real-time x2 duplication for everything in P, then Q should be up to date and the disconnection of M and K from L should allow P to automatically backfill L from Q (as and when it has room). In practice you could verify this by manually comparing the contents of the nested poolparts of P that are in M and K with the one in Q and copy any missing files (excluding anything in any folder named ".covefs", "$RECYCLE.BIN" and "System Volume Information") to where they should be in the latter. Fiddly but possible.

Specifically, you would compare the content of "M:\PoolPart.a5d0f5ec-ca52-48e9-a975-af691ace6a16\PoolPart.2fa37324-d52b-431d-8eaa-3d7175d11cd4\" with that of "Q:\PoolPart.2fa37324-d52b-431d-8eaa-3d7175d11cd4\", and the content of "K:\PoolPart.4d2b0ebb-14f2-4d9d-a192-641de224b2cb\PoolPart.2fa37324-d52b-431d-8eaa-3d7175d11cd4\" with that of "Q:\PoolPart.2fa37324-d52b-431d-8eaa-3d7175d11cd4\", to see if there's anything in M or K that is not in Q (there will, of course, be content in Q that is not in M or K, because Q should be at least the sum of M & K & L). Reminder, you must exclude the ".covefs", "$RECYCLE.BIN" and "System Volume Information" folders from the comparisons. If you don't already have a useful GUI tool for comparing folder trees, I can suggest TreeComp.

"And in that case, can I manually move the cache files for M: & K: to another drive to give F: some headroom?"

Unfortunately I don't know of a way to manually move a cloud drive's local cache files (I imagine/hope it's theoretically doable but I've never had to figure out how), and the GUI requires the upload cache to be empty (which we can't do for M and K being over-quota) to complete a detach before it can be safely re-attached to a different local drive. The alternatives would be detaching F (since it's the drive that's writable) instead to re-attach its cache to a different drive (one with plenty of room) or destroying the M and K cloud drives via the GUI (which I imagine you won't want to do if you're worried there's any unique files sitting in their caches).

[dan66215]

5 hours ago, Shane said:

The alternatives would be detaching F (since it's the drive that's writable) instead to re-attach its cache to a different drive (one with plenty of room) or destroying the M and K cloud drives via the GUI (which I imagine you won't want to do if you're worried there's any unique files sitting in their caches).

Oh that's interesting, I didn't know detach/reattach allowed for a change in cache location. It sounds like that's the path forward.  Is it as simple as it sounds?
Given what you said about real-time replication I'm fairly confident everything is on Q, as I've had duplication on forever (well before the quota fiasco). 

Regarding destroying the M & K cloud drives via the GUI - I suspect eventually, once I confirm everything is squared away, that I will have to do this right? I've noticed that even though we did the ignore, they're still both constantly and hopelessly attempting to upload files to Onedrive.

 

[Shane]

"detach/reattach allowed for a change in cache location. It sounds like that's the path forward.  Is it as simple as it sounds?"

Pretty much. When re-attaching a drive (or creating a new one) expand the "Advanced Settings" triangle icon and you should be able to choose the drive it uses for local cache (note: I'd never pick the boot drive if at all feasible, despite the manual's screenshot; I'd pick some other physical drive).

"Regarding destroying the M & K cloud drives via the GUI - I suspect eventually, once I confirm everything is squared away, that I will have to do this right? I've noticed that even though we did the ignore, they're still both constantly and hopelessly attempting to upload files to Onedrive."

Correct; the ignore just disconnected them from the pool, it doesn't affect their own nature as cloud drives.

[dan66215]

The detach was successful. I had to do a reboot between the detach and the re-attach.  However when I re-attached F, the configuration screen came up, I selected another physical drive on which to put the cache. That was some time ago and it has not yet appeared in the GUI.  I believe it's adding it, but maybe very slowly, as the Clouddrive app is using about 40% of the CPU. The other detached clouddrives (formerly M & K) are visible in the GUI but have lost their drive letters.  Also the Drivepool GUI will not come up, though I have access to all the lettered drives via windows explorer and command prompt.

I guess I'll give it overnight and hope that F: gets re-attached and that is the cause of the Drivepool GUI not responding.

 

[dan66215]

After running all night the clouddrive (F) reattached and I can see it's using the new physical drive I chose for the cache.  The Drivepool GUI will still not come up. As earlier, I have access to all the lettered drives via windows explorer and command prompt.  Because I can't access the GUI, I can't kick off the balancing to get the cloudpool loaded up with the duplicates from the local pool.  I let it load overnight and all that happens is the drivepool GUI starts to load with a yellow bar in the center saying "starting service". It ran like that for 23 hours.
I killed it, rebooted and tried repairing by running the drivepool setup program. It hangs on "processing: Microsoft .NET Framework 4.8" and has been for about an hour.  Do you think what we did has somehow corrupted the drivepool app? Any thoughts on how to best resolve? I sure don't want to lose all my drivepool settings.

Thanks.

 

[Shane]

I can't say with absolute certainty that it couldn't have, but on the other hand all we've done outside the GUI (in terms of actual changes) so far is used dpcmd to ignore those poolparts, unless you manually copied anything between the poolparts (which should not cause DrivePool GUI's to hang anyway AFAIK).

Your settings are the *.json files in your C:\ProgramData\StableBit DrivePool\Service\ and C:\ProgramData\StableBit DrivePool\Service\Store\Json\ folders in case you wish to back those up.

DrivePool does a nightly duplication consistency check independent of balancing; I believe it will also automatically attempt to correct any duplication issues found rather than perform it as part of balancing.

How far along is CloudDrive in uploading F's cache to the cloud?

Did you try TreeComp (or similar) to look for differences between Q and L?

Regarding the .NET processing, I'm not sure. I would be inclined to wait until F is finished uploading and checking whether your content in Q and L match before proceeding with anything that might break drivepool (further).

[dan66215]

>>unless you manually copied anything between the poolparts (which should not cause DrivePool GUI's to hang anyway AFAIK).<<
I did not.

Thanks for the settings location. Got that backed up.

>>DrivePool does a nightly duplication consistency check independent of balancing; I believe it will also automatically attempt to correct any duplication issues found rather than perform it as part of balancing.
How far along is CloudDrive in uploading F's cache to the cloud?<<

It does not appear to be duplicating or uploading from the cache. The size of F (and therefore L:) on disk and in the Clouddrive GUI has not changed since reattaching it (140GB).  Of course I can't tell if the cache is empty or full from looking at the cache files on disk, but the GUI doesn't show anything to be uploaded (attached)

>>Did you try TreeComp (or similar) to look for differences between Q and L?<<
I did not because there's an obvious 2.86 TB difference between Q & L.

>>Regarding the .NET processing, I'm not sure. I would be inclined to wait until F is finished uploading and checking whether your content in Q and L match before proceeding with anything that might break drivepool (further).<<
I did let the install/repair continue running and it eventually completed with a "Repair failed - One or more issues caused the setup to fail. Please fix the issues and then retry setup.  0x80004004 - Operation aborted"

It really seems like the issue is drivepool not running "properly". It's running enough to mount the pool drives and I can access them, but not enough to do the balancing or maintenance tasks or run the GUI.

Another related symptom, though I don't know how it's related, is that when I run windows explorer (it defaults to opening "this PC") it hangs on "This PC" - like it can't see all the drives. However if I interrupt it and chose a path manually, then it shows the location normally.

[Shane]

>>It really seems like the issue is drivepool not running "properly". It's running enough to mount the pool drives and I can access them, but not enough to do the balancing or maintenance tasks or run the GUI.

Yes, and I'm not sure how to proceed here. I'm also wondering why M and K lost their drive letters; as I'm understanding it those should be via CloudDrive, not DrivePool.

Perhaps try a repair/reinstall of Microsoft .NET 4.8, just in case that's an issue, then try reinstalling DrivePool again (the latest release version is 2.3.8.1600 btw)?

But given Explorer is now hanging on This PC you might need to open a support ticket with StableBit to get their help with this.

[dan66215]

On 7/17/2024 at 8:28 AM, Shane said:

Correct; the ignore just disconnected them from the pool, it doesn't affect their own nature as cloud drives.

On 7/17/2024 at 9:36 PM, dan66215 said:

The other detached (disconnected not detached) clouddrives (formerly M & K) are visible in the GUI but have lost their drive letters.  Also the Drivepool GUI will not come up, though I have access to all the lettered drives via windows explorer and command prompt.

I went through this thread again from the beginning and noted a couple of clues and discrepancies. M & K did eventually get their drive letters back, just like F. However, when I said I had access to all the lettered drives via explorer and command prompt, I obviously (now) wouldn't have been able to check M & K because they didn't have drive letters. After the drive letters returned, I am not able to access them via windows explorer or command line. I suspect this is what's hanging up explorer on "This PC" - because it knows M & K are there but can't access them. I also wonder if this is what is preventing the drivepool GUI from loading completely.

 

[dan66215]

I also got a little brave/creative/foolish and had the thought that maybe I could kick off the duplication process on P manually via command line since the GUI wouldn't come up and it wasn't duplicating on it's own from the nightly job.

So I did a dpcmd check-pool-fileparts P:\ 1 which completed successfully. When I checked F in clouddrive, the usage has increased and there's 1TB+ queued. So it looks like it's duplicating. That's one good thing so far!