A few questions about measuring.....

[msq]

There is a couple of issues and questions about measuring.

 

Why it is not optimized / cached / smarter?

Case 1:

If there is a pool of 2 or more drives and one is disconnected for whatever reason - once the drive is back entire pool is measured. If that is large, cloud based pool with a lot of small files - a short interruption in connectivity triggers hours of measuring.

Would that be possible to cache results of previous measure run and measure only the drive that was reconnected?

Case 2:

Four cloud based drives CLD1, CLD2, CLD3 and CLD4 are coupled in two pools: CLD1+2 and CLD3+4. Those two are paired again as a pool CLD1+2&CLD3+4. And this pool is coupled with physical drive in the highest level pool: HDD1&CLD1+2&3+4. Apart from CLD1+2 and CLD3+4 - every other pool is set to duplicate all files.

Now - when measure is running it is starting from the highest level, then is measuring middle tier and two on the lowest level. So files on the cloud based drives are actually measured three times.

Would that be possible to start from the lowest level, then cache and re-use data it on higher levels?

Why it is so sensitive?

Case 3:

Let's say there is more than one pool running measure. As far as I can see this is slightly misleading - as only one pool can be measure at a time - other pools are actually awaiting. If for whatever reason one drive in the pool that is NOT currently effectively being measured disconnected and reconnected - measure on the other drive is restarted. Why..?

And is it possible by change in configuration to allow more than one measure thread to run?

Would that be possible to set specific measure order for pools?

Case 4 (feat. CloudDrive):

As mentioned above - short interruption in internet connectivity can disconnect all cloud drives, enforce manual intervention (re-connect) and massive re-measure.

Would that be possible to make CloudDrive drives a bit more offline-friendly and disconnection resistant ? In cases when there is no pool activity from users?

And can we receive some feature that will automatically re-connect cloud drives, please?

Thanks

 

EDIT: typos, wording

[Christopher (Drashna)]

23 hours ago, msq said:

Case 1:

If there is a pool of 2 or more drives and one is disconnected for whatever reason - once the drive is back entire pool is measured. If that is large, cloud based pool with a lot of small files - a short interruption in connectivity triggers hours of measuring.

Would that be possible to cache results of previous measure run and measure only the drive that was reconnected?

The problem is that the contents of the disk may have changed.  Measuring makes sure that the statistics are accurate. 

For the CloudDrives, yeah, it can take a while.  

On 5/13/2018 at 11:09 AM, msq said:

Case 2:

Four cloud based drives CLD1, CLD2, CLD3 and CLD4 are coupled in two pools: CLD1+2 and CLD3+4. Those two are paired again as a pool CLD1+2&CLD3+4. And this pool is coupled with physical drive in the highest level pool: HDD1&CLD1+2&3+4. Apart from CLD1+2 and CLD3+4 - every other pool is set to duplicate all files.

 Now - when measure is running it is starting from the highest level, then is measuring middle tier and two on the lowest level. So files on the cloud based drives are actually measured three times.

Would that be possible to start from the lowest level, then cache and re-use data it on higher levels?

 Why it is so sensitive?

Yes and no.  The pooled data is in PoolPart folders.  So the information may not be completely correct for higher level pools, since all of the data may not be pooled data.

But I've flagged this: https://stablebit.com/Admin/IssueAnalysis/27821

On 5/13/2018 at 11:09 AM, msq said:

Case 3:

Let's say there is more than one pool running measure. As far as I can see this is slightly misleading - as only one pool can be measure at a time - other pools are actually awaiting. If for whatever reason one drive in the pool that is NOT currently effectively being measured disconnected and reconnected - measure on the other drive is restarted. Why..?

And is it possible by change in configuration to allow more than one measure thread to run?

Would that be possible to set specific measure order for pools?

Hmm, not sure about this, actually. 

Though, probably a performance/driver related reason. 

https://stablebit.com/Admin/IssueAnalysis/27822

On 5/13/2018 at 11:09 AM, msq said:

 Case 4 (feat. CloudDrive):

As mentioned above - short interruption in internet connectivity can disconnect all cloud drives, enforce manual intervention (re-connect) and massive re-measure.

 Would that be possible to make CloudDrive drives a bit more offline-friendly and disconnection resistant ? In cases when there is no pool activity from users?

This is by design.  Loss of internet connectivity translates into I/O issues.  And Windows is very sensitive to I/O issues.  So, we disconnect the drive, rather than allowing the system to lock up and potentially lose data.   

There are some settings that may help:
http://wiki.covecube.com/StableBit_CloudDrive_Advanced_Settings

"CloudFsDisk_MaximumConsecutiveIoFailures", for instance. But some of the IoManager may help. 

 

As for "offline friend".... sure, if you don't mind potentially corrupt data..... Seriously.  
There is a "mount disconnected option: https://stablebit.com/Support/CloudDrive/Manual?Section=Mounting%20a%20Disconnected%20Drive

 

 

On 5/13/2018 at 11:09 AM, msq said:

And can we receive some feature that will automatically re-connect cloud drives, please?

No.  You wouldn't want this.  This can cause issues, it could cause serious performance/reliability issues?  
Like would you use a drive long term that would randomly and frequently disconnect from the USB hub?  

 

[msq]

22 hours ago, Christopher (Drashna) said:

On 5/13/2018 at 7:09 PM, msq said:

And can we receive some feature that will automatically re-connect cloud drives, please?

No.  You wouldn't want this.  This can cause issues, it could cause serious performance/reliability issues?  
Like would you use a drive long term that would randomly and frequently disconnect from the USB hub?  

No, I wouldn't.

But what I meant by "feature that will automatically re-connect cloud drives" is some functionality that will perform exactly the same action as user from the front end by clicking 'Retry' - that's all. Every x minutes if drive gets disconnected. If there will be no connectivity - nothing will happen. But if the connectivity will be back - drive will be just reconnected.

I'm on a rather crappy internet here and sometimes cloud drives get disconnected. All I have to do is click 'Retry' and it is connected again. I'm asking to get this 'Retry' action automated. IUt could be turned off by default and hidden deep into advanced json setup - that's fine.

If that is gui action someone will finally write a script that will mimic such click

[Christopher (Drashna)]

I'm sorry to hear about your ISP quality...

 

That said, you want to modify the "CloudFsDisk_MaximumConsecutiveIoFailures" option then.  That should help to prevent it from disconnecting in the first place.  Set it to 5 or 8, and see if that helps. 

[msq]

On 5/15/2018 at 9:07 PM, Christopher (Drashna) said:

That said, you want to modify the "CloudFsDisk_MaximumConsecutiveIoFailures" option then.  That should help to prevent it from disconnecting in the first place.  Set it to 5 or 8, and see if that helps. 

Thank you - good point. I had it set to something non-default already but most likely during some re-installation this has been lost.

But hey - I just noticed something in change log

Thank you once again and will test that next few days!

 

.988
* Added an option to automatically retry mounting drives. See "Automatically retry mounting" under Options.
	- Drives that were unmounted due to an incorrect or missing encryption key, an unknown storage provider, or 
	  damaged metadata will not retry mounting automatically.

 

[Christopher (Drashna)]

Yup.  Alex saw your post (i'm assuming, it was one of them like this) and decided to add it.