Jonibhoni

Sorry, I didn't mention: Upload verification was disabled. I opened a ticket.

Hi, isn't there some priorization taking place under the hood when deciding about which chunk to upload first? I just did few experiments with Google Cloud Storage and 100 MB chunk size, cache size 1 GB (initially empty except pinned metadata and folders), no prefetch, latest public CloudDrive: a) pause upload b) copy a 280 MB file to the cloud drive c) resume upload With this sequence, the whole plan of actions seems to be well defined before the actual transfer starts. So lot's of opportunity for CloudDrive for batching, queueing in order etc. Observing the "Technical Details" window for the latest try, the actual provider I/O (in this order) was: Chunk 3x1 Read 100MB because: "WholeChunkIoPartialMasterRead", length 72 MB Chunk 3x1 Write 100MB because: "WholeChunkIoPartialMasterWrite", length 72 MB Chunk 4x1 Write 100MB because: "WholeChunkIoPartialMasterWrite", length 80 MB Chunk 10x1 Read 100MB because: "WholeChunkIoPartialMasterRead", length 4 kb, + 3 "WholeChunkIoPartialSharedWaitForRead" with few kb (4 kb, 4 kb, 8 kb) Chunk 10x1 Write 100MB because: "WholeChunkIoPartialMasterWrite", length 4 kb, + 3 "WholeChunkIoPartialSharedWaitForCompletion" with few kb (4 kb, 4 kb, 8 kb) Chunk 0x1 Read 100MB because: "WholeChunkIoPartialMasterRead", length 4 kb Chunk 0x1 Write 100MB because: "WholeChunkIoPartialMasterWrite", length 4 kb Chunk 4x1 Read 100MB because: "WholeChunkIoPartialMasterRead", length 23 MB Chunk 4x1 Write 100MB because: "WholeChunkIoPartialMasterWrite", length 23 MB Chunk 5x1 Write 100MB, length 100 MB Chunk 6x1 Write 100MB because: "WholeChunkIoPartialMasterWrite", length 11 MB Chunk 10x1 Read 100MB because: "WholeChunkIoPartialMasterRead", length 16 kb, + 4 "WholeChunkIoPartialSharedWaitForRead" with few kb (4 kb, 4 kb, 4 kb, 12 kb) Chunk 10x1 Write 100MB because: "WholeChunkIoPartialMasterWrite", length 16 kb, + 4 "WholeChunkIoPartialSharedWaitForCompletion" with few kb (4 kb, 4 kb, 4 kb, 12 kb) So my questions / suggestions / hints to things that shouldn't happen (?) in my opinion: The chunk 10x1 is obviously just for filesystem metadata or something; it's few kb, for which a chunk of 100 MB has to be downloaded and uploaded - so far so unavoidable (as described in here). Now the elephant in the room: Why is it downloaded and uploaded TWICE? The whole copy operation and all changes were clear from the beginning of the transmission (that's what I paused the upload for until copying completely finished). Ok, may be Windows thought to write some desktop.ini or stuff while CloudDrive was doing the work. But then why did it have to be read again and wasn't in the cache on the second read? Caching was enabled with enough space, also metadata pinning was enabled, so shouldn't it then be one of the first chunks to cache?. Why is chunk 4x1 uploaded TWICE (2 x 100MB) with 80 MB productive data the first time and 20 MB the second?! Isn't this an obviuous candidate for batching? If chunk 5x1 is known to be fully new data (100 MB actual WORTH of upload), why does it come after 3x1, 4x1 and 10x1, which were all only "partial" writes that needed the full chunk to be downloaded first, only to write the full chunk back with only a fraction of it actually being new data. Wouldn't it be more efficient to upload completely new chunks first? Especially the filessystem chunks (10x1 and 0x1 I'm looking at you) are very likely to change *very* often; so prioritizing them (with 2x99 MB wasted transfered bytes) over 100 MB of actual new data (e.g. in chunk 5x1) seems a bad decision for finishing the job fast, doesn't it? Also, each upload triggers a new file version of 100 MB at e.g. Google Cloud Storage, which get's billed (storage, early deletion charges, ops...) without any actual benefit for me. So regarding network use (which is billed by cloud providers!): Naive point of view: I want to upload 280 MB of productive data Justified because of chunking etc.: 300 MB download (partial chunks 0x1, 3x1, 10x1) + 600 MB upload (4x1, 5x1, 6x1, 0x1, 3x1, 10x1) Actually transfered in the end: 500 MB download + 800 MB upload. That's 66% resp. 33% more than needed?

Wouldn't it be more efficient to just load the parts of the chunk that were not modified by the user, instead of the whole chunk? One could on average save half the downloaded volume, if I'm correct. Egress is expensive with cloud providers. 😅 Think: I modify the first 50 MB of a 100 MB chunk. Why is the whole 100 MB chunk downloaded just to overwrite (= throw away) the first 50 MB after downloading?

I remember there to be some message category in the Gear Icon -> Troubleshooting -> Service Log... for which you have to set the Tracing level to something lower ("Information" or "Verbose"). Then a text log item will pop up for each file touched during balancing (which is a lot). Unfortunately I don't remember the exact category and tracing level needed. :/

Well it also worked fine for me with either Bitdefender or read striping running, but not with both in combination, so you cannot really say that either of them causes it alone. It seemed to be the combination of read striping reading files in stripes and antivirus checking them on-the-fly. Specifically the curruption that was taking place was a duplication of exactly the first 128 MB of the file, which seems very coincidental and un-random. I would suspect that either Bitdefender scans files in chunks of 128 MB or DrivePool stripes them in 128 MB chunks. And somewhere on the clash of on-access-scanning of Bitdefender (and Windows Defender?) and read striping in the CoveFS driver (and maybe Windows file caching playing along, too), something unexpected happens.

Good to hear, for your case. When I measured it, read striping didn't give me any practical performance increase anyway, so it's not so much a problem to turn it off. But that the file corruption problem for you appears with Windows Antivirus, too, is actually a new level of that problem! I think you should file a bug report to the support, mentioning this! I mean, silent file corruption is somewhat a nightmare for people using what not technology to have their files stored savely. I had extensively documented the kind of corruption (what conditions, which files, even the kind of corruption in the file format) that happened for me back then, but the StableBit guys somewhat refused to do anything because they saw it as a problem on Bitdefender side. Which is partly disappointing but also partly understandable, because Bitdefender probably really does some borderline stuff. But if now it happens with the regular Windows Antivirus ... they should take it serious! I mean DrivePool + Read Striping + Windows Defender is not really an exotic combination, it's pretty much the most ordinary one. It's bits here at stake, StableBit! @Christopher (Drashna)

Does the problem still appear if you disable Read Striping (Manage Pool -> Performance -> Read striping)? And do you have an anti virus software active that checks files on access? I have an issue with random file corruption when using the combination of both (Bitdefender + Read Striping), which is why I disabled read striping.

There is some confusion around the forum whether DrivePool is the culprit or not.. It's a longtime open issue. If you search the forum you will find some older topics which report the same issue, with no straightforward solution, afaik. https://community.covecube.com/index.php?/topic/4734-drive-sleep/ https://community.covecube.com/index.php?/topic/48-questions-regarding-hard-drive-spindownstandby/ https://community.covecube.com/index.php?/topic/5788-hard-drives-wont-sleep-always-running-hot/ https://community.covecube.com/index.php?/topic/7157-i-have-to-ask-what-is-the-deal-with-drives-and-sleep-or-lackthereof/ If you solve your case, maybe be so kind and tell me your solution.

Just out of curiousity: Does the problem still appear if you disable Read Striping (Manage Pool -> Performance -> Read striping)? And do you have an anti virus software active that checks files on access? I once had an issue with the combination of both.

I won't judge about your procedure. Seeding usually involves shutting the service down, it may work without, or it may produce errors; I cannot say. I would stick closely to the seeding tutorial on the knowledge base (linked in my first post). You're probably right in that there is no (easy) method of listing non-duplicated data. Though I remember people on the forum somewhere had something with PowerShell and the official command-line tool that offered something similar like that, locating files and so on... Anyway, the functionality that you describe here is actually (sorry ^^) in the very official easy clickable GUI solution I pointed out before. If you simply remove a drive in the GUI, DrivePool will offer an option... ...which will exactly force the fast moving of unduplicated files from the to-be-removed drive, and then recreating the duplicated files from their existing copies in the pool later after the removal.

I don't think you can do it that way. I guess DrivePool would not recognize the moved PoolPart folder because of its naming scheme. But I'm not sure. The so called seeding procedure usually works by moving the contents (!) of existing (!) PoolPart folders (with DrivePool service shut down before), but it's also an advanced procedure with potential data loss if things go wrong. Honestly, why don't you just add the new drive to the pool and then remove the existing one with the GUI? It would migrate the data as you wish and probably not be much less efficient than doing it manually.

DrivePool doesn't have the concept of a "primary" and a "secondary" pool. So there is no "only duplicates" in the meaning of "the copy of the original" because there is no "original" - all two copies of a 2x-duplicated file are valid and equal. DrivePool fetches any of them when you access it and just makes sure that there are always 2x the file stored physically within your pool, without any of them being superior. So much for the theory. In practice, what you want to do ... ... is perfectly doable. You create a third, "parent" pool that contains pool 1 and pool 2 (and nothing else). In this parent pool, you set duplication for the whole pool to 2x. (In the lower pools 1 and 2, you disable duplication). Then you move all your files to the parent pool and DrivePool will distribute one copy of each file on pool 1 and one copy on pool 2. If you simply move the files from your original pool (now "pool 1") to the parent pool, you will be fine, you just have to wait for all the 20 TB being moved. If you are more adventurous, you can also go through a procedure referred to as "seeding" (https://wiki.covecube.com/StableBit_DrivePool_Q4142489) which will skip half of the disk writing work (but is more difficult and risky).

Probably the internal DrivePool metadata ("[METADATA]"). It's locked to 3x-duplication by default.

Afaik, yes. To my understanding, SSD optimizer cache functionality works with "real-time file placement limits", which kind of says where files are allowed to be put on *creation* / first write. The regular file placement rules kick in on (subsequent) balancing passes. So I would remove the SSD from file placement rules, if you don't want files to be archived there. (but I'm not using SSD optimizer myself) Also, you could just try to temporarily deactivate SSD optimizer, to see if a subsequent re-balancing pass fixes the thing. Then we at least know who is the culprit. (The log time is relative time since service start. So all fine here. )

Strange, hmm... How is your SSD Optimizer set-up? Do you have at least 2 drives marked as SSD and the rest as archive? And if you open *gear icon* -> *Troubleshooting* -> *Service log* after you have done a "re-balance" attempt, is there anything with error or warning or critical level?