Jump to content
Covecube Inc.
  • 0
PizzaGuy

Sharefile - 5000 item limit per directory

Question

Happy Thursday!  I suspect this is why other users are reporting corruption issues with their drives on other services as well.

I am using CloudDrive with Sharefile via their FTP.  It transferred about 400 GB no problem.  Shortly after hitting that I was getting the "Location is not available" (attached) error on a few folders that a few other users are reporting.  CloudDrive also started reporting many yellow and red errors indicating i/o errors (screenshot attached).  I enabled "Upload verification" and over a few days I noticed it wasn't making any progress on the "To upload:" number.  Checking the "Technical details..." screen indicated it was retrying the same chunk over and over for days.  

To find out what was happening, I connected to Sharefile FTP via SmartFTP and WinSCP and started copying 100 MB files named similarly to the CloudDrive chunks into that folder and they were disappearing nearly immediately.  Without upload verification, CloudDrive naturally wouldn't notice this.  Looking into the documentation of Sharefile they state folders "should not" exceed 5000 items.  This includes versions and recycle bin, so me hitting just over 4000 items after deletes and stuff makes sense that I would hit this limit.  Documentation here:

https://support.citrix.com/article/CTX227202

This effectively limits a CloudDrive on Sharefile to about 400GB with 100MB chunks or 40GB with 10 MB.

I suspect other services have similar limits and that's been causing grief to other users who hit it.  A possible solution is to set a limit for the number of files CloudDrive creates in a directory and spread the files across multiple directories and to always enable upload verification so one can catch it before it hits.

For now, I'm transferring the data that I can access back to local storage and then transfer the remaining chunks back to my local storage and I'm going to attempt to connect it via the Local Disk function just to see if any of it is accessible.

 

I hope this info helps someone and I'm happy to do any additional testing with Sharefile FTP to help improve CloudDrive.

Location is not available.jpg

Chunk rewriting loop.jpg

Share this post


Link to post
Share on other sites

2 answers to this question

Recommended Posts

  • 0

There is a new beta version that should take care of this by using hierarchical folders. 

Quote

* [Issue #28102] New cloud drives created using the FTP storage provider will now organize their chunks hierarchically.
    - Existing cloud drives can be upgraded to the hierarchical format by setting Ftp_UpgradeChunkOrganization to true 
      and rebooting (or detaching and reattaching the cloud drive). Upgrading existing drives is disabled by default.

http://dl.covecube.com/CloudDriveWindows/beta/download/?C=N;O=D

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...