Jump to content
  • 0

Feature request: Native compression


RottenBit

Question

Since CloudDrive uses its own filesystem and has full control over it, has it been considered to implement native compression? And by compression, I don't mean NTFS file level compression, I mean optional native compression in CloudDrive's filesystem. This would reduce the amount to transfer data over the network at the expense of CPU usage (and in some cases RAM) during compression.

I guess we could have two types of implementations of the compression feature. A simple mode and one advanced mode.

The simple mode would just use one compression method that applies to whatever content is stored, for instance LZMA. It would be nice if the user could at least control the compression level (low / medium / high). And it would be useful if it was an adaptive compression technique where it is smart enough to not compress already compressed data like compressed pictures and movies.

The advanced mode would give the user choice to tinker more with the settings. The adaptive compression technique could be extended, for instance different compression methods for different types of data. For instance, a rule could be set up to use PPMd for data containing a lot of text (text files, word, databases etc) which is highly effective on this kind of data, and other compression methods for other kind of data.

Automatic compression that magically works behind the scenes in CloudDrive would be a very powerful feature. It would reduce the amount of data to transfer, and thus reduce the cost for both bandwidth and cloud storage since less data is stored.

It has a cost of course; depending on the compression method used, CloudDrive would require more CPU and sometimes more RAM usage during file transfer. The time to download / upload data would be reduced but the time to compress and decompress would be increased. Giving the user to choice to enable compression would be very nice, and giving power users to control even more settings would be the cherry on top.

So I'm curious - is this something you have considered? Is it something that users want? Needless to say I would like it a lot :) I believe compression, encryption and cloud storage go hand in hand.

 

 

Link to comment
Share on other sites

7 answers to this question

Recommended Posts

  • 0

Thanks for adding the request. I did search thoroughly on the forums on this topic and was actually very surprised that I didn't find any discussion about compression at all.

I then searched the web for alternative solutions that offered a virtual drive with native compression, but again I didn't find anything usable.

So I actually started working on my own filesystem with this in mind, having a virtual drive with built in compression. But I took to my senses and at least wanted to ask you guys to implement this before going down that dark road myself... :lol: But for you guys I believe applying real-time compression on the file streams would be very easy... unless of course there is something else complicating things.

Link to comment
Share on other sites

  • 0

See, that's part of the problem,  the way that the software is designed, encryption occurs in the driver. So as soon as data hits the drive, it's encrypted. 

So, even the cache data is encrypted.  So, .... do to this, the software would have to compress first, and then encrypt in the driver. And decrypt and decompress for reads.  That's a lot of work for a kernel driver... and could be a problem.  And Alex (just) said as much (see link about)

 

 

Link to comment
Share on other sites

  • 0

I didn't expect this would be easy, awesome features rarely are. AFAIK you could get an 'easy' start by using the native kernel compression methods in Windows like MSZIP, XPRESS, XPRESS_HUFF, and LZMS. LZMS is best (not shown on picture).image.thumb.png.6afe503af34c49d5581bbfa08cfff356.png

For instance I just compressed my entire C: drive now, 242 GB used compressed to 147 GB. 95 GB saved, that's 40% size reduction. Didn't take much longer than raw copy either, it sometimes is actually faster because you don't need to write as much data. And this was achieved with a one-size-fits all compression method.

I guess I'll be doing some more file system development myself.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...