Jump to content
Covecube Inc.
  • Announcements

    • Christopher (Drashna)

      Login issues   11/07/17

      If you have issues with logging in, make sure you use your display name and not the "username" or email.  Or head here for more info.   http://community.covecube.com/index.php?/topic/3252-login-issues/  
    • Christopher (Drashna)

      Getting Help   11/07/17

      If you're experiencing problems with the software, the best way to get ahold of us is to head to https://stablebit.com/Contact, especially if this is a licensing issue.    Issues submitted there are checked first, and handled more aggressively. So, especially if the problem is urgent, please head over there first. 
  • 0
thepregnantgod

Any update on REFS?

Question

26 answers to this question

Recommended Posts

  • 0

So update for all, it seems that the REFS driver uses a lot of memory when moving data onto a REFS drive.

 

I've tested it - and unless you're saying it's hardware specific (i.e. my specific ASUS motherboard or my mouse driver...), on a clean install of Win10 x64, with 96gb of DDR4 and a beta install of Drivepool.

 

When I move 3-4TB of data onto a REFS drive the mem usage climbs all the way up to 96gb.  It resets when I reboot.  And stays steady unless I start moving data to/from a REFS drive.

 

So those folks not seeing memleaks, I suspect it's because they're not moving terabytes of data back/forth from/to REFS drives.

Share this post


Link to post
Share on other sites
  • 0

See, that's just the thing, I've moved TB of data around and didn't see massive memory usage.   And by TBs, I mean when I reformatted my entire pool, one/two disks at a time, moving +100TB of data around. And my server stays up for weeks, and I add a LOT of data, especially in batches. 

 

So, this doesn't appear to affect every system, which is why I said it's "system specific".  There are probably a bunch of factors that contribute to the issue, which is probably why Microsoft hasn't issued a hotfix/patch to fix this bug yet. 

Share this post


Link to post
Share on other sites
  • 0

Exactly.  And it's this type of system specific issue that is hard to troubleshoot.

 

And being on both ends of this, it's absolutely frustrating.   While I know I haven't experienced it, I know that you and others have. It's definitely an issue.  And ... there isn't anything we really can do about it. 

Share this post


Link to post
Share on other sites
  • 0

From my works perspective (Windows 2016, Veeam, 90+TB, Moved to 64k ReFS format, Add 64GB Ram) MS has fixed the issue as long as you use 64k ReFS format, fit as much RAM as possible, and add a registry key or two.

 

https://support.microsoft.com/en-us/help/4016173/fix-heavy-memory-usage-in-refs-on-windows-server-2016-and-windows-10

 

We are using Option 1 registry key only.

Share this post


Link to post
Share on other sites
  • 0

For those with Mem Leak issues you may be interested in this Util I wrote - Info over here - https://yabb.jriver.com/interact/index.php/topic,112264.0.html

 

 

 

 

I've had issues on and off over the years with something eating memory till the "In Use" memory approaches Total Installed memory.... and I've noticed it starting again.  Tracking these down can be a PITA and while a reboot fixes it (temporarily) it can be inconvenient if you are doing something.  

I really should rebuild this PC and get off the Insider Previews but for now....Here is a very simple AutoHotKey script that I wrote awhile ago to test the memory when trying to track the leak.  One great side affect is that (for me) it actually also releases this lost memory.  All it does is increase a variables memory allocation and when the app closes the "In Use" memory is now greatly reduced.

Attached is the AHK Script and a compiled 64-Bit Version.

Share this post


Link to post
Share on other sites
  • 0

From my works perspective (Windows 2016, Veeam, 90+TB, Moved to 64k ReFS format, Add 64GB Ram) MS has fixed the issue as long as you use 64k ReFS format, fit as much RAM as possible, and add a registry key or two.

 

https://support.microsoft.com/en-us/help/4016173/fix-heavy-memory-usage-in-refs-on-windows-server-2016-and-windows-10

 

We are using Option 1 registry key only.

Thanks - I'll also give this a go.  I'm not running out of memory to the point that Windows halt just very heavy mem usage after a day

Share this post


Link to post
Share on other sites
  • 0

I am not 100% but I am pretty sure that ReFS metadata is the problem, this does not show up in task manager as used memory. Look at the animated pink metadata map half way down this page https://forums.veeam.com/veeam-backup-replication-f2/refs-4k-horror-story-t40629-375.html  The MS registry settings relate flushing this data. Use rammap to get an idea of what is actually using your memory.

Share this post


Link to post
Share on other sites
  • 0

I've got some big issues on my new Build with the MetaFile.  Even immediately after a cold boot I'm at 95% Memory Usage and the MetaFile has eaten all my memory and just grows!  I've tried some of the recommendations including:

- Disabling "Storage Services"

- Adding to the Registry : Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\RefsEnableLargeWorkingSetTrim = 1

 

My New Build is:

- Windows 10 Pro - 1709 (16299.19)

- i7-8700K

- ASUS ROG Maximus X Hero MB (latest BIOS 0505)
- Corsair H110i CPU Cooler
- G.Skill Trident Z 32GB (4x8GB) PC4-25600 (3200MHz) DDR4, 16-18-18-38-2N, 1.35v, Quad Channel Kit
- Samsung 512GB SSD, 960 PRO Series, m.2 (PCIE), Read up to 3500MB/s, Write up to 2100MB/s, Type 2280

- Drive Pool 2.2.0.871_x64_Beta pooling 7 x 8TB HDD Refs 3.2 64Bit Clusters (mix of Seagates and HDS)

 

Any suggestions?

Share this post


Link to post
Share on other sites
  • 0

So I managed to flush the MetaFile back to a few 100K using my mem util and rebooting (several times).  DP is now "checking" all the files and as it is doing that the MetaFile is growing steadily.  At the Checking 20% mark I've lost 6GB to the MetaFile so far.....and at:

- 25%, 8.5GB

- 50%, 24GB

- it then seemed to "stopped" growing when it hits 26GB!

Share this post


Link to post
Share on other sites
  • 0
  • 0

I believe every time an reFS system is booted the file system is checked with some scheduled tasks. Might need to disable them as some others have, but you have to think if MS set them up and enabled them they are important and should be run at some stage.

Share this post


Link to post
Share on other sites
  • 0

Can not install the KB - Get a "The update is not applicable to your computer".  I'm guessing this is because "Cumulative Update for Windows 10 Version 1607 for x64 based Systems (KB4013429)" is earlier than the Falls Creator Update version that I had.

Share this post


Link to post
Share on other sites
  • 0

No Difference with also adding registry setting #3.  As soon as DP access the pool, Metafile grows.  At present I have 20GB in the Metafile and 6GB in the Mapped File. 

 

Both of these figures however are included in the "Standby" Memory (not active memory) so (from what I understand) can be used as needed by other apps (eg I can flush it down using my mem test app).  If so - does it matter if the Metafile grows to consume all available RAM?  I've been having instability probs with my new build but it may be a co-incidence....

Share this post


Link to post
Share on other sites
  • 0

Okay, in that case, could you open a ticket at https://stablebit.com/Contact.

 

And do this: 

http://wiki.covecube.com/StableBit_DrivePool_System_Freeze

 

Make sure that the memory leak is apparent before doing this. 

This will BSOD the system, as that is how it gets the memory dump. 

Share this post


Link to post
Share on other sites
  • 0

I'm really hoping you find something as my write speed to the pool is appalling (ends up going to 1KB/Sec after a time unless I reboot).... read speeds are OK. 

Share this post


Link to post
Share on other sites
  • 0

If it's the ticket/dump that I think it is.... then unfortunately, not really.

 

At least, it's not our code.  It's specifically the "memory mapped files" that are eating up all of the memory.  So, this would definitely be ReFS' fault.  And not a lot we can do.

 

Try limiting balancing, as that could help. 

Otherwise, these disks were formatted on Windows 10, right?

If so, they're probably using the 4k cluster size, and that's contributing to the issue.  Reformatting with the 64k cluster size may help... but that would be tedious....

Share this post


Link to post
Share on other sites
  • 0

I've already migrated all the drives over to ReFS V3 / 64Bit / Check On.  I'll have to have a think on what to do and why my new build is different to the old one.

Share this post


Link to post
Share on other sites
  • 0

Too early to call it solved, but I seemed to have fixed the instability by changing my XMP RAM on the new build from G.Skill to Corsair.  So far DP seems to be working well (before it would not load the GUI or the GUI would freeze once the memory had all been allocated).  I also reinstalled DP (this took many attempts of installing / uninstalling various beta versions with windows complaining it could not find the correct MSI - but I got it sorted).  Currently running on 852 instead of 871.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×