Aspnetcore: Asp.net Core not Collecting Garbage

@rynowak

Although I might be overlooking something... To my understanding, the memory usage should not be increasing with every call to this method. After the object is created and sent to the UI, the memory should have been freed.

The large object heap is likely biting you here. If you allocate objects > 85KB in size then it'll be put in the LOH and get compacted very rarely. See http://stackoverflow.com/questions/8951836/why-large-object-heap-and-why-do-we-care for more details (or https://github.com/dotnet/coreclr/blob/master/Documentation/botr/garbage-collection.md#design-of-allocator if you want to go deeper).

I can't understand why Asp.net core doesn't seem to be collecting garbage. Last week I let a web service run for a few of days, and my memory usage reached 20GB

What is your service doing that it's creating such large objects? You should try taking a memory dump before it gets too big, that will clearly show you what objects are sticking around and why it's being held onto (you can use visual studio to look at dumps or a more advanced tool like windbg or perfview).

Try to allocate the array from the beginning rather than calling Enumerable.Repeat
or compact the memory using GCSettings.LargeObjectHeapCompactionMode (supported in .Net Standard)

Thanks @davidfowl and @MhAllan for the replies. But this example was a contrived one. I just wanted something that would use a noticeable amount of memory so that I could take a screenshot. The truth is that this is happening with any application regardless of the size of the object in question. And to answer your question @davidfowl, my service was just pulling some data from the database with dapper, and returning the resulting object. It was one row of data for each call. So it took a few days for the memory to grow to that amount. I was actually trying to test the DB when I stumbled onto this peculiarity. I had written a little console app at kept calling the method over and over again.

@zorthgo sure it's not Dapper? If you create your scripts by injecting parameters directly to your SQL scripts like in PHP you'll end up with lot of cached SQL scripts. Here is how Dapper does it
https://github.com/StackExchange/Dapper/blob/fe5c270aceab362c936456087a830e6fe1603cac/Dapper/SqlMapper.cs
You should use a memory profiler to tell what is keeping references to allocated memory. Visual Studio 2017 should be able to help you just take some snapshots from memory before and after multiple calls to your app and compare those.

@zorthgo Yes I have also seen this. Was using servicestack in my .net core console app and every time i make a call to the api the memory usage increased by massive amounts 50mb. I assumed it was a VS2017 bug but then confirmed the high usage in the task manager. As zorthgo stated by just making simple calls to api the memory usage increase significantly and does not seem to release memory.

Does this only happen on ASP.NET Core on .NET Core or is it also A problem with ASP.NET Core on .NET Framework?

Can you write a similar application using MVC 5 (on System.Web) and verify that you don't see the same behavior?

I can't make any progress from this issue in its current state.

In my instance I i was using only target framework .NetCoreApp 1.1 and my console app was referencing an object model in a shared project.

Not sure if this will help anyone. a sample app that calls hellorequest. In this app the startup memory is 85mb , then by repetitive request i managed to push up the memory use to about 145mb. It falls back sometimes to 125mb but then stays there. Not sure if this is normal behavior as I'm not used to .Net Core console apps. i always assumed I was doing something wrong or not instantiating correctly.

https://drive.google.com/open?id=0B0Gm-m-z_U84TU5keWFTMzc1ZWc

Facing same issue here on an Asp.Net Core application deployed to production with 3000-5000 active users..the memory on the server increased to 15GB yesterday...i have had to configure IIS to recylce the AppPool every 3 hours while i still try to figure out what the issue is.

Did anybody take a memory dump and look at what is taking up all of the memory in your particular applications?

@davidfowl @Pinox
I'm working on a big company, and we want to start a new project with ASP.NET Core, But when I saw this issue, I was afraid :worried: . this is a critical issue and can block lifecycle of our project.

So please, is it related to ASP.NET Core or .NET Core (CoreCLR) ? we will target Full .NET (4.6) that's why I'm asking.

@ikourfaln in my case I was using .net core console app , servicestack (.net core version) and kestrel. The weird thing is the memory use goes up to a level , then it stops suddenly and it does not go up again. I guess best is to test it on your side with small sample and check behaviour.

Perhaps @zorthgo can check on his side if he sees similiar behaviour in that memory is used to a certain level and then stops increasing as that is the behaviour that I'm seeing. I have updated my sample app to include @zorthgo example and I dont see the memory running away. It goes up but eventually stops.

I did change the source slightly:

public object Any(TestGC request)
{
const string message = "TEST";
return Enumerable.Repeat(message, 100000);
}

@Pinox
Thank you, I will check behavior on my side.

How about this bug in 2.0?

Any news to this? I'm using Core 2 over Net Framework and this is still happening. Each call to a _Controller_ incresed the used memory, but it never goes down. (_I used the WebApi Template_)

Hi,

I have the same issue with ASP.NET Core 2. I've took a memory dump and tried analyzing. From what I see the problem is exactly as the OP said. My application starts with allocation about 75 MB, and very quickly it goes all the way to ~750MB, out of it 608MB is "Unused memory allocated to .NET".

First snapshot at app start:

Second snapshot after 3 minutes and 100 requests:

we are also facing same issue, our controller is dealing with large amount of data, ( which is bad design and will be replaced soon ), each call to this controller is causing memory to grow. The memory reduces but only 40-50%( gain 50 Mb, reduces 30-35 Mb ), each call increases memory in range of 10-15 Mb each time. Service is hosted inside service fabric.

It looks like I have a similar issue in our production service (20-100 req/s) using a combination of:

• ASP.NET Core 2.0.4
• ServiceStack.Core 1.0.44
• SkiaSharp 1.59.2
• Docker v17.09.0-ce, build afdb6d4 on Ubuntu 16.04 x64
• x4 servers @ 40 CPUs, 128 GB memory
• Server GC is true
• each Docker container is set to 12k (mhz) cpu shares, 8GB ram

The application has a front-end web server and worker (shown respectively in the graphs below).

web server (last 6h)

worker (last 6h)

They both make use of large byte arrays because the service is acting as an object storage proxy, and consequently puts objects in the LOH. My question is, is this a known limitation of .NET Core at the moment? It seems as though the LOH in never fully cleaned up or fragmented.

Having said that, SOH seems to be working fine, as typical web api objects are cleaned up. Any suggestions? Is there a problem with my setup? I've analyzed the code and can't find any glaring memory leaks, and I'm not using anything special outside of ServiceStack library.

@sebastienros - any thoughts on this? Have we observed any similar behavior in our systems?

• We only measure this on 2.1, I will look into adding the same thing for 2.0
• All comments seem to be related to to the LOH issue that was mentioned, which should be taken into account in the application and pool large arrays as much as possible

@sebastienros, several questions:

1. I used Ants profiler to measure memory usage, according to it, no LOH fragmentation was detected. Can you advise how can I verify if my application suffers from LOH fragmentation issues?
2. What are the results on .net core 2.1? Is the issue resolved because Kestrel is using Span?
3. What if we can't pool arrays - can you provide a workaround? Should we use GCSettings.LargeObjectHeapCompactionMode.CompactOnce?

What are the results on .net core 2.1? Is the issue resolved because Kestrel is using Span?

We personally haven't seen any evidence that the issue is in Kestrel. It still looks like an application problem.

What if we can't pool arrays - can you provide a workaround? Should we use GCSettings.LargeObjectHeapCompactionMode.CompactOnce?

@jkotas @Maoni0 Any advice here?

how can I investigate if I suffer the same issue? the LOH according redgate memory profiler is almost emty as @sinapis describes but still using easy more then 1gb for just one user

Collect trace and analyze it using perfview. There are number of tutorials by Vance and others on how to trace down .NET memory leaks: https://www.bing.com/search?q=.NET%20memory%20leak%20perfview .

https://github.com/dotnet/coreclr/blob/master/Documentation/project-docs/linux-performance-tracing.md has Linux specific instructions for collecting traces.

If you believe that there is no memory leak and GC is just keeping more memory around that you would like, you can try:

• Using Windows job object or docker container memory limits to limit how much memory the process can use. The GC takes these limits into account and runs more aggressively when it is close to them.
• Or, switching from Server GC to Workstation GC. It is not unusual for server GC to have much higher peak working set. Workstation has lower higher peak working set, but also lower throughput.

Hi,

I think i am having the same kind of problem with a .Net Core Web API in production.

The application is running on Windows Server 2016 with .Net Core 2.0.3. The machine is a Hyper-V VM with 28 CPU cores and 24GB of RAM. If we don't recycle the IIS application pool often enough, we will eventually use all the available memory. When the application starts using a lot of memory (>=95% of total system memory) the CPU usage also increase strongly (from 2% to 70% sometimes). I am not sure sure if an OOM exception is triggered or not, we always recycle the pool before it happens (the maximum memory usage I have seen was 98% of memory used by dotnet.exe).

Analyzing a production memory dump with ".Net Memory Porfiler" (SciTech Software) here is what i found:

If this analyze is correct, about 95% of the memory is in "overhead > unused". Here is how this memory profiler editor is describing this category (on their forum):
_"Overhead->Unused" is memory committed by the .NET runtime for the managed heap. It is currently unused, but it is available for future instance allocations. There are lot of rules the runtime uses to decide whether to keep the committed memory or to release it to the OS. It depends on factors like the available memory, allocation patterns, the number of processors, whether the server GC is used, etc._

@jkotas I will apply your recommendations (Windows job object, and switching to workstation GC) and I will let you know the result. Please let me know if i can extract any other useful information from the production memory dumps i have.

Thanks

@sinapis @ronald7
Would any of you be able to share an app that shows the issue? If I could repro it we would be able to find the reason, or at least remove some code piece by piece and isolate a minimal repro.

@sebastienros I can't share the app, but I can share the session from PerfView session + memory dump.
Some description: I have a ASP.NET Core 2 Web API, I've created a load test of 200 users all sending the same request over 10 seconds. Overall 775 requests were processed.

This app jumped to almost 1 GB memory usage in task manager and stayed like that. Looking at the dump I can count about 18 MB:

So the questions is where did almost 1 GB go?

@sinapis Thanks

The behavior you are describing is not unexpected, the GC will allocated some memory as necessary on the peak load, and just release it over time. It's the GC Server mode, and usually wait for idle periods to release it and not affect your app perf. The amount of memory it will reserve depends of the total memory available on the system.

We would definitely see an issue if it kept increasing. I assume that if you don't send anymore requests and let your app run you will see the memory usage going down.

Could you run the same thing until it consumes most of your system memory? Or at least long enough with the same load that it will show it growing continuously? I will still get a look at your current dumps.

Also can you take dumps during and at the end of the jobs, so we can see the detals.

Hi @sebastienros

Unfortunately, I cannot share the app nor the memory dumps, but I will create a dummy application (with the same architecture and dependencies), run it on the same machine, if I can reproduce this behavior i will share this one with you. Please let me know if there any useful information I could extract for you from the memory dumps.

I have updated the GC mode from _server_ to _workstation_ on one production server, I will let you know in a few hours from now if it changes anything on the memory usage.

I also performed another test: we are running our application behind a load balancer, on 4 virtual machines. After removing one of the machine from the load balancer pool, the memory used by dotnet.exe did not decrease and remained at the same level even after 30 minutes. (However, the application was still processing a few requests: one request sent by SCOM on a dummy endpoint every 30 seconds). No memory was released and returned to the system.

Thank you

@sinapis I looked at your ETW trace. it is puzzling to me - you survived very little in the last induced gen2 GC yet we still chose to keep that much memory committed. your application seems edge case (you mostly just did only background GCs due to LOH allocations) - I wonder if we have some accounting errors there (another possibility is errors in the numbers reported but if you already verified that you have that much committed that's a smaller possibility). if you could repro with something you can share with me, that'd be fantastic; otherwise if it's possible to run your app with some logging from GC (I can give you a commit that does that) that'd be helpful too.

@Maoni0 please share how should I enable GC logging
If there is some other data you would like me to provide in order to disprove accounting error please let me know what should I provide you with and how (maybe tell perfview to collect more data?)
I'll try creating a minimum repro, but not sure I'll succeed since I don't know where the problem is.

@sebastienros hopefully I will provide another dump with more memory consumption today

Hi @sebastienros @Maoni0 ,

I ran our application with workstation GC mode for 12 hours but same result. I also recompiled the application with .Net 2.1 Preview 2 on a single production node for 1 hour, i will let you know the result, but for now the process is already using 2GB+ of RAM.

I have PerfView running on this same machine and i am collecting GC dumps, is there an email address where i could send you the OneDrive link, unfortunately i cannot share it directly in this thread.

If it can help i can also collect more metrics or GC logs. Thank you

@ronald7 _redacted_ I can forward to @Maoni0

Hi @sebastienros @Maoni0 I just sent you an email with two PerfView gcdump and a VMMap file, I hope this can help. On my side I am still trying to reproduce this high memory usage behavior with a dummy application.

Thanks!

I am also experiencing the same problem. The garbage collection never happens! The screenshot shows memory usage after doing about 50 requests using a fairly simple dotnet core web api app.

I just upgraded an ASP.NET Core app running on Ubuntu 16.04 from 1.1 to 2.0 and ran into this problem. It's pretty severe, causing the kernel to kill the app frequently due to OOM errors, and I'm contemplating whether to downgrade back to 1.x. There are certain pages we can't load at all - even after a Kestrel restart, the app immediately exhausts available memory after a single request! I thought about upgrading the server, but based on the comments here about ASP.NET Core apps using all available memory, I'm not hopeful that will help. Our stack is basically ASP.NET MVC Core + EF Core...nothing too fancy. If I get some time I'll try to create a sample to reproduce the issue - I don't think it should be that hard, given the simplicity of our stack.

FWIW, the system that I upgraded also has a .NET Core console app, and that does not appear to have any memory problems after the 2.0 upgrade, so this definitely appears to be an ASP.NET Core-related issue.

Probably related: https://github.com/aspnet/KestrelHttpServer/issues/2214

@danports have you tried calling GC.Collect() see if the memory usage goes down dramatically? that would give us a clue where we should start. if GC.Collect() (or the GC.Collect/GC.WaitingForPendingFinalizers/GC.Collect sequent) is not able to make memory usage go down dramatically it means there's simply that much memory that needs to be live so GC cannot reclaim it.

@Maoni0 I haven't tried that yet. I don't think my issue is with GC, because I did see memory usage drop from time to time - it just seems like my .NET Core 2.0 apps consume roughly 2-3x the memory they did compared to when they were running on .NET Core 1.1. 😞

I downgraded back to .NET Core 1.1 for now and will revisit this later when I have more time, probably after .NET Core 2.1 is released. (I ran into a pile of issues with 2.0 and this was just one of them.)

GC.Collect() does not help. Tried a very simple ASP.NET Core 2.0 and 2.1 Web API that has one controller that returns a dictionary of 200k ints. The allocated memory keeps going up with each request, even though the app does not use any more memory.

@Serjster returning 200K integers (4B) would take 800KB. In this case you are hitting the issue that is explained in this comment: https://github.com/aspnet/Home/issues/1976#issuecomment-289336916

In this case you should use an array pool to reuse them across requests.

Also good to know is that if code is running in 64bit mode then arrays/list etc. that contains pointers are twice the size compared to 32bit. If I remember correctly full framework runs any cpu code 32bit in 64bit OS by default. So people migrating their code might accidentaly hit LOH issues.

I am working with @Serjster , and here is what I have found. If I create a vanilla web api project using asp.net core (I used 2.1 in my latest test), I notice that when I run the diagnostic tool (or even check the process working memory set in code), the number of bytes return keeps climbing as I hit an endpoint. For example, if I have a single web api endpoint returning a Dictionary with 20,000 items in it, the following happens:

1. First visit to the controller method puts the Process Memory at 83MB.
2. I wait a few seconds, and then second visit it moves to 86MB.
3. I wait a few seconds, and third visit moves to 90MB.
4. Again - 94MB.
5. I do this n number of times, and it finally reaches about 304MB. Once it does this it levels off.

If the return object is a different sized object, all the numbers above are just bigger/smaller (including the level off amount), but the growth pattern is the same (aka, it will grow and grow until it levels off after many requests).

If I add GC.Collect in the method call (so it occurs on every single request, the level of is much lower, but there is still a period of growth until it levels off.

The other interesting point of detail is the number of objects and the heap size when doing snapshots is largely unchanged with each visit. But the Process Memory graph keeps showing a higher and higher number (this is also true if you grab the process and pull off the working memory set value).

I am beginning to suspect that the graph is showing allocated memory (and this memory grows based on some asp.net core useage/demand forecasting logic), but this is not necessarily consumed/leaked memory. I don't know enough to confirm though, so wondering if someone more knowledgeable may be able to chime in.

EDIT - re @davidfowl comment: Regarding your comment about things getting collected rarely... this could make sense. But how long does it typically take? I can go 30+ seconds in between requests, and the GC never seems to bring that memory number in the diagnostic chart back down. I am sure I am ignorant on something here, but just curious.

EDIT 2 - Now that I have read the SO link that david posted above in more detail, I am starting to think this is definitely the issue we are seeing. If we are running in an environment with limited memory (which we are in our dev environment where we are seeing this because we are being cheap) we run into problems with this.

Edit 3 - One lingering question. Why is the process memory going up consistently, but the heap size not going up if this is a LOH issue? Actually, I may understand this now. The heap is the used memory. The processor allocated memory is the used memory plus the fragmented memory blocks that are unused.

@RemyArmstro can you change Dictionary<int, int> to SortedDictionary<int, int>? Dictionary is probably allocating continous memory, might even add some extra data to every entry. Way SortedDictionary is implemented it will make many small allocations instead of one big one.

Edit: If you serializing to string and not directly to response output Stream then that might also cause LOH allocations.

@wanton7 Your response is missing the point. Dictionary is just the tip of the iceburg. We can use lists, arrays, etc. etc. and they all do the same thing. However, as was pointed out if the LOH is causing this, as it sounds like it is, then this behavior is probably fine? Except this might have some concerning side effects, like what happens when you run out of memory? Does your app just crash?

@Serjster ok I thought you had just small cases where this is happening. To me it's very unusual to have big lists, arrays like this and sending this much data in one api call if it's not binary. Usually when you have some sort of web api and get some data from it, you use paging. You shouldn't be sending 10000 entries to client side.
But if you have lot of problems like this and there is no way to change how your api works, then I think you should create your own chunked List and Dictionary implementations. If you really use arrays this big then you can replace them with your chunked lists or try to pool them when application starts.

I do wish Microsoft would create chunked implementations that everyone could use in situations like this.

@wanton7 yet again you're missing the point. It doesn't matter the size of the list. Even a single item or a small list causes this problem to happen.

@Serjster maybe i'm just blind but I don't see any posts from you where you said sending single item or small list will cause this to happen. Did you delete it?

Or from @RemyArmstro He talks about different sized dictionaries. I checked corefx and Dictionary will allocate array or these

private struct Entry
{
  public int hashCode;    // Lower 31 bits of hash code, -1 if unused
  public int next;        // Index of next entry, -1 if last
  public TKey key;           // Key of entry
  public TValue value;         // Value of entry
}

85000 byte allocation will cause LOH allocation so Dictionary with capacity of 5313 entries of int key and int value will cause LOH allocation. Capacity is not same as number or entries, capacity seems to be expanded by primes, check private Dictionary's private Resize method. Each struct could have extra allocation plus memory padding so even lower entries could cause LOH allocation.

Dictionary implementation details Dictionary.cs

Edit: fixed url

@wanton7 Thank you. I think we realize what the issue is now. It is just a bummer there is no great + simple solution to this. It basically comes down to being more aware of it and adjusting how you write code. The downside is that this unmanaged memory starts to feel a bit more managed. :( In the end, we may only have a few areas that truly breach this allocation limit, but one of the areas is pretty core to our app, so we see it a lot currently. I think we just need to rethink that piece, monitor, and try to ferret out any other areas we notice this creep. Thanks again!

We actually have similar situation soon and we need to create chunked IList<T> implementation. I'm going to use some size for chunks that is bitshiftable so I can just use bitshift and mask for indexing.

I would like to know which one is more beneficial for GC, bigger chunk or smaller chunk? From sizes between 1KB and 64KB. Smaller chunks mean more references for GC but I would guess that bigger chunk might be worse for compacting and fragmentation.

your understanding is exactly correct - I would go with sizes not too large; probably try 4k/8k.

@Maoni0 thank you!

I chose 4KB, so that we don't get any nasty surprises if we ever run our code under Mono. Found out by reading http://www.mono-project.com/docs/advanced/garbage-collector/sgen/working-with-sgen/ that LOH threshold is only 8000 bytes under Mono.

Is there any progress for this issue?

We are observing the same issue where process memory continues to grow despite the heap size remaining the same (or decreasing).

@sebastienros can you take another look at this? Maybe we can provide some detailed instructions to help people investigate their scenarios further?

Here are some items to consider with our situation:

• We're only returning a Dictionary<int, int> object with 1000 integer values stored. Nothing more than that.
• We converted our project from .NET Core 2.0 --> 2.1
• We are seeing this issue in MS Visual Studio 2017

Code as follows:

    public async Task<IActionResult> SampleAction()
    {
        var list = new Dictionary<int, int>();
        for (int n = 0; n < 1000; n++) list.Add(n, n);
        return Ok(list);
    }

To reproduce you must simulate some form of moderate load. We just clicked rapidly using Postman and could observe this behavior. Once we stopped simulating load, we watched the heap size decrease yet the process memory remained the same (even when we forced GC).

I am seeing this as well in one of my projects, but I also can re-create it in a brand new .net core API targeting .Net Core 2.1 (SDK 2.1.302).

I attached the sample API project that I created using Visual Studio 15.8.0 Preview 4. To show the memory increasing I had a .net core console app hit the default values GET endpoint every half second to get the 2 strings. The process memory usage is slow but in a project that returns more data this can grow quickly.

WebApplication1.zip

I found this post on stack exchange:

https://stackoverflow.com/questions/48301031/why-doesnt-garbage-collector-in-net-core-2-0-free-all-memory

Has anyone profiled their applications in release mode to see if this behavior is present? I'll give this is a shot today and see if the issue persists.

Edit: I tried profiling in Release mode and the issue still persists. I even force a GC to see if that will have any effect.

@chrisaliotta thanks for linking to that post, I was unaware of that behavior. It would indeed be interesting to see if that explains what people are seeing.

@Eilon @chrisaliotta Thanks but this is not related to this discussion. .NET not releasing memory in debug mode is a well known behavior and this is why we only measure memory consumption (and potential leaks) in release mode. Also even on release mode you'll see the memory increase to an extent overtime because of the Server GC mode. So this example doesn't prove anything for two different reasons.

@sebastienros So is the behavior that @beef3333 and I are observing consistent with what is to be expected? Namely, where Private Bytes remains elevated despite heap size decreasing? If so, it seems strange to me that each incremental request would continue to cause the private bytes to grow despite there being free heap space.

In Debug mode yes. So please try to use Release mode, and run your stress for a long time. If the memory increases indefinitely then there is a memory leak. If the memory is recycled (even if it takes some significant amount of memory) then all is fine in your case.

I just tested it again using the same project I attached in Release mode and I am seeing the same exact behavior.

I will test your application then, thanks.

I ran the application provided by @beef3333 locally for 2 hours, with a rate of 5K RPS, and the memory is stable (variance by 1MB overall, at 400MB on a machine with 32GB). The GC is correctly called regularly. I also inspected multiple dumps overtime and the various instances get created and collected as expected. I am using 2.1.1.

@sebastienros Thank you for looking into this. I think the takeaway in all of this is that:

• Memory management will behave differently for a .NET Core web application versus your run-of-the-mill desktop application.
• Developers should focus on average memory consumption as a function requests per second (RPS) over a period of time.
• Growth in private bytes may not always be indicative of a memory leaks.

Correct me if I'm wrong, but it appears that .NET Core will grow allocated memory based on average requests in order to ensure the fastest response time? If true, would it be safe to assume that it is likely not to release this allocated memory until the Application Pool resets -- or will it release this allocated memory over time if RPS decreases?

Same issue.
I have an asp.net webapi backend service.
The stacks are asp.net mvc, autofac, automapper, castle.dynamicproxy, entity framework core.
All memory will be eaten, then service crash.

Version is 2.1.0

@atpyk Update to 2.1.1. If that doesn't help you should really profile what's keeping that memory. I've used https://www.jetbrains.com/dotmemory/ but there are probably other tools that can do this as well. It can show what's actually allocated in Large Object Heap (LOH).

Are you running in 32bit mode? Because Large Object Heap allocations (bigger than ~85000 bytes) can cause out of memory exceptions in 32bit mode due to fragmentation. You can get over this limit very easily using Dictionary. Check this comment https://github.com/aspnet/Home/issues/1976#issuecomment-393833505

If you are running your code in full .Net Framework default behavior is running any cpu code in 32bit mode. You need to uncheck Prefer 32bit from project settings or set some registry setting in your servers registry to default to 64bit.

@wanton7 Many thanks. I will try your solutions.

I updated to 2.1.2 and deploy on Microsoft Azure webapp with win-x64, but no effect. @wanton7

@atpyk please create a memory snapshot (dump_ and analyze it (Visual Studio, MemoScope) to see what object are taking all the memory, or just increasing in count. You can also take two and compare them overtime.

@sabastienros I do believe that enough people have raised a concern over this that you/MS should actually start analyzing this yourself. Perhaps this is working as you intended, but then the design is flawed.

Our apps are eventually running out of memory and crashing, and this all running in a production Azure environment. This is not acceptable.

@atpyk then it sounds like a memory leak. Profile your memory to see what is keeping that memory around like @sebastienros said.

One question, are you even using ASP.NET Core? I read you first comment again and you mention ASP.NET MVC. ASP.NET MVC and ASP.NET Core are two completely different products. These issues and this repo for ASP..NET Core.

Edit: Just from version numbers sounds like you are using ASP.NET Core, but wanted to make sure.

We use .net core MVC. @wanton7
I am doing analyze the memory. Maybe the Castle Dynamic Proxy lead to memory leak.

@Serjster are your programs running in 32bit .NET Core and crashing with out of memory exceptions? If your code does lot of LOH allocations then memory fragmentation is probably the reason.
If you are running in 32bit environment you have two choices, fix your code to avoid LOH or switch to 64bit environment.

I'm not very familiar with Azure but after little bit googling found this https://blogs.msdn.microsoft.com/webdev/2018/01/09/64-bit-asp-net-core-on-azure-app-service/

@Serjster I believe not all reports here are equal, and prefer to check the validity of each different case. Something like "my app has a memory leak" doesn't mean it's because of the framework, so I prefer to ensure each case it an actual one.

Taking your case for instance, "I believe" that I answered the reason why your memory is increasing. Have you been able to fix it after my explanation? Or it didn't solve the issue and in this case can you give an application I can run locally to reproduce the issue?

@sebastienros What we ended up finding was that memory allocation just kept increasing, even though memory consumption did not. I know ASP.NET core is doing some heuristics to tell it that it should grab more, but it seemed to be constantly allocating more and more on each request. It almost appears to greedy to me in this regard, but I could be wrong.

Either way, I think @Serjster point is that this thread keeps growing because there is clearly some confusion here. In the old ASP.NET land (before core 1 I believe), we did not need see this behavior (at least not at this magnitude. It may not really be a bug/issue, but it is definitely something causing a lot of people to pose the same question over and over again.

It would be nice if there was an official article that really address this thread from top to bottom, instead of it going in circles as it has been. Hope that helps clarify.

It would be nice if there was an official article that really address this thread from top to bottom, instead of it going in circles as it has been. Hope that helps clarify.

I second that. It would be good for us to know why .NET Core 2.1 is more "opportunistic" when it comes to memory management than prior versions.

@sebastienros can we please have a summary of the issues here? there are 81 comments - I'm under the impression they are not all about the issues (although I have not read them carefully at all so I might be mistaken). if so can we list all the distinct issues here and see if we have repros for each of them? there are enough folks who have mentioned the memory increase I think it justifies us to get repros and figure out whether these are general issues or not.

Sure. I am currently trying to identify all these threads and see what the status is for each of them. I will eventually close this issue and reopen more specific ones that focus on each report, because this single thread isn't sustainable anymore. I'll link them here for those on this thread that want to follow them.

In parallel I will work on a document that lists all the recommendations, the known memory issues (LOB, HttpClient, ...) and the ways to analyze and report these issues.

Just to assure you we do care about this issue and in order to detect memory leaks preemptively, for the past 6 months we have been running an ASP.NET applications continuously on Azure, on both Linux and Windows, for 24h and 7 days. These tests fetch the latest ASP.NET source on every iteration (daily or weekly). We measure RPS, latency, CPU and memory usage. The application uses EF Core, MVC and Razor, and is sustained to 50% CPU to simulate a significant load.

You can see the results publicly here on this site (browse to find the daily report): https://msit.powerbi.com/view?r=eyJrIjoiYTZjMTk3YjEtMzQ3Yi00NTI5LTg5ZDItNmUyMGRlOTkwMGRlIiwidCI6IjcyZjk4OGJmLTg2ZjEtNDFhZi05MWFiLTJkN2NkMDExZGI0NyIsImMiOjV9&pageName=ReportSectioneeff188c61c9c6e3a798

This has allowed us to fix some problems by the past, and to be confident that there is no fundamental leaks in the system right now. But it's nowhere close to using all the components that we ship, and there might be issues that we need to identify. Providing dumps and applications that reproduce the issues are the main ways you can help us.

@sebastienros Thank you for the updates. I know in our case the issue was more about a new "greedy memory allocation" concern, which we original mistook as a memory leak. I am not even sure there is an issue now, it could just be that the new optimization heuristics are a lot more aggressive. Not sure... but I think you are on the right track with really assessing this thread and coming up with some consolidated explanations/summaries on what people are seeing/misunderstanding. Good luck!

All individual reports have been isolated and will get individual follow-ups, and be closed if they are already solved. Feel free to subscribe to these to be kept in the loop.

In parallel I will work on a document that lists all the recommendations, the known memory issues (LOB, HttpClient, ...) and the ways to analyze and report these issues.

This is a huge +1 from me. I feel that one of the biggest issues here is how hard it 'feels' to gather information, to then try and help determine _what_ is the problem. Having some great docs that can allow us to follow some instructions to both (i) gather, (ii) attempt a self-diagnose and (iii) post our dumps/findings in a way that is efficient for the MS team can really help both sides of the fence.

If we (the developers) can be better-enabled to diagnose and/or provide information, then this is a win-win for all.

Thanks again for listening @sebastienros - very much appreciated, mate!

What do you think about situation in the picture below?

We run 4 WebApps inside the same Plan. Initial it was B1, scaled up to S2 and memory kept growing until I set to the hungry webapp csproj:

<ServerGarbageCollection>false</ServerGarbageCollection>

and also disable Application Insights

1. I believe that since memory could be kept under control with setting above, there is no memory lea. Correct?
2. Is the presented behavior normal?

Same situation here as @alexiordan, we had a .net core 2.1 console, that ran some IHostedServices hosted in Kube, FROM microsoft/dotnet:2.1-runtime AS base. We wanted to enable HealthChecks so we added asp.net with just the HealthChecks middleware and changed the base image to microsoft/dotnet:2.1-aspnetcore-runtime. The result was OOM. We have managed to stabilize the memory allocation by adding false in csproj.

Our analysis shown that in an asp.net app The GC collects less frequently, also the Finalizer Queue is traversed less frequently.

Also, if we forced the GC to collect and traverse the finalizer queue by adding the following in our pipeline,

System.GC.Collect();
System.GC.WaitForPendingFinalizers();
System.GC.Collect();

the memory allocation remained stable.

What do you think about situation in the picture below?

We run 4 WebApps inside the same Plan. Initial it was B1, scaled up to S2 and memory kept growing until I set to the hungry webapp csproj:

<ServerGarbageCollection>false</ServerGarbageCollection>

and also disable Application Insights

1. I believe that since memory could be kept under control with setting above, there is no memory lea. Correct?
2. Is the presented behavior normal?

Hey @alexiordan

We're seeing a very similar memory profile when using AI (web app on net core 2.1) as well, have you progressed any further on solving this? Obviously we want to keep AI in the apps.

It is weird that the usage grows on each request, but setting the above to false seems to stop it for me? weird because you'd think true would be the value that enables it, but it seem seems the other way around...

I forgot to mention that soon after announcing my intention to write an article about the issues described in this thread I actually did it. You can see it here: https://github.com/sebastienros/memoryleak

It comes with a small application that renders the patterns on a chart, live.

but setting the above to false seems to stop it for me? weird because you'd think true would be the value that enables it, but it seem seems the other way around...

Client garbage collection (optimised for memory sharing with many apps and keeping memory free) is more aggressive than Server garbage collection (optimised for throughput and concurrency).

Setting SGC to false, my asp.net core api fell from 150mb to 48mb, and didn't grow on each request after that. So in reality is this the best setting for production, for now?

@kgrosvenor actually, it depends. Quoting from the excellent @sebastienros article:

On a typical web server environment the CPU resource is more critical than memory, hence using the Server GC is better suited. However, some server scenarios might be more adapted for a Workstation GC, for instance on a high density hosting several web application where memory becomes a scarce resource.

Thank you that's very handy, I will bear in mind - will follow this thread for any more updates, that being said I'm absolutely loving asp.net core :)

Also suffer from this with a .net core 2.1 console app. constant memory growth. have set docker containers at a low maximum so it hits it and restarts, which is working okay, but it's ugly.

Any news on this side? We also have the same behavior in ASP.Net Core v2.1. From what I can see in the commit https://github.com/aspnet/AspNetCore/commit/659fa967a1900653f7a82f02624c7c7995a3b786 it seems there was a memory management problem that is going to be fixed in v3.0?

@flo8 have you tried upgrading to 2.2 and checked the behaviour? https://dotnet.microsoft.com/download

Running the latest 2.2 and also have this issue - merely creating a list of 1 million ints and returning an emptyResult() will increase my heap a few hundred MB every request until I run out of memory. Setting ServerGarbageCollection to false cured it, although it doesn't seem like the correct fix...

@dre99gsx as it seems you got an easy repro, could you share a project and steps so I can do the same thing locally?

In this case it should fill the LOB but they should be collected on gen2. Also please share what environment I can repro this on, OS, memory, load.

Sorry for the cryptic response. Quite easy:
(Windows 7 64bit, 16GB ram, Google chrome for http requests, VS2017 community) - I'm still getting used to adding code to these threads, pardon the appearance)

• start a new .NET Core 2.2 Web App
• Inject a service class, scoped (nothing special..) into the controller constructor
• Have the Index() action of the controller call a method of this service class
• Create a model class (DumbClass) with one property: public int ID {get; set;}
• In the service class method, instantiate a list and populate it:
  var lst = new List();
  for (i=0; i<10000000; i++) <--- note: 10 million itterations
  {
  lst.add(new DumbClass(){ID=i});
  }
• return from the method, don't need to pass anything back, but you can pass that list back as well...
• index() return new EmptyResult();

Now just call the action every 8 seconds, and watch the profiler memory go up. On my system, this is what I see on Private Bytes:

Start application: 155MB
1st http get request: 809MB
2nd: 1.2GB
3rd: 1.4GB
4th: 1.8GB
5th: 2.1GB ... 2.3GB.. 2.6GB...

Now, at some point, GC seems to kick in, but it never drops below 3GB in this example. As mentioned, if I turn the ServerGC to false, never rises above 1GB, although it still climbs up there and hovers at 1GB.

Let me know if that helps, and if you can reproduce it. Oh, I read through your github post: "Memory Management and Patterns in ASP.NET Core", fantastic write up ,thanks for contributing!

@dre99gsx 👋

I'm still getting used to adding code to these threads, pardon the appearance)

No probs at all :) Question: could you actually put the entire sample application up to GitHub (free repo's) or somewhere similar? That's the simplest way for anyone else to quickly clone/download _the entire_ exact sample app/repo, which you've been playing with.

Also, screen shots of memory usage using TaskManager would help (if on windows - or the equiv on *nix .. which is top command ??)

Great effort so far!

/me goes back to silently watching this thread with earnest.

A reminder you can see some demos and explanation for all the symptoms that are described in the thread here: https://github.com/sebastienros/memoryleak

Also each of these issues have been treated individually and none have proven to be bugs in dotnet core __so far__, but expected behavior.

@dre99gsx Now going back to the recent comment, I would urge you to file a separate issue from this thread. Being on my phone I didn't realize it was "this one" ;). From your first comment you state

until I run out of memory

So I would expect an OutOfMemory exception, which is why I asked for a repro. But in your next comment you state:

Now, at some point, GC seems to kick in, but it never drops below 3GB in this example

So there is no memory issue. This is typical from a machine with lots of cores and lots of available memory. The GC will free up the managed heaps, but the memory will still be committed as there is no reason to un-commit it (lots of available memory). This is a standard behavior in .NET, and I demo it in the article I pointed to. You can also read this: https://blogs.msdn.microsoft.com/maoni/2018/11/16/running-with-server-gc-in-a-small-container-scenario-part-0/

I know that the runtime team is currently working on ways to constrain .NET applications running in containers so it doesn't use that much memory, targetting 3.0, to solve some microservices scenarios.

And as you found out by yourself, if your application can't use all the available memory on the server, you should use the workstation GC mode.

Correct, when I stated "run out of memory", I'm basing that on visually seeing no memory available for any other application via Windows Private Working Set (Task Manager); not a "Out of Memory Exception".

You know what, you're right. This is looking more and more like expected behavior, and if there is so much memory available, why spend resources freeing it up if no one else needs it! I get it. As long as GC is smart enough to free up memory so that other applications aren't limited, I'll just keep it as is and let it do its' thing. Thanks again!

I have developed my application in Asp.net core 2.2 also facing same issue related to memory release.
each call increases memory in range of 40-50 Mb each time never get release.

I have also added mentioned tag ServerGarbageCollection>false
still facing same issue for 50 user it is utilizing approx 2GB RAM in In-Process mode (w3wp iis worker process)

Please help.

Please help !! the same issue as ankitmori14

@ankitmori14 , @ikourfaln - if you've read @sebastienros 's comment at https://github.com/aspnet/AspNetCore/issues/1976#issuecomment-449675298 and still believe there is a memory issue, please file a new issue with detailed steps to reproduce the issue, plus any other information and traces you have about the behavior. Please remember that unless the app/process is actually having errors, it is unlikely (but not impossible) that there is a bug. The default 'Server' Garbage Collector doesn't try to use the least amount of memory possible; it rather kicks in when it needs to, such as when memory is actually running out. So even a small app might use 2GB memory and that's not a problem because there is still 4GB free.

Hi,

We are experiencing this problem : https://stackoverflow.com/questions/53868561/dotnet-core-2-1-hoarding-memory-in-linux

Basically the memory grows over the days in the process until Kubernetes kills it because reaches the configured limit of 512Mb. Funny enough, the memory went down drastically while taking a memory dump, without the process restarting or anything. Examining the memory dump, we saw lot of objects with no root.

We disabled also the concurrent GC (ie: background GC) yesterday and it seems to be better now, but we will have to wait at least a week to confirm.

<PropertyGroup>
  <ServerGarbageCollection>false</ServerGarbageCollection>
  <ConcurrentGarbageCollection>false</ConcurrentGarbageCollection>
</PropertyGroup>

@vtortola question, when you configured the 512Mb limit for your application, did you have an idea of the number of concurrent request you wanted to handle or tested how many concurrent request your app could handle before falling over??

We did some preliminary and rough tests and checked we could handle 500 concurrent websockets per pod using 512Mb. We ran for hours with 2 pods and 1000 concurrent connections with memory being less than 150Mb . The deployed application, with 2 pods, has between 150 and 300 concurrent connections at any moment, and the memory varies from less than 100Mb on the first few days, till reaching the 512Mb in around 2 weeks. There no seems to be correlation between the number of connections and the memory used. More than 70% of the connections last 10 minutes.

Could you share a memory dump when it's at 100MB and 512MB to see what instances are still alive?

I am afraid I cannot share a dump, since it contains cookies, tokens, and a lot of private data.

Can you compare then locally then? In terms of what object take the most memory, and if their number doesn't correlate to your load. Like if you have 300 connections there shouldn't be 3K connection objects.

Unfortunately the test of setting <ConcurrentGarbageCollection>false</ConcurrentGarbageCollection> did not help and the process is still hoarding memory.

We have a linux dump of the process, the only tool we have to analyze it is lldb and we are very noob in this matter.

Here some data in case it rings a bell:

(lldb) eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00007F8481C8D0B0
generation 1 starts at 0x00007F8481C7E820
generation 2 starts at 0x00007F852A1D7000
ephemeral segment allocation context: none
         segment             begin         allocated              size
00007F852A1D6000  00007F852A1D7000  00007F853A1D5E90  0xfffee90(268430992)
00007F84807D0000  00007F84807D1000  00007F8482278000  0x1aa7000(27947008)
Large object heap starts at 0x00007F853A1D7000
         segment             begin         allocated              size
00007F853A1D6000  00007F853A1D7000  00007F853A7C60F8  0x5ef0f8(6222072)
Total Size:              Size: 0x12094f88 (302600072) bytes.
------------------------------
GC Heap Size:            Size: 0x12094f88 (302600072) bytes.

And the result of dumpheap -stat : https://pastebin.com/ERN7LZ0n

You can find how Grafana was reporting the state of memory in https://stackoverflow.com/questions/53868561/dotnet-core-2-1-hoarding-memory-in-linux

We have another bunch of services in dotnet core that are working flawless, although they do not use websockets.

@vtortola would you mind creating a new issue so we don't fork this one. I think the fact it's using WebSockets makes it unique and we need to create at least a sample app that would behave like yours and stress it over a long period of time. Would you be able to describe what your clients do with the websocket, how they connect to it, and for how long? I could setup an app like this and run it over days, and take memory dumps if I see the memory growing.

It might also be some conflict between the GC and Kubernetes, in a way that the GC doesn't get warned that it's close to the pod's limit.

@richlander is working on a similar issue. Rich does this case (read the last 6 comments only) could be related to what you are working on?

if it makes any difference, I have the same (or very similar) issue. I have a dozen+ .net core 2.1 and 2.2 console apps running on docker - each container with a 512mb memory limit, also using websockets (client), and the apps hit the container memory limit every ~3 days. The container then shuts down and restarts.

Everything is using authenticated services, so I can't provide what I have, but I might be able to put something together that uses a test site to reproduce the issue.

What parameters do you use to set the memory limites? Just --memory or also --memory-swap?

When I test it, I can see it's taken into account very well, never going over the limit I set, even with a super low limit like 16MiB, however it swaps on disk like crazy. And I am testing with an app that does db queries (EF) and Razor views rendering.

@sebastienros absolutely, I created https://github.com/aspnet/AspNetCore/issues/6803

Let me know if you need more information.

I'm currently experiencing memory issues.

After an hour in production all memory is used by the w3wp.exe process (running .NET Core InProcess). Changing the GC to Workstation, did not do the trick for me.

After analyzing a memory dump, I found this similar issue, https://github.com/aspnet/AspNetCore/issues/6102.

I'm hoping to test it out in production later today, after upgrading to the latest .NET Core run time, currently 2.2.3. I'll let you know how it goes.

I am having similar problems with memory use. If I set limits on my Linux containers, it just makes it OOM faster. It even happens using the basic templates in Visual Studio. One thing I've found - Core 2.1 and 2.2 are affected. Core 2.0 is not - but it's EOL :(

See above

I know that the runtime team is currently working on ways to constrain .NET applications running in containers so it doesn't use that much memory, targetting 3.0, to solve some microservices scenarios.

And as you found out by yourself, if your application can't use all the available memory on the server, you should use the workstation GC mode.

Sadly workstation mode doesn't help at all - sadly I'm not really in a position to try one of the 3.0 previews or similiar.

@PrefabPanda make sure you are actually running in workstation GC mode by checking that System.Runtime.GCSettings.IsServerGC is false.

Then this could be genuine memory leak. Open a separate issue maybe the best step.

@wanton7 - thanks, I've double-checked and that is definitely set.

@DAllanCarr - will do

I am almost certain it's not a memory leak. More like Docker settings might not be enforced correctly and the GC doesn't kick in before the limit is reached. I know that 3.0 introduces fixes in this sense.

@richlander does this issue look like something that is known not to work in 2.2?

@PrefabPanda would you mind sharing the exact docker version you are using and the docker compose file? I am trying to repro but between docker and docker-compose versions I have a hard time replicating this issue locally.

@sebastienros , @richlander - Thanks for getting back to me. I really appreciate it.

My Docker versions:

Docker Desktop Community
Version 2.0.0.2 (30215)

Engine: 18.09.1
Compose: 1.23.2

See attached the entire test project:
WebApplication1.zip

test curl requests.zip

Just in case, my Dockerfile:

FROM mcr.microsoft.com/dotnet/core/aspnet:2.2 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build
WORKDIR /src
COPY ["WebApplication1/WebApplication1.csproj", "WebApplication1/"]
RUN dotnet restore "WebApplication1/WebApplication1.csproj"
COPY . .
WORKDIR "/src/WebApplication1"
RUN dotnet build "WebApplication1.csproj" -c Release -o /app

FROM build AS publish
RUN dotnet publish "WebApplication1.csproj" -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "WebApplication1.dll"]

docker-compose.yml:

version: '2.4'

services:
webapplication1:
image: ${DOCKER_REGISTRY-}webapplication1
mem_reservation: 128m
mem_limit: 256m
memswap_limit: 256m
cpus: 1
build:
context: .
dockerfile: WebApplication1/Dockerfile

docker-compose-override.yml:

version: '2.4'

services:
webapplication1:
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=https://+:443;http://+:80
- ASPNETCORE_HTTPS_PORT=44329
- DOTNET_RUNNING_IN_CONTAINER=true
- DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true
- ASPNETCORE_preventHostingStartup=true
ports:
- "50996:80"
- "44329:443"
volumes:
- ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro
- ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro

@sebastienros - even if there's a way I can put a separate environment variable into the container for the GC to look at, that would work just fine for me.

I have tried calling it explicitly, but it doesn't help - I assume even when calling it in code it still sees the wrong memory size of the container/machine.

@PrefabPanda when you said "I have tried calling it explicitly", could you please elaborate what means exactly? GC sees the correct container memory limit if you have one specified, whether it's triggered naturally or induced. if you called GC.Collect(), it will do a full blocking collection; and if you do GC.Collect(2, GCCollectionMode.Default, true, true) it will do a full compacting GC, when this returns the heap is at the smallest size possible, regardless of the container limit or anything else.

Hi @Maoni0 - I've tried GC.Collect(2, GCCollectionMode.Default, true, true)

I've just seen another comment saying 256MB is too small for 2.2, but might be 'fixed' in 3.0. Looks like I'll need to experiment some more...

if you've tried GC.Collect(2, GCCollectionMode.Default, true, true) and the memory did not go down as you expect, it means you have a true memory leak. not sure how much tooling is available in your environment. can you run sos commands at all? if so you could always look to see what remains on the heap after your induced GC

Thanks @Maoni0 - bear in mind I'm using a template in Visual studio - there is literally no code of my own in this test project I attached. I've got some other feedback to work through, I'll be in touch. Many thanks.

@Maoni0 - I tried setting the higher limit, no difference. Looks like I'll have to try out the 3.0 preview

I'd like to lock the comments on this issue as it's a very old one that got all its cases treated in separate issues. I commented on this one by mistake thinking I was commenting on https://github.com/aspnet/AspNetCore/issues/10200

Thanks