Pomelo.entityframeworkcore.mysql: Database CPU spikes and cripples database after upgrade from .NET Core 2.2 to .NET Core 3.0 with Pomelo 3.0.0-rc1.final

Steps to reproduce

I have recently converted my website that uses Identity Framework from .NET Framework MVC to .NET Core 2.2 MVC. I went live with it last week using Pomelo 2.2.0 and it's worked a treat. No problems at all with regards to Entity Framework or my MySQL database.

Yesterday I saw on this thread that 3.0.0-rc1.final for Pomelo was released so I thought it would be wise as I had just upgraded to .NET Core to at least be on the latest version.

I followed this migration guide and I converted my site to .NET Core 3, using the 3.0.0-rc1.final version of Pomelo. I tested the site locally and in my test environment and all looked good.

Unfortunately when it came to going live in my production environment, where there are around 100-150 people on the site at any one time my database spiked to 80-100% CPU as soon as a single container span up. Also the log was outputting database timeout errors immediately.

I tried it on two occasions to bring a container in using the new version. You can see these two occasions clearly on the metrics for my AWS RDS MySQL database below:

The issue

The log is being flooded with command timeout expired errors at the same time as the spike and it only takes one container to come into the load balancer to bring my database to it's knees. Sadly at this point I'm not sure what to do to troubleshoot it. It looks like all of the exceptions stem from the Authorization/Session middleware:

at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)

(snippet from full stack trace below).

The migration guide says that the Authorization middleware is required now, so I don't even think that I can remove the new calls that I added to the startup.cs.

Here's my startup class:

public class Startup 
    { 
        private IConfiguration Configuration { get; } 

        private static readonly ILoggerFactory EfLoggerFactory = LoggerFactory.Create(builder => 
        { 
            builder 
                .AddFilter((category, level) => 
                    category == DbLoggerCategory.Database.Command.Name 
                    && level == LogLevel.Information) 
                .AddConsole(); 
        }); 

        private readonly IWebHostEnvironment environment; 

        public Startup(IConfiguration configuration, IWebHostEnvironment environment) 
        { 
            this.environment = environment; 
            Configuration = configuration; 

            var builder = new ConfigurationBuilder() 
                .SetBasePath(environment.ContentRootPath) 
                .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true) 
                .AddJsonFile($"appsettings.{environment.EnvironmentName}.json", optional: true) 
                .AddEnvironmentVariables(); 

            Configuration = builder.Build(); 
        } 

        // This method gets called by the runtime. Use this method to add services to the container. 
        public void ConfigureServices(IServiceCollection services) 
        { 
            var redis = ConnectionMultiplexer.Connect(Configuration.GetConnectionString("RedisConnection")); 
            services.AddDataProtection() 
                .PersistKeysToStackExchangeRedis(redis, "DataProtectionKeys"); 

            services.Configure<CookiePolicyOptions>(options => 
            { 
                // This lambda determines whether user consent for non-essential cookies is needed for a given request. 
                options.CheckConsentNeeded = context => false; 
                options.MinimumSameSitePolicy = SameSiteMode.None; 
            });

            services.AddDbContextPool<MyDbContext>( 
                options => options.UseLazyLoadingProxies() 
                    .UseLoggerFactory(EfLoggerFactory) 
                    .UseMySql(Configuration.GetConnectionString(ConnectionStringConstants.DefaultConnectionName), 
                        mySqlOptions => { mySqlOptions.ServerVersion(new Version(5, 7, 24), ServerType.MySql); } 
                    )); 

            services.AddIdentity<User, Role>(options => 
                { 
                    // TODO: Probably should improve the password complexity - requires changes to the registration logic to handle errors when creating user  
                    options.Password.RequireDigit = false; 
                    options.Password.RequiredLength = 6; 
                    options.Password.RequireNonAlphanumeric = false; 
                    options.Password.RequireUppercase = false; 
                    options.Password.RequireLowercase = false; 
                }) 
                .AddEntityFrameworkStores<MyDbContext>(); 

            services.AddAuthentication() 
                .AddFacebook(o => 
                { 
                    o.AppId = Configuration.GetValue<string>("Facebook:FacebookAppId"); 
                    o.AppSecret = Configuration.GetValue<string>("Facebook:FacebookAppSecret"); 
                }); 

            services.AddAuthorization(options => 
            { 
                options.FallbackPolicy = new AuthorizationPolicyBuilder() 
                    .RequireAuthenticatedUser() 
                    .Build(); 
            }); 

            services.ConfigureApplicationCookie(options => 
            { 
                options.LoginPath = "/Welcome"; 
                options.AccessDeniedPath = "/Welcome"; 
            }); 

            services.AddStackExchangeRedisCache(options => 
            { 
                options.Configuration = Configuration.GetConnectionString("RedisConnection"); 
                options.InstanceName = Configuration.GetValue<string>("ApplicationRedisKey"); 
            }); 

            services.AddSession(); 
            services.AddResponseCaching(); 

            RegisterServices(services); 

            services.AddJsEngineSwitcher(options => options.DefaultEngineName = ChakraCoreJsEngine.EngineName) 
                .AddChakraCore(); 

            services.AddReact(); 

            services.AddMvc() 
            .AddJsonOptions(options => 
            { 
                options.JsonSerializerOptions.PropertyNamingPolicy = null; // Enables PascalCase JSON serialization (default for .NET core is camelCase) 
            }) 
            .SetCompatibilityVersion(CompatibilityVersion.Version_3_0); 

            services.AddSignalR().AddStackExchangeRedis(Configuration.GetConnectionString("RedisConnection"), options => 
            { 
                options.Configuration.ChannelPrefix = Configuration.GetValue<string>("SignalRRedisKey"); 
            }); 

            SetStaticConfigValues(); 

            var builder = services.AddControllersWithViews(); 

            if (environment.IsDevelopment()) 
            { 
                builder.AddRazorRuntimeCompilation(); 
            } 

            // AWS Config 
            services.AddDefaultAWSOptions(Configuration.GetAWSOptions()); 
            services.AddAWSService<IAmazonLambda>(); 
            services.AddAWSService<IAmazonS3>(); 
            services.AddAWSService<IAmazonSimpleEmailService>(); 
        } 

        public void ConfigureContainer(ContainerBuilder builder) 
        { 
            var listOfModules = new List<string> 
            { 
                "Blah.Bl.Core.Bootstrapper.BlBootstrapper, Blah.Bl.Core", 
                "Blah.Dal.Core.Bootstrapper.DalBootstrapper, Blah.Dal.Core", 
                "Blah.Domain.Core.Bootstrapper.DomainBootstrapper, Blah.Domain.Core", 
                "Blah.WebUI.Core.Bootstrapper.WebBootstrapper, Blah.WebUI.Core" 
            }; 

            var serviceBootstrappers = new List<Module>(); 

            foreach (var moduleToLoad in listOfModules) 
            { 
                var boostrapperType = Type.GetType(moduleToLoad); 

                var serviceBootstrapper = (Module)Activator.CreateInstance(boostrapperType); 
                serviceBootstrappers.Add(serviceBootstrapper); 
            } 

            serviceBootstrappers.ForEach(module => builder.RegisterModule(module)); 
        } 

        private void SetStaticConfigValues() 
        { 
            ImageHelper.ImageLocation = Configuration.GetValue<string>("ImageLocation"); 
            ImageHelper.ImageBucketName = Configuration.GetValue<string>("ImageBucketName"); 
        } 

        private void RegisterServices(IServiceCollection services) 
        { 
            services.AddOptions(); 

            services.AddScoped<IPasswordHasher<User>, BlowfishPasswordHasher>(); 
            services.AddTransient<IViewRenderService, ViewRenderService>(); 
            services.AddSingleton<IHttpContextAccessor, HttpContextAccessor>(); 
            services.AddSingleton<IActionContextAccessor, ActionContextAccessor>(); 
            services.AddScoped<IIdentityWrapper, IdentityWrapper>(); 
            services.Configure<PaypalConfigSettings>(options => Configuration.GetSection("Paypal").Bind(options)); 
            services.Configure<FacebookConfigSettings>(options => Configuration.GetSection("Facebook").Bind(options)); 
        } 

        private ILoggerFactory GetLoggerFactory() 
        { 
            IServiceCollection serviceCollection = new ServiceCollection(); 
            serviceCollection.AddLogging(builder => 
                builder.AddConsole() 
                    .AddFilter(DbLoggerCategory.Database.Command.Name, LogLevel.Information));  
            return serviceCollection.BuildServiceProvider() 
                .GetService<ILoggerFactory>(); 
        } 

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline. 
        public void Configure(IApplicationBuilder app, IWebHostEnvironment env) 
        { 
            if (env.IsDevelopment()) 
            { 
                app.UseDeveloperExceptionPage(); 
                app.UseDatabaseErrorPage(); 
            } 
            else 
            { 
                app.UseExceptionHandler("/Home/Error"); 
                // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts. 
                // TODO: Do I want HSTS yet...? probably, uncomment when ready 
                // app.UseHsts(); 
            } 

            app.UseHttpsRedirection(); 
            app.UseResponseCaching(); 

            app.UseWhen(x =>  
                x.Request.HttpContext.Request.Path.ToString().StartsWith("/content/images/"), 
                appBranch => 
                { 
                    appBranch.UseImageHandler(); 
                }); 

            app.UseCookiePolicy(); 
            app.UseSession(); 

            app.UseRewriter(new RewriteOptions() 
                .AddRewrite("^embed.js", "dist/embed/embed.js", true)); 

            app.UseStaticFiles(); 
            app.UseRouting(); 
            app.UseAuthentication(); 
            app.UseAuthorization(); 

            app.UseEndpoints(endpoints => 
            { 
                //.. other routes including signalr setup

                // Default route 
                endpoints.MapControllerRoute( 
                    name: "Default", 
                    pattern: "{controller=Root}/{action=Index}/{urlId:regex((?i)^[a-z0-9]{{6}}$)?}/{takerId?}"); 
            }); 

            // Initialise ReactJS.NET. Must be before static files. 
            app.UseReact(config => 
            { 
                // If you want to use server-side rendering of React components, 
                // add all the necessary JavaScript files here. This includes 
                // your components as well as all of their dependencies. 
                // See http://reactjs.net/ for more information. Example: 
                //config 
                //    .AddScript("~/Scripts/First.jsx") 
                //    .AddScript("~/Scripts/Second.jsx"); 

                // If you use an external build too (for example, Babel, Webpack, 
                // Browserify or Gulp), you can improve performance by disabling 
                // ReactJS.NET's version of Babel and loading the pre-transpiled 
                // scripts. Example: 
                //config 
                //    .SetLoadBabel(false) 
                //    .AddScriptWithoutTransform("~/Scripts/bundle.server.js"); 

                config.JsonSerializerSettings.ContractResolver = new DefaultContractResolver(); // Enables PascalCase JSON serialization (default for .NET core is camelCase) 

                config 
                    .SetReuseJavaScriptEngines(true) 
                    .SetLoadBabel(true) 
                    .SetLoadReact(false) 
                    .AddScriptWithoutTransform("./dist/reactjs/runtime.js") 
                    .AddScriptWithoutTransform("./dist/reactjs/vendors.js") 
                    .AddScriptWithoutTransform("./dist/reactjs/components.js"); 
            }); 
        } 
    } 

I'm really not sure how I can go about troubleshooting this without all of the traffic that is going through my site. I'd appreciate any help that I can get.

If you are seeing an exception, include the full exceptions details (message and stack trace).

Exception message:
Stack trace:

[2019-10-13 20:31:13.7365] 1 - ERROR - Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware - An unhandled exception has occurred while executing the request.
MySql.Data.MySqlClient.MySqlException (0x80004005): The Command Timeout expired before the operation completed.
---> MySql.Data.MySqlClient.MySqlException (0x80004005): The Command Timeout expired before the operation completed.
---> System.IO.IOException: Unable to read data from the transport connection: Connection timed out.
---> System.Net.Sockets.SocketException (110): Connection timed out
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Security.SslStream.FillBufferAsync[TReadAdapter](TReadAdapter adapter, Int32 minSize)
at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at MySqlConnector.Protocol.Serialization.StreamByteHandler.<ReadBytesAsync>g__DoReadBytesSync|6_0(ArraySegment`1 buffer_) in C:\projects\mysqlconnector\src\MySqlConnector\Protocol\Serialization\StreamByteHandler.cs:line 39
at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in C:\projects\mysqlconnector\src\MySqlConnector\Protocol\Serialization\BufferedByteReader.cs:line 37
at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPacketAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in C:\projects\mysqlconnector\src\MySqlConnector\Protocol\Serialization\ProtocolUtility.cs:line 410
at MySqlConnector.Protocol.Serialization.ProtocolUtility.DoReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in C:\projects\mysqlconnector\src\MySqlConnector\Protocol\Serialization\ProtocolUtility.cs:line 468
at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 cache, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in C:\projects\mysqlconnector\src\MySqlConnector\Protocol\Serialization\ProtocolUtility.cs:line 463
at MySqlConnector.Protocol.Serialization.StandardPayloadHandler.ReadPayloadAsync(ArraySegmentHolder`1 cache, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in C:\projects\mysqlconnector\src\MySqlConnector\Protocol\Serialization\StandardPayloadHandler.cs:line 42
at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 732
--- End of stack trace from previous location where exception was thrown ---
at MySqlConnector.Core.ServerSession.ReceiveReplyAsyncAwaited(ValueTask`1 task) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 773
at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ResultSet.cs:line 49
at MySql.Data.MySqlClient.MySqlDataReader.ActivateResultSet() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlDataReader.cs:line 111
at MySql.Data.MySqlClient.MySqlDataReader.CreateAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary`2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlDataReader.cs:line 338
at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(IReadOnlyList`1 commands, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\CommandExecutor.cs:line 63
at MySql.Data.MySqlClient.MySqlCommand.ExecuteDbDataReader(CommandBehavior behavior) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlCommand.cs:line 218
at System.Data.Common.DbCommand.ExecuteReader()
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)
at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.QueryingEnumerable`1.Enumerator.MoveNext()
at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression)
at System.Linq.Queryable.First[TSource](IQueryable`1 source, Expression`1 predicate)
at Blah.Bl.Core.Services.QuizStartOgTagsService.GetOgTagsForQuizStart(String urlId, Nullable`1 personalityId) in /Blah.Bl.Core/Services/QuizStartOgTagsService.cs:line 35
at Blah.WebUI.Core.Components.DisplayOgTagsViewComponent.InvokeAsync(String urlId, Nullable`1 personalityId) in /Blah.WebUI.Core/Components/DisplayOgTagsViewComponents.cs:line 22
at Microsoft.AspNetCore.Mvc.ViewComponents.DefaultViewComponentInvoker.InvokeAsyncCore(ObjectMethodExecutor executor, ViewComponentContext context)
at Microsoft.AspNetCore.Mvc.ViewComponents.DefaultViewComponentInvoker.InvokeAsync(ViewComponentContext context)
at Microsoft.AspNetCore.Mvc.ViewComponents.DefaultViewComponentHelper.InvokeCoreAsync(ViewComponentDescriptor descriptor, Object arguments)
at AspNetCore.Views_Quiz_Start.<>c__DisplayClass14_0.<<ExecuteAsync>b__3>d.MoveNext() in /Blah.WebUI.Core/Views/Quiz/Start.cshtml:line 161
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Mvc.Razor.RazorPage.RenderSectionAsyncCore(String sectionName, Boolean required)
at Microsoft.AspNetCore.Mvc.Razor.RazorPage.RenderSection(String name, Boolean required)
at AspNetCore.Views_Shared__QuizLayout.<ExecuteAsync>b__8_0() in /Blah.WebUI.Core/Views/Shared/_QuizLayout.cshtml:line 6
at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperExecutionContext.SetOutputContentAsync()
at AspNetCore.Views_Shared__QuizLayout.ExecuteAsync()
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderLayoutAsync(ViewContext context, ViewBufferTextWriter bodyWriter)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ActionContext actionContext, IView view, ViewDataDictionary viewData, ITempDataDictionary tempData, String contentType, Nullable`1 statusCode)
at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor.ExecuteAsync(ActionContext context, ViewResult result)
at Microsoft.AspNetCore.Mvc.ViewResult.ExecuteResultAsync(ActionContext context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|29_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultFilters>g__Awaited|27_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

Further technical details

MySQL version: 5.6.41
Operating system: Amazon ECS, Docker, .NET Core 3.0 Runtime bionic image
Pomelo.EntityFrameworkCore.MySql version: 3.0.0-rc1.final
Microsoft.AspNetCore.App version: 3.0.0

Other details about my project setup: