アプリケーションの1つでECSからS3クライアントと対話するときに、断続的なエラー(「SSLハンドシェイクがOpenSSLエラーで失敗しました-SSL_ERROR_ZERO_RETURN」)が発生しています。 エラーはランダムで時折発生するようであり、S3からオブジェクトをロードまたは取得しているときに発生する可能性があります。
これは、S3からオブジェクトをロードした直後に取得したときに発生した例外の例です。
{
"message": "The SSL connection could not be established, see inner exception.",
"data": {},
"innerException": {
"ClassName": "System.Security.Authentication.AuthenticationException",
"Message": "Authentication failed, see inner exception.",
"Data": null,
"InnerException": {
"message": "SSL Handshake failed with OpenSSL error - SSL_ERROR_ZERO_RETURN.",
"data": {},
"stackTrace": " at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, Byte[] recvBuf, Int32 recvOffset, Int32 recvCount, Byte[]& sendBuf, Int32& sendCount)
\n at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, SecurityBuffer inputBuffer, SecurityBuffer outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)",
"source": "System.Net.Security",
"hResult": -2146233088
},
"HelpURL": null,
"StackTraceString": " at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
\n at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
\n at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)\n--- End of stack trace from previous location where exception was thrown ---
\n at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
\n at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
\n--- End of stack trace from previous location where exception was thrown ---
\n at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)",
"RemoteStackTraceString": null,
"RemoteStackIndex": 0,
"ExceptionMethod": null,
"HResult": -2146233087,
"Source": "System.Private.CoreLib",
"WatsonBuckets": null
},
"stackTrace": " at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
\n at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
\n at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
\n at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
\n at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
\n at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
\n at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
\n at MyNameSpace.AwsS3Context.GetContentVersionId(Guid id) in /filePath/AwsS3Context.cs:line XX
\n at MyNameSpace.AwsS3Context.PutEncryptedContentAsync(Guid id, Stream content) in /filePath/AwsS3Context.cs:line XX
\n at MyNameSpace.ManagerClass.SaveContentAsync(Guid id, Stream content) in /filePath/ManagerClass.cs:line XX
\n at MyNameSpace.ControllerClass.CreateAsync() in /filePath/ControllerClass.cs:line XX
\n at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
\n at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
\n at MyNameSpace.APIClass.Startup.<Configure>b__8_1(HttpContext context, Func`1 next) in /filePath/Startup.cs:line XX
\n at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)",
"source": "System.Net.Http",
"hResult": -2146233087
}
S3クライアントは次のように構成されています。
// At startup
services.AddAWSService<IAmazonS3>();
// S3 client class
public class AwsS3Context : IAwsS3Context
{
private readonly IAmazonS3 awsS3Client;
public AwsS3Context(IAmazonS3 awsS3Client)
{
this.awsS3Client = awsS3Client;
}
public async Task<string> GetContentVersionId(Guid id)
{
var getS3Request = new GetObjectRequest
{
BucketName = "bucket_name",
Key = id.ToString()
};
using (var s3Item = await awsS3Client.GetObjectAsync(getS3Request).ConfigureAwait(false))
{
return s3Item.VersionId;
}
}
public async Task<string> PutEncryptedContentAsync(Guid id, Stream content)
{
var config = new TransferUtilityConfig();
try
{
using (var awsTransferUtility = new TransferUtility(awsS3Client, config))
{
var request = new TransferUtilityUploadRequest
{
BucketName = "bucket_name",
Key = id.ToString(),
InputStream = content,
ServerSideEncryptionMethod = ServerSideEncryptionMethod.AES256
};
await awsTransferUtility.UploadAsync(request, CancellationToken.None).ConfigureAwait(false);
return await GetContentVersionId(Guid);
}
}
finally
{
content?.Dispose();
}
}
}
S3を呼び出すときの断続的なSSLハンドシェイクエラーは予期されていません。
このエラーを一貫して再現することはできません。 テストするために、単一のコンテナーをプロビジョニングする開発環境でアプリケーションを分離し、それを数日間定期的にロードして、リクエストのレートとサイズを調整してみました。 この後、前述の例外が数回発生しましたが、エラーを引き起こす根本的な原因や状態は明らかではありません。
これは、ECSで実装されたシンプルなAPIであり、さまざまなタイプのファイルを受信して内部でS3バケットに保存し、クライアントにリソースIDを提供します。
S3からデータを読み取っているときに同じ問題が断続的に発生する
AWSSDK.S3、v.3.3.110.50のAmazon.S3.Transfer.TransferUtility
を使用する場合、同じタイプの問題が発生します
同じエラーが断続的に発生します。 Lambdaで実行しています。
Exception Message: SSL Handshake failed with OpenSSL error - SSL_ERROR_ZERO_RETURN.
Stack Trace:
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
at internal method...
同じ問題が発生しました。
低スペックのEC2マシンで大量のファイルセット(サイズの異なる約80ファイル)をダウンロードすると、常に発生します。
私の場合、ファイル数が少ない(例:10)場合は問題ありません。
ConcurrentServiceRequests = 3およびDownloadAsyncメソッドでTransferUtilityクラスを使用しています。
SemaphoreSlim(例:maxThreadCount = 3)を使用してスレッド数を制限する(1スレッド= 1ファイルのダウンロード)ことは、私にとっての回避策です。
更新
AWS TransferUtilityConfigのドキュメントでは、ConcurrentServiceRequestsプロパティによって、ファイルのアップロード/ダウンロードに使用されるアクティブなスレッドの数または同時非同期Webリクエストの数が決定されると記載されています。 SDKコードで、このプロパティはマルチパートアップロードメソッドでのみ使用されることがわかりました。
最も参考になるコメント
AWSSDK.S3、v.3.3.110.50の
Amazon.S3.Transfer.TransferUtility
を使用する場合、同じタイプの問題が発生します