QueueListener exception causes host restart

[gabbsmo]

While debugging other issues with our web job (SDK v3 and Extensions v5), I found that under high load, our web jobs process will shutdown and then automatically restart.

From Application Event Logs I see .NET Runtime events with Id 1026 and the stack trace below.

I can also see that there are 50+ open TCP connections to Azure Storage from Resource Health. I have 11 queue triggers in my executable. Is this expected? To me it looks like connections are not being reused properly.

I found a previous issue #2767 that was closed because it was transferred to a support ticket and was never updated with the outcome. That issue also mentioned that they were using a newer version of the storage extension package, that included improvements to error handling.

Application: {{ redacted }}.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Threading.Tasks.TaskCanceledException
   at Azure.Core.CancellationHelper.ThrowOperationCanceledException(System.Exception, System.Threading.CancellationToken)
   at Azure.Core.CancellationHelper.ThrowIfCancellationRequested(System.Threading.CancellationToken)
   at Azure.Core.Pipeline.HttpWebRequestTransport+<ProcessInternal>d__8.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Azure.Core.Pipeline.HttpWebRequestTransport+<ProcessAsync>d__7.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Azure.Core.Pipeline.HttpPipelineTransportPolicy+<ProcessAsync>d__4.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Azure.Core.Pipeline.RequestActivityPolicy+<ProcessAsync>d__11.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Azure.Core.Pipeline.ResponseBodyPolicy+<ProcessAsync>d__5.MoveNext()

Exception Info: System.Threading.Tasks.TaskCanceledException
   at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(System.Threading.CancellationToken, System.Threading.CancellationToken, System.Exception, System.TimeSpan)
   at Azure.Core.Pipeline.ResponseBodyPolicy+<ProcessAsync>d__5.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Azure.Core.Pipeline.LoggingPolicy+<ProcessAsync>d__9.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy+<InnerProcessAsync>d__5.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Azure.Core.Pipeline.RedirectPolicy+<ProcessAsync>d__7.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__5.MoveNext()

Exception Info: System.AggregateException
   at Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__5.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy+<InnerProcessAsync>d__5.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Azure.Storage.Queues.MessageIdRestClient+<DeleteAsync>d__11.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Azure.Storage.Queues.QueueClient+<DeleteMessageInternal>d__104.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Azure.Storage.Queues.QueueClient+<DeleteMessageAsync>d__103.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Microsoft.Azure.WebJobs.Host.Queues.QueueProcessor+<DeleteMessageAsync>d__16.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Microsoft.Azure.WebJobs.Host.Queues.QueueProcessor+<CompleteProcessingMessageAsync>d__12.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Microsoft.Azure.WebJobs.Extensions.Storage.Common.Listeners.QueueListener+<ProcessMessageAsync>d__44.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

[gabbsmo]

I have attached a .NET Framework WebJobs project for reproduction. To reproduce the issue locally

1. Start Azurite (Azure Storage Emulator)
2. Run the WebJob1 project
3. Open a PowerShell prompt and run Get-NetTCPConnection -RemotePort 10001
4. Notice that there is one connection per trigger, instead of a single connection being reused for all.

WebJob1.zip

[JAdluri]

Hello @gabbsmo I was able to reproduce the issue and let you know furtherly .