ARTICLE AD BOX
I have an ASP.NET Core 8 Web API running in Google Cloud Run. It uses HttpClient to call another internal API exposed via a Google Cloud HTTP(S) Load Balancer.
The load balancer adds the following header:
Alt-Svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000Problem
Intermittently, requests fail with this error:
System.Net.NetworkInformation.NetworkInformationException (95): Operation not supported at System.Net.NetworkInformation.NetworkChange.CreateSocket() at System.Net.NetworkInformation.NetworkChange.add_NetworkAddressChanged(NetworkAddressChangedEventHandler value) at System.Net.Http.HttpConnectionPoolManager.StartMonitoringNetworkChanges() at System.Net.Http.HttpConnectionPool.HandleAltSvc(IEnumerable`1 altSvcHeaderValues, Nullable`1 responseAge) at System.Net.Http.HttpConnectionPool.ProcessAltSvc(HttpResponseMessage response) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken) at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendCoreAsync(HttpRequestMessage request, Context context, CancellationToken cancellationToken) at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, CancellationToken cancellationToken, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext) at Polly.AsyncPolicy`1.ExecuteInternalAsync(Func`3 action, Context context, Boolean continueOnCapturedContext, CancellationToken cancellationToken) at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Redacted.Signon.Client.Service.SusMessageHandler.<>c__DisplayClass3_0.<<SendAsync>b__2>d.MoveNext() in /workspace/Redacted.Billing/Redacted.Signon.Client/Service/SusMessageHandler.cs:line 51 --- End of stack trace from previous location --- at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, CancellationToken cancellationToken, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext) at Polly.AsyncPolicy`1.ExecuteInternalAsync(Func`3 action, Context context, Boolean continueOnCapturedContext, CancellationToken cancellationToken) at Redacted.Signon.Client.Service.SusMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /workspace/Redacted.Billing/Redacted.Signon.Client/Service/SusMessageHandler.cs:line 44 at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Redacted.Signon.Client.Service.SusService.GetUsersByIds(IEnumerable`1 susUserIds, Boolean includeProviderPictureUrls, CancellationToken cancellationToken) in /workspace/Redacted.Billing/Redacted.Signon.Client/Service/SusService.cs:line 114 at Redacted.Signon.Client.Service.SusService.GetUserById(String userId, Boolean includeProviderPictureUrls, CancellationToken cancellationToken) in /workspace/Redacted.Billing/Redacted.Signon.Client/Service/SusService.cs:line 126 at Redacted.Billing.Services.Authentication.TokenValidationService.ValidateUserTokenAsync(IServiceProvider services, ClaimsPrincipal principal, ClaimsIdentity claimsIdentity, CancellationToken cancellationToken) in /workspace/Redacted.Billing/Redacted.Billing/Services/Authentication/TokenValidationService.cs:line 101 at Redacted.Billing.Services.Authentication.TokenValidationService.ValidateTokenAsync(IServiceProvider services, ClaimsPrincipal principal, CancellationToken cancellationToken) in /workspace/Redacted.Billing/Redacted.Billing/Services/Authentication/TokenValidationService.cs:line 35 at Redacted.Billing.App_Start.Services.Auth.SusConfiguration.<PostConfigure>b__4_0(TokenValidatedContext ctx) in /workspace/Redacted.Billing/Redacted.Billing/App_Start/Services/Auth.cs:line 77 at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()From the stack trace, the exception occurs while HttpClient is processing the Alt-Svc header.
Observations
This only occurs in Cloud Run (cannot reproduce locally)
Requests otherwise succeed over HTTP/2
Failures appear intermittently, likely due to internal Alt-Svc caching / backoff behaviour in HttpClient
The exception is thrown when HttpClient starts monitoring network changes (via NetworkChange)
HttpClient configuration
services .AddHttpClient(Constants.RedactedApiClient) .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler() { AllowAutoRedirect = false }) .ConfigureHttpClient(client => { client.DefaultRequestVersion = HttpVersion.Version20; client.DefaultVersionPolicy = HttpVersionPolicy.RequestVersionOrLower; });Usage:
var client = factory.CreateClient(Constants.RedactedApiClient); using var response = await client.GetAsync(url, cancellationToken); response.EnsureSuccessStatusCode();What I've tried
Forcing HTTP/2 (see code above):
DefaultRequestVersion = HttpVersion.Version20; DefaultVersionPolicy = HttpVersionPolicy.RequestVersionOrLower;Disabling HTTP/3 via environment variable:
DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP3SUPPORT=falseReviewing related issue:
https://github.com/dotnet/runtime/issues/94794
None of these prevent the exception.
How can I solve this issue so that requests flow normally? At the moment, I've resorted to adding a retry when NetworkInformationException is thrown.
