How to prevent "Operation not supported" exception when making HTTP request from Cloud Run web instance?

12 hours ago 1
ARTICLE AD BOX

I have an ASP.NET Core 8 Web API running in Google Cloud Run. It uses HttpClient to call another internal API exposed via a Google Cloud HTTP(S) Load Balancer.

The load balancer adds the following header:

Alt-Svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000

Problem

Intermittently, requests fail with this error:

System.Net.NetworkInformation.NetworkInformationException (95): Operation not supported at System.Net.NetworkInformation.NetworkChange.CreateSocket() at System.Net.NetworkInformation.NetworkChange.add_NetworkAddressChanged(NetworkAddressChangedEventHandler value) at System.Net.Http.HttpConnectionPoolManager.StartMonitoringNetworkChanges() at System.Net.Http.HttpConnectionPool.HandleAltSvc(IEnumerable`1 altSvcHeaderValues, Nullable`1 responseAge) at System.Net.Http.HttpConnectionPool.ProcessAltSvc(HttpResponseMessage response) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken) at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendCoreAsync(HttpRequestMessage request, Context context, CancellationToken cancellationToken) at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, CancellationToken cancellationToken, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext) at Polly.AsyncPolicy`1.ExecuteInternalAsync(Func`3 action, Context context, Boolean continueOnCapturedContext, CancellationToken cancellationToken) at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Redacted.Signon.Client.Service.SusMessageHandler.<>c__DisplayClass3_0.<<SendAsync>b__2>d.MoveNext() in /workspace/Redacted.Billing/Redacted.Signon.Client/Service/SusMessageHandler.cs:line 51 --- End of stack trace from previous location --- at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, CancellationToken cancellationToken, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext) at Polly.AsyncPolicy`1.ExecuteInternalAsync(Func`3 action, Context context, Boolean continueOnCapturedContext, CancellationToken cancellationToken) at Redacted.Signon.Client.Service.SusMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /workspace/Redacted.Billing/Redacted.Signon.Client/Service/SusMessageHandler.cs:line 44 at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Redacted.Signon.Client.Service.SusService.GetUsersByIds(IEnumerable`1 susUserIds, Boolean includeProviderPictureUrls, CancellationToken cancellationToken) in /workspace/Redacted.Billing/Redacted.Signon.Client/Service/SusService.cs:line 114 at Redacted.Signon.Client.Service.SusService.GetUserById(String userId, Boolean includeProviderPictureUrls, CancellationToken cancellationToken) in /workspace/Redacted.Billing/Redacted.Signon.Client/Service/SusService.cs:line 126 at Redacted.Billing.Services.Authentication.TokenValidationService.ValidateUserTokenAsync(IServiceProvider services, ClaimsPrincipal principal, ClaimsIdentity claimsIdentity, CancellationToken cancellationToken) in /workspace/Redacted.Billing/Redacted.Billing/Services/Authentication/TokenValidationService.cs:line 101 at Redacted.Billing.Services.Authentication.TokenValidationService.ValidateTokenAsync(IServiceProvider services, ClaimsPrincipal principal, CancellationToken cancellationToken) in /workspace/Redacted.Billing/Redacted.Billing/Services/Authentication/TokenValidationService.cs:line 35 at Redacted.Billing.App_Start.Services.Auth.SusConfiguration.<PostConfigure>b__4_0(TokenValidatedContext ctx) in /workspace/Redacted.Billing/Redacted.Billing/App_Start/Services/Auth.cs:line 77 at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()

From the stack trace, the exception occurs while HttpClient is processing the Alt-Svc header.

Observations

This only occurs in Cloud Run (cannot reproduce locally)

Requests otherwise succeed over HTTP/2

Failures appear intermittently, likely due to internal Alt-Svc caching / backoff behaviour in HttpClient

The exception is thrown when HttpClient starts monitoring network changes (via NetworkChange)

HttpClient configuration

services .AddHttpClient(Constants.RedactedApiClient) .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler() { AllowAutoRedirect = false }) .ConfigureHttpClient(client => { client.DefaultRequestVersion = HttpVersion.Version20; client.DefaultVersionPolicy = HttpVersionPolicy.RequestVersionOrLower; });

Usage:

var client = factory.CreateClient(Constants.RedactedApiClient); using var response = await client.GetAsync(url, cancellationToken); response.EnsureSuccessStatusCode();

What I've tried

Forcing HTTP/2 (see code above):

DefaultRequestVersion = HttpVersion.Version20; DefaultVersionPolicy = HttpVersionPolicy.RequestVersionOrLower;

Disabling HTTP/3 via environment variable:

DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP3SUPPORT=false

Reviewing related issue:
https://github.com/dotnet/runtime/issues/94794

None of these prevent the exception.

How can I solve this issue so that requests flow normally? At the moment, I've resorted to adding a retry when NetworkInformationException is thrown.

Read Entire Article