C# HttpClient randomly getting error while copying content to a stream

I use the following code to preform a HTTP GET request to a remote ASP .Net WebApi running on an IIS server. The request returns a JSON object of about 5 Megabyte in size. Almost all of the time, this request succeeds and I get correct JSON response, but something (about 2-3% of the time, randomly distributed) I am getting an incomplete JSON in the response body. The response code is 200 OK, but the JSON data is being cut off in a random location. In Fiddler I indeed see the response JSON is cut in a random location, So it seems like the server is closing the connection in the middle for some reason.

I’ve read in other SO threads that moving to HTTP V1.0 or setting the Connection header to “closed” might solve the problem but it didn’t worked for me. Is it an IIS server setting that i’m missing here? Or a wrong client side configuration?

The problem seems to not reproduce when using same code on other routes from same api which returns much smaller JSON response (under 1 Megabyte).

It also important to note that we have some load balancer mechanism between our client and the IIS server (Incapsula). This load balancer is also providing DDos prevention.

Here’s my C# code (I’m using the latest HttpClient version):

 using (var httpClient = new HttpClient())
 {
     var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, new Uri("https://api-real.bgtm.com/Orders"));

     using (var httpResponseMessage = await httpClient.SendAsync(httpRequestMessage))
     {
          InstResponse s = httpResponseMessage.Content.ReadAsAsync<InstResponse>().GetAwaiter().GetResult();
         Console.Write("Finished without error.");
     }
 }

here is the full exception details I’m getting when trying to read the incomplete JSON from the response stream:

{System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: The server returned an invalid or unrecognized response.
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.CopyToExactLengthAsync(Stream destination, UInt64 length, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at ConsoleApp1.Program.MainAsync ()}

I can also add a screenshot from WireShark which shows the server is the first to send the FIN message. Server IP is 107.154.192.59, Client IP is 10.10.80.146:ScreenShot
Pretty hopeless, Is someone ever witnessed this kind of inconsistent behavior?

Leave a Reply