受欢迎的博客标签

Make Custom HttpClient Request emulate a web browser in ASP.NET Core

Published

I use C# in my Web Project. I want to send a GET http request to a website, but I want to send it in a way, so that it will look like a request from a browser.
Now I have a program which sends a GET request and gets a response. I use HttpClient class for sending GET requests.
I know that browsers add some information to their requests like browser name, OS name and the computer name.
My question is how can I add this information to my HttpClient? To what properties all that information (browser name, OS name) should be assigned?

 

Several things to take note of.

1.That site requires you to provide a user agent, or it returns a 500 HTTP error.
2.A GET request to livescore.com responds with a 302 to livescore.us. You need to handle the redirection or directly request livescore.us
3.You need to decompress a gzip-compressed response

 

Table of content

Embedding an css File in HTML Code

Capturing Redirects with HttpClient

 

 an example of  C# HttpClient Get request

https://www.iaspnetcore.com/blog/tag/HttpClient

You should use Fiddler to capture the request that you want to simulate. You need to look at the inspectors > raw. This is an example of a request to the fiddler site from chrome

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zu;q=0.6
Cache-Control: no-cache
Connection: keep-alive
Cookie: Nop.customer=28c0b116-a950-4555-b6c7-ddb4aa5c6c8c
DNT: 1
Host: www.iaspnetcore.com
Pragma: no-cache
Referer: https://www.iaspnetcore.com/blog/tag/HttpClient
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36

Make HttpClient Request emulate a web browser,Custom header to Httpclient request.

Several things to take note of.

1.That site requires you to provide a user agent, or it returns a 500 HTTP error.
2.A GET request to livescore.com responds with a 302 to livescore.us. You need to handle the redirection or directly request livescore.us
3.You need to decompress a gzip-compressed response

Step 1.Create an HttpClient instance correctly  in Controller

First we need to create  an HttpClient instance correctly in ASP.NET Core by creating a GoodController.

Examine the static HttpClient property.HttpClient is intended to be instantiated once and reused throughout the life of an application. HttpClient is intended to be instantiated once and re-used throughout the life of an application. Instantiating an HttpClient class for every request will exhaust the number of sockets available under heavy loads. This will result in SocketException errors. Below is an example using HttpClient correctly.

using System.Net.Http;
 public class GoodController : Controller
    {
        //A new HttpClient is created.
        static readonly HttpClient client = new HttpClient();  // correctly
        private readonly ILogger<GoodController> _logger;

        public GoodController(ILogger<GoodController> logger)
        {

            _logger = logger;
           

        }

step 2.Adding Custom Http Headers to HttpClient  before  send a request

The next step is to create a HttpRequestMessage class that our MVC application can use to add http headers.
you need to add http headers to the HttpClient before you send a request to a web server.

1.Create a HttpRequestMessage

2.set the Method to GET

3.set your headers and then use SendAsync instead of GetAsync.

urlModel.cs

Create a urlModel object

we need to create a urlModel object to hold information about page URL

 public class urlModel
    {
        public string WebUrl { set; get; } = String.Empty;
        public string Content { set; get; } = String.Empty;
    }

Set custom http headers

public class GoodController : Controller
    {
     ....
        [HttpPost]
        [ValidateAntiForgeryToken]
        public async Task<IActionResult> Index(urlModel model)
        {

         var targetUri = new Uri(model.WebUrl);
          
            var requestMessage = new HttpRequestMessage();

            _logger.LogInformation($"model.WebUrl:{model.WebUrl}\r\n");

            _logger.LogInformation($"before set head:requestMessage.ToString():\r\n{requestMessage.ToString()}");

            requestMessage.Headers.TryAddWithoutValidation("Accept", " text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng");
            //  requestMessage.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
            requestMessage.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36");
            requestMessage.Headers.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zu;q=0.6");

            // CopyFromOriginalRequestContentAndHeaders(this.HttpContext, requestMessage);

            requestMessage.RequestUri = targetUri;
            requestMessage.Headers.Host = targetUri.Host;
            requestMessage.Method = new HttpMethod(HttpMethods.Get);
      ....
     }
}

step 3.Making the Call

Call asynchronous network methods in a try/catch block to handle exceptions.

 try
            {
                HttpResponseMessage responseMessage =  await httpClient.SendAsync(requestMessage, HttpCompletionOption.ResponseHeadersRead, this.HttpContext.RequestAborted);
                responseMessage.EnsureSuccessStatusCode();
                //failure
                if (responseMessage.StatusCode != HttpStatusCode.OK)
                {
                    model.Content = $"StatusCode:{responseMessage.StatusCode.ToString()}";
                    return View(model);
                }

                responseBody = await responseMessage.Content.ReadAsStringAsync();
            }
            catch (HttpRequestException e)
            {
                Console.WriteLine("\nException Caught!");
                Console.WriteLine("Message :{0} ", e.Message);

                model.Content = $"\nException Caught!Message :"+ e.Message.ToString();
                return View(model);
            }

 

C# HttpClient status code

HTTP response status codes indicate whether a specific HTTP request has been successfully completed. Responses are grouped in five classes:

Informational responses (100–199)
Successful responses (200–299)
Redirects (300–399)
Client errors (400–499)
Server errors (500–599)

Comparing the  output from a call to the homepage before and after adding  http headers gives:

Before:

before set head:requestMessage.ToString():
      Method: GET, RequestUri: '<null>', Version: 1.1, Content: <null>, Headers:
      {
      }

After:

after set head:requestMessage.ToString():
      Method: GET, RequestUri: 'https://www.bing.com', Version: 1.1, Content: <null>, Headers:
      {
        Accept: text/html
        Accept: application/xhtml+xml
        Accept: application/xml; q=0.9
        Accept: image/webp
        Accept: image/apng
        User-Agent: Mozilla/5.0
        User-Agent: (Windows NT 6.3; Win64; x64)
        User-Agent: AppleWebKit/537.36
        User-Agent: (KHTML, like Gecko)
        User-Agent: Chrome/81.0.4044.129
        User-Agent: Safari/537.36
        Accept-Language: en-US
        Accept-Language: en; q=0.9
        Accept-Language: zh-CN; q=0.8
        Accept-Language: zh; q=0.7
        Accept-Language: zu; q=0.6
        Host: www.bing.com
      }

 

 

 

other:

private static readonly HttpClient _HttpClient = new HttpClient();

private static async Task<string> GetResponse(string url)
{
    using (var request = new HttpRequestMessage(HttpMethod.Get, new Uri(url)))
    {
        request.Headers.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml");
        request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
        request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0");
        request.Headers.TryAddWithoutValidation("Accept-Charset", "ISO-8859-1");

        using (var response = await _HttpClient.SendAsync(request).ConfigureAwait(false))
        {
            response.EnsureSuccessStatusCode();
            using (var responseStream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
            using (var decompressedStream = new GZipStream(responseStream, CompressionMode.Decompress))
            using (var streamReader = new StreamReader(decompressedStream))
            {
                return await streamReader.ReadToEndAsync().ConfigureAwait(false);
            }
        }
    }
}

call such like:

var response = await GetResponse("http://www.livescore.com/").ConfigureAwait(false); 
// or
 var response = GetResponse("http://www.livescore.com/").Result;

step 2:add compression support:

var compressclient = new HttpClient(new HttpClientHandler() 
{ 
AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip 
}); 

C# HttpClient POST request

var request = new HttpRequestMessage {
    RequestUri = new Uri("[your request url string]"),
    Method = HttpMethod.Post,
    Headers = {
        { "X-Version", "1" } // HERE IS HOW TO ADD HEADERS,
        { HttpRequestHeader.Authorization.ToString(), "[your authorization token]" },
        { HttpRequestHeader.ContentType.ToString(), "multipart/mixed" },//use this content type if you want to send more than one content type
    },
    Content = new MultipartContent { // Just example of request sending multipart request
        new ObjectContent<[YOUR JSON OBJECT TYPE]>(
            new [YOUR JSON OBJECT TYPE INSTANCE](...){...}, 
            new JsonMediaTypeFormatter(), 
            "application/json"), // this will add 'Content-Type' header for the first part of request
        new ByteArrayContent([BINARY DATA]) {
            Headers = { // this will add headers for the second part of request
                { "Content-Type", "application/Executable" },
                { "Content-Disposition", "form-data; filename=\"test.pdf\"" },
            },
        },
    },
};

 

other:

public class HttpHeaderSetHeaderExample4_3Before {

 private static String URI = "http://www.google.com";

 public static void main(String[] args) throws ClientProtocolException, IOException {

  Header header1 = new BasicHeader(HttpHeaders.CONTENT_TYPE, "application/json");
  Header header2 = new BasicHeader(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate, br");
  Header header3 = new BasicHeader(HttpHeaders.ACCEPT_LANGUAGE, "en-US,en;q=0.5");
  Header header4 = new BasicHeader(HttpHeaders.ACCEPT,
    "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
  Header header5 = new BasicHeader(HttpHeaders.CONNECTION, "keep-alive");
  Header header6 = new BasicHeader(HttpHeaders.CACHE_CONTROL, "max-age=0");
  Header header7 = new BasicHeader(HttpHeaders.USER_AGENT,
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:50.0) Gecko/20100101 Firefox/50.0");
  Header header8 = new BasicHeader(HttpHeaders.HOST, "http://www.google.com");

  List<Header> headers = Arrays.asList(header1, header2, header3, header4, header5, header6, header7, header8);
  HttpClient client = HttpClients.custom().setDefaultHeaders(headers).build();
  HttpUriRequest request = RequestBuilder.get().setUri(URI).build();
  client.execute(request);

 }

Capturing Redirects with HttpClient

If we pass an HttpClientHandler to the HttpClient, it has a property to allow prevention of automatic redirection. The property is “AllowAutoRedirect.” The setup then looks like the below code. Since the HttpClient doesn’t follow redirects, we can check the status code and perform our own logic on redirects. You can see in the handling of the redirect, the “MakeRequest” method effectively becomes a recursive routine. A redirect can specify an absolute or relative URL. This must be handled as indicated in the code as well.

public bool MakeRequest(url)
{
    using (var client = new HttpClient(new HttpClientHandler() { AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip }) { Timeout = _timeout })
    {
        var request = new HttpRequestMessage()
        {
            RequestUri = new Uri(url),
            Method = HttpMethod.Get
        };
 
        HttpResponseMessage response = client.SendAsync(request).Result;
        var statusCode = (int)response.StatusCode;
 
        // We want to handle redirects ourselves so that we can determine the final redirect Location (via header)
        if (statusCode >= 300 && statusCode <= 399)
        {
            var redirectUri = response.Headers.Location;
            if (!redirectUri.IsAbsoluteUri)
            {
                redirectUri = new Uri(request.RequestUri.GetLeftPart(UriPartial.Authority) + redirectUri);
            }
            _status.AddStatus(string.Format("Redirecting to {0}", redirectUri));
            return MakeRequest(redirectUri);
        }
        else if (!response.IsSuccessStatusCode)
        {
            throw new Exception();
        }
 
        return true;
    }
}

 

Useful links

https://stackoverflow.com/questions/15026953/httpclient-request-like-browser