受欢迎的博客标签

Make HTTP requests emulate a web browser using IHttpClientFactory in ASP.NET Core

Published

Make HttpClient Request emulate a web browserr with c# 

Several things to take note of.

1.That site requires you to provide a user agent, or it returns a 500 HTTP error.
2.A GET request to livescore.com responds with a 302 to livescore.us. You need to handle the redirection or directly request livescore.us
3.You need to decompress a gzip-compressed response

 

step 1:  register IHttpClientFactory 

register IHttpClientFactory  by calling AddHttpClient.

path:/Startup.cs

using System.Net.Http
public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddHttpClient();

 

step 2: Create an HttpClient instance

uses IHttpClientFactory to create an HttpClient instance.

 public class AdminController : Controller
    {
        
        private readonly IHttpClientFactory _clientFactory;

        private readonly ILogger<AdminController> _logger;

        public AdminController(ILogger<AdminController> logger, IHttpClientFactory clientFactory)
        {
          
            _logger = logger;
            _clientFactory = clientFactory;
        }
    public async Task OnGet()
    {
        var request = new HttpRequestMessage(HttpMethod.Get,
            "https://api.github.com/repos/aspnet/AspNetCore.Docs/branches");
        request.Headers.Add("Accept", "application/vnd.github.v3+json");
        request.Headers.Add("User-Agent", "HttpClientFactory-Sample");

        var client = _clientFactory.CreateClient();

        var response = await client.SendAsync(request);

        if (response.IsSuccessStatusCode)
        {
            using var responseStream = await response.Content.ReadAsStreamAsync();
          
        }

     string responseBody = await response.Content.ReadAsStringAsync();
      
    }
}

 

You should use Fiddler to capture the request that you want to simulate. You need to look at the inspectors > raw. This is an example of a request to the fiddler site from chrome

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zu;q=0.6
Cache-Control: no-cache
Connection: keep-alive
Cookie: Nop.customer=28c0b116-a950-4555-b6c7-ddb4aa5c6c8c
DNT: 1
Host: www.iaspnetcore.com
Pragma: no-cache
Referer: https://www.iaspnetcore.com/blog/tag/HttpClient
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36

 

private static readonly HttpClient _HttpClient = new HttpClient();

private static async Task<string> GetResponse(string url)
{
    using (var request = new HttpRequestMessage(HttpMethod.Get, new Uri(url)))
    {
        request.Headers.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml");
        request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
        request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0");
        request.Headers.TryAddWithoutValidation("Accept-Charset", "ISO-8859-1");

        using (var response = await _HttpClient.SendAsync(request).ConfigureAwait(false))
        {
            response.EnsureSuccessStatusCode();
            using (var responseStream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
            using (var decompressedStream = new GZipStream(responseStream, CompressionMode.Decompress))
            using (var streamReader = new StreamReader(decompressedStream))
            {
                return await streamReader.ReadToEndAsync().ConfigureAwait(false);
            }
        }
    }
}

call such like:

var response = await GetResponse("http://www.livescore.com/").ConfigureAwait(false); 
// or
 var response = GetResponse("http://www.livescore.com/").Result;

step 2:add compression support:

var compressclient = new HttpClient(new HttpClientHandler() 
{ 
AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip 
});