Make HttpClient Request emulate a web browserr with c#
Several things to take note of.
1.That site requires you to provide a user agent, or it returns a 500 HTTP error.
2.A GET request to livescore.com responds with a 302 to livescore.us. You need to handle the redirection or directly request livescore.us
3.You need to decompress a gzip-compressed response
step 1: register IHttpClientFactory
register IHttpClientFactory by calling AddHttpClient.
path:/Startup.cs
using System.Net.Http
public class Startup
{
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}
public IConfiguration Configuration { get; }
public void ConfigureServices(IServiceCollection services)
{
services.AddHttpClient();
step 2: Create an HttpClient instance
uses IHttpClientFactory to create an HttpClient instance.
public class AdminController : Controller
{
private readonly IHttpClientFactory _clientFactory;
private readonly ILogger<AdminController> _logger;
public AdminController(ILogger<AdminController> logger, IHttpClientFactory clientFactory)
{
_logger = logger;
_clientFactory = clientFactory;
}
public async Task OnGet()
{
var request = new HttpRequestMessage(HttpMethod.Get,
"https://api.github.com/repos/aspnet/AspNetCore.Docs/branches");
request.Headers.Add("Accept", "application/vnd.github.v3+json");
request.Headers.Add("User-Agent", "HttpClientFactory-Sample");
var client = _clientFactory.CreateClient();
var response = await client.SendAsync(request);
if (response.IsSuccessStatusCode)
{
using var responseStream = await response.Content.ReadAsStreamAsync();
}
string responseBody = await response.Content.ReadAsStringAsync();
}
}
You should use Fiddler to capture the request that you want to simulate. You need to look at the inspectors > raw. This is an example of a request to the fiddler site from chrome
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zu;q=0.6
Cache-Control: no-cache
Connection: keep-alive
Cookie: Nop.customer=28c0b116-a950-4555-b6c7-ddb4aa5c6c8c
DNT: 1
Host: www.iaspnetcore.com
Pragma: no-cache
Referer: https://www.iaspnetcore.com/blog/tag/HttpClient
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36
private static readonly HttpClient _HttpClient = new HttpClient();
private static async Task<string> GetResponse(string url)
{
using (var request = new HttpRequestMessage(HttpMethod.Get, new Uri(url)))
{
request.Headers.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml");
request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0");
request.Headers.TryAddWithoutValidation("Accept-Charset", "ISO-8859-1");
using (var response = await _HttpClient.SendAsync(request).ConfigureAwait(false))
{
response.EnsureSuccessStatusCode();
using (var responseStream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
using (var decompressedStream = new GZipStream(responseStream, CompressionMode.Decompress))
using (var streamReader = new StreamReader(decompressedStream))
{
return await streamReader.ReadToEndAsync().ConfigureAwait(false);
}
}
}
}
call such like:
var response = await GetResponse("http://www.livescore.com/").ConfigureAwait(false);
// or
var response = GetResponse("http://www.livescore.com/").Result;
step 2:add compression support:
var compressclient = new HttpClient(new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
});