受欢迎的博客标签

HttpClient用HttpClientHandler添加Refer、Cookie及UserAgent请求头,解析中文GBk编码

Published

There are various HTTP message handles that can be used. These include the following.

 

DelegatingHandler - A class used to plug a handler into a handler chain.
HttpMessageHandler - A simple class to derive from that supports the most common requirements for most applications. detail:https://thecodebuzz.com/httpclienthandler-client-certificate-basic-auth-httpclientfactory/
HttpClientHandler - A class that operates at the bottom of the handler chain that actually handles the HTTP transport operations.
WebRequestHandler - A specialty class that operates at the bottom of the handler chain class that handles HTTP transport operations with options that are specific to the System.Net.HttpWebRequest object.

基本操作

HttpClient更加简洁,下面就是一个下载www.windows.com页面的示例:

1.请求

string uri = "http://www.windows.com/";
    HttpClient client = new HttpClient();
    string body = await client.GetStringAsync(uri); 

它支持编码识别和对压缩的http流解压,省去了我们的不少代码。除GetStringAsync()之外,

还有GetByteArrayAsync()、GetStreamAsync()、PostAsync ()、DeleteAsync()等函数。

2.响应

HttpClient.GetStringAsync()是一个简化的函数,用这个函数的时候,我们看不到HttpResponse的相关信息,如果需要看到Http响应的信息,可以用如下标准方式:

   

HttpResponseMessage response = await client.GetAsync(uri);
    response.EnsureSuccessStatusCode();
    string responseBody = await response.Content.ReadAsStringAsync(); 

 3.请求头

自定义HttpHeader

有时我们需要在发送Get请求时在HttpHeader中加入一些额外的信息,常见的的有Refer、Cookie及UserAgent等。这个时候我们就要用到HttpClientHandler了,具体方法如下:

首先自定义一个HttpClienHanlder类,重载SendAsync方法。
   

HttpClientHandler handler = new HttpClientHandler()
{
    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};

string uri = "http://www.windows.com/";
    HttpClient client = new HttpClient();

    class MyHttpClienHanlder:HttpClientHandler
    {
        protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            request.Headers.Referrer = new Uri("http://www.google.com/");
            request.Headers.Add("UserAgent", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727)");

            return base.SendAsync(request, cancellationToken);
        }
    }

在HttpClient构造函数中加入自定义的HttpClienHanlder类。 
   

string uri = "http://www.windows.com/";
using (var client = new HttpClient(handler))
{
    
   
    string body = await client.GetStringAsync(uri);
    // your code
}

可见,HttpClienHanlder其实就是是一个常见的代理模式的设计,它在HttpClient.GetStringAsync()中加了一层封装,拦截了HttpClient的输入和输出,从而实现一些自定义的操作。

 

HttpClient常见问题

HttpClient中文乱码

排查步骤

HttpClient支持编码识别和对压缩的http流解压。

首先排查是否对对压缩的http流正确的解压

step 1:gzip解压

  HttpClientHandler handler = new HttpClientHandler();
            handler.AutomaticDecompression = DecompressionMethods.GZip;


HttpClient client = new HttpClient(handler)

原因:

client.DefaultRequestHeaders.TryAddWithoutValidation(HeaderNames.AcceptEncoding, "gzip, deflate, sdch");

内容是压缩了的

 

 

编码问题

1.如果HttpResponse的HttpHeader中包含CharSet信息时,   string responseBody = await response.Content.ReadAsStringAsync();本身支持编码识别。

Content-Type:text/html; charset=utf-8

2.如果HttpResponse的HttpHeader中不含CharSet信息时,便采用默认编码方式进行字符串解码,它的默认编码方式是无法解析中文的,此时便会出现中文乱码。

一种常见的做法是:如果HttpHeader中不含CharSet信息时,采用GBK方式来解码。要实现这个功能的话,还是需要用到前面提到的HttpClientHandler

 

 class MyHttpClienHanlder:HttpClientHandler
    {
        protected async override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            var rsponse = await base.SendAsync(request, cancellationToken);
            var contentType = rsponse.Content.Headers.ContentType;
            if (string.IsNullOrEmpty(contentType.CharSet))
            {
                contentType.CharSet = "GBK";
            }
            return rsponse;
        }
    }

当然,这么做仍然不是很完善,有的时候如果要更精确的话还需要从Html页面中获取charset信息,甚至通过相应的库函数进行编码猜测。这儿我写了一个稍微完善的版本:

<!DOCTYPE html>
<html>
<head>
<title>Asp .Net Core cstml中文乱码解决方案-iaspnetcore.com</title>
<meta http-equiv="Content-type" content="text/html; charset=utf-8" /

 

 1     class HtmlTextHandler : HttpClientHandler
 2     {
 3         protected async override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
 4         {
 5             var response = await base.SendAsync(request, cancellationToken);
 6 
 7             var contentType = response.Content.Headers.ContentType;
 8             contentType.CharSet = await getCharSetAsync(response.Content);
 9 
10             return response;
11         }
12 
13         private async Task<string> getCharSetAsync(HttpContent httpContent)
14         {
15             var charset = httpContent.Headers.ContentType.CharSet;
16             if (!string.IsNullOrEmpty(charset))
17                 return charset;
18 
19             var content = await httpContent.ReadAsStringAsync();
20             var match = Regex.Match(content, @"charset=(?<charset>.+?)""", RegexOptions.IgnoreCase);
21             if (!match.Success)
22                 return charset;
23 
24             return match.Groups["charset"].Value;
25         }
26     }
View Code
  1. 响应内容过长导致HttpRequestException

HttpClient有一个属性MaxResponseContentBufferSize,它表示的是读取响应内容时最大字节数缓冲区。它的默认值是64k,当页面内容很多,超过64k的时候,就会抛出一个HttpRequestException,导致Get失败。

这个属性必须是个正整数,也就是说,它是不支持自适应的,这个非常令人费解,不知道MS为什么非要自己估算页面大小,在Get操作前支持为合适的值,这个是个不够好用的地方。

我查了一下MSDN,目前对这个属性的说明比较少,不知道更改这个值的大小会影响什么地方。即使把他设置成int.Max貌似也不会有过多的内存占用。不过为了安全起见,还是把它设置在一个合理的范围吧,像我一般就把它设置为1m。(PS: 在最新的.Net 4.5 RC中,这个值已经更新成了int.MaxValue,希望RTM版不要恢复成64k,确实不够用)

    HttpClient client = new HttpClient() { MaxResponseContentBufferSize = 1024 * 1024 }; 

最后提一个不是问题的问题:HttpClient全部都是异步方法,没有同步方法,如果要在同步函数中使用,必须通过Task.Wait()来等待任务完成,稍稍有些不便。