set a cookie on HttpClient's HttpClientHandler()
C# HttpClient Cookie验证解决方法(写文件 序列化传回来的cookie)
Several things to take note of.
1.That site requires you to provide a user agent, or it returns a 500 HTTP error.
2.A GET request to livescore.com responds with a 302 to livescore.us. You need to handle the redirection or directly request livescore.us
3.You need to decompress a gzip-compressed response
//验证
代码代码如下:
HttpClient httpClient = new HttpClient(url, null, true);
httpClient.PostingData.Add(key,value);//登录用户名
httpClient.PostingData.Add(key,value);//密码
string str = httpClient.GetString();写文件 序列化传回来的cookie
复制代码代码如下:
CookieCollection cookies = httpClient.Context.Cookies;//保存一个全局的cookie文件
FileStream fileStream = new FileStream("xxx.dat", FileMode.Create);
BinaryFormatter b = new BinaryFormatter();
b.Serialize(fileStream, cookies);
fileStream.Close();读文件 反序列化cookies 赋给httpClient的cookies
复制代码代码如下:
FileStream fileStream = new FileStream("xxx.dat", FileMode.Open, FileAccess.Read, FileShare.Read);
BinaryFormatter b = new BinaryFormatter();
CookieCollection cookies = b.Deserialize(fileStream) as CookieCollection;
HttpClient httpClient = new HttpClient("url");//取值的url
httpClient.Context.Cookies = cookies;
string str = httpClient.GetString();1.Decompressing GZip Stream from HTTPClient Response
Just instantiate HttpClient like this:
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using (var client = new HttpClient(handler))
{
// your code
}https://stackoverflow.com/questions/20990601/decompressing-gzip-stream-from-httpclient-response
2.get cookie out of response with HttpClient
private async Task<string> Login(string username, string password)
{
try
{
string url = "http://app.agelessemail.com/account/login/";
Uri address = new Uri(url);
var postData = new List<KeyValuePair<string, string>>
{
new KeyValuePair<string, string>("username", username),
new KeyValuePair<string, string>("password ", password)
};
HttpContent content = new FormUrlEncodedContent(postData);
var cookieJar = new CookieContainer();
var handler = new HttpClientHandler
{
CookieContainer = cookieJar,
UseCookies = true,
UseDefaultCredentials = false
};
var client = new HttpClient(handler)
{
BaseAddress = address
};
HttpResponseMessage response = await client.PostAsync(url,content);
response.EnsureSuccessStatusCode();
string body = await response.Content.ReadAsStringAsync();
return body;
}
catch (Exception e)
{
return e.ToString();
}
}https://stackoverflow.com/questions/13318102/struggling-trying-to-get-cookie-out-of-response-with-httpclient-in-net-4-53.
3.How do I set a cookie on HttpClient's HttpRequestMessage
Here's how you could set a custom cookie value for the request:
var baseAddress = new Uri("http://example.com");
var cookieContainer = new CookieContainer();
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer })
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
var content = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>("foo", "bar"),
new KeyValuePair<string, string>("baz", "bazinga"),
});
cookieContainer.Add(baseAddress, new Cookie("CookieName", "cookie_value"));
var result = await client.PostAsync("/test", content);
result.EnsureSuccessStatusCode();
}https://stackoverflow.com/questions/12373738/how-do-i-set-a-cookie-on-httpclients-httprequestmessage
4.HttpClient传递Cookie
使用CookieContainer
var baseAddress = new Uri("http://example.com");
var cookieContainer = new CookieContainer();
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer })
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
//此处是顺便演示如何传Post参数
var content = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>("foo", "bar"),
new KeyValuePair<string, string>("baz", "bazinga"),
});
cookieContainer.Add(baseAddress, new Cookie("CookieName", "cookie_value"));
var result = client.PostAsync("/test", content).Result;
result.EnsureSuccessStatusCode();
}http://www.cnblogs.com/walkerwang/p/3258612.html
5.HttpClient Request like browser
private static readonly HttpClient _HttpClient = new HttpClient();
private static async Task<string> GetResponse(string url)
{
using (var request = new HttpRequestMessage(HttpMethod.Get, new Uri(url)))
{
request.Headers.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml");
request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0");
request.Headers.TryAddWithoutValidation("Accept-Charset", "ISO-8859-1");
using (var response = await _HttpClient.SendAsync(request).ConfigureAwait(false))
{
response.EnsureSuccessStatusCode();
using (var responseStream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
using (var decompressedStream = new GZipStream(responseStream, CompressionMode.Decompress))
using (var streamReader = new StreamReader(decompressedStream))
{
return await streamReader.ReadToEndAsync().ConfigureAwait(false);
}
}
}
}call such like:
var response = await GetResponse("http://www.livescore.com/").ConfigureAwait(false);
// or
var response = GetResponse("http://www.livescore.com/").Result;https://stackoverflow.com/questions/15026953/httpclient-request-like-browser.
other：
Google Scholar Proxy based on HttpClient
谷歌学术搜索并不提供API，简单使用Http请求很容易跳转到人工验证页面，尤其是采用新的验证方式让代理或者调用谷歌学术的搜索结果越来越麻烦。
SCholarProxy在同一个Ip地址下，利用Cookie模拟多台客户端向服务器请求数据，减少服务器判断为机器提交数据的概率。
Web基于Aps.net的HttpModule,截获网络请求，然后分析请求类型，转发Html请求，图片类型请求转化为本地请求。对于Html请求可以实现对代码的修改。
https://github.com/chen198328/ScholarProxy/blob/master/Library/HttpClientHelper.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Net.Http;
using System.Net;
namespace GoogleProxy
{
public class HttpClientHelper
{
public NCookieManager cookieManager { set; get; }
public HttpClientHelper()
{
cookieManager = new NCookieManager();
}
public string GetHtml(string url)
{
NCookie ncookie = cookieManager.Get();
ncookie.Refere = url;
CookieContainer cookiecontainer = new CookieContainer();
foreach (var nc in ncookie.Cookies)
{
cookiecontainer.Add(new Cookie(nc.Key, nc.Value, "/", "scholar.google.com"));
}
using (HttpClientHandler handler = new HttpClientHandler() { CookieContainer = cookiecontainer, UseCookies = true })
using (HttpClient httpclient = new HttpClient(handler))
{
httpclient.DefaultRequestHeaders.Add("Accept", "text/html, application/xhtml+xml, */*");
httpclient.DefaultRequestHeaders.Add("Accept-Language", "zh-Hans-CN,zh-Hans;q=0.8,en-US;q=0.5,en;q=0.3");
httpclient.DefaultRequestHeaders.Add("User-agent", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)");
httpclient.DefaultRequestHeaders.Add("Host", "scholar.google.com");
httpclient.DefaultRequestHeaders.Add("DNT", "1");
httpclient.DefaultRequestHeaders.Add("Connection", "Keep-Alive");
return httpclient.GetStringAsync(url).Result;
}
}
}
}
set a cookie on HttpClient's HttpRequestMessage
there are some situations where you want to set the cookie header manually. Normally if you set a "Cookie" header it is ignored, but that's because HttpClientHandler defaults to using its CookieContainer property for cookies. If you disable that then by setting UseCookies to false you can set cookie headers manually and they will appear in the request, e.g.
var baseAddress = new Uri("http://example.com");
using (var handler = new HttpClientHandler { UseCookies = false })
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
var message = new HttpRequestMessage(HttpMethod.Get, "/test");
message.Headers.Add("Cookie", "cookie1=value1; cookie2=value2");
var result = await client.SendAsync(message);
result.EnsureSuccessStatusCode();
}
var uriBuilder = new UriBuilder("test.php", "test");
var httpClient = new HttpClient();
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, uriBuilder.ToString());
httpRequestMessage.Headers.Add("Host", "test.com");
httpRequestMessage.Headers.Add("Connection", "keep-alive");
// httpRequestMessage.Headers.Add("Content-Length", "138");
httpRequestMessage.Headers.Add("Pragma", "no-cache");
httpRequestMessage.Headers.Add("Cache-Control", "no-cache");
httpRequestMessage.Headers.Add("Origin", "test.com");
httpRequestMessage.Headers.Add("Upgrade-Insecure-Requests", "1");
// httpRequestMessage.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
httpRequestMessage.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36");
httpRequestMessage.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
httpRequestMessage.Headers.Add("Referer", "http://www.translationdirectory.com/");
httpRequestMessage.Headers.Add("Accept-Encoding", "gzip, deflate");
httpRequestMessage.Headers.Add("Accept-Language", "en-GB,en-US;q=0.9,en;q=0.8");
httpRequestMessage.Headers.Add("Cookie", "__utmc=266643403; __utmz=266643403.1537352460.3.3.utmccn=(referral)|utmcsr=google.co.uk|utmcct=/|utmcmd=referral; __utma=266643403.817561753.1532012719.1537357162.1537361568.5; __utmb=266643403; __atuvc=0%7C34%2C0%7C35%2C0%7C36%2C0%7C37%2C48%7C38; __atuvs=5ba2469fbb02458f002");
var httpResponseMessage = httpClient.SendAsync(httpRequestMessage).Result;
var httpContent = httpResponseMessage.Content;
string result = httpResponseMessage.Content.ReadAsStringAsync().Result;
HttpClient自动保存Cookie，只要是同一个HttpClient请求就可以实现保持登录状态。以下测试代码
public static void Login()
{
HttpClient httpClient = new HttpClient();//创建请求
//httpClient.MaxResponseContentBufferSize = 256000;
httpClient.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134");
String url = "http://my.gfan.com/doLogin";
List<KeyValuePair<String, String>> paramList = new List<KeyValuePair<String, String>>();//构建表单数据
paramList.Add(new KeyValuePair<string, string>("loginName", "8"));
paramList.Add(new KeyValuePair<string, string>("password", "do"));
paramList.Add(new KeyValuePair<string, string>("autoLogin", "0"));
paramList.Add(new KeyValuePair<string, string>("referer", "http://my.gfan.com/login"));
HttpResponseMessage response = httpClient.PostAsync(new Uri(url), new FormUrlEncodedContent(paramList)).Result;
var cookie = response.Headers.GetValues("Set-Cookie");
String result = response.Content.ReadAsStringAsync().Result;
Console.WriteLine(result);
//Get请求访问账号信息页面
string url_2 = "http://my.gfan.com/account";
HttpResponseMessage responseMessage = httpClient.GetAsync(new Uri(url_2)).Result;//使用之前的HttpClient
String result_2 =responseMessage.Content.ReadAsStringAsync().Result;
Console.WriteLine(result_2);
}
Use Cookie Authentication With Web API And HttpClient
https://forums.asp.net/t/2136882.aspx?Using+HTttpClient+to+Login+and+Post