These are my fieldnotes from experimenting with the ForwardedHeaders middleware on ASP.NET Core 1.1 and 2.0 - sparked by having a production log full of Parameter count mismatch between X-Forwarded-For and X-Forwarded-Proto warnings.

I've set up a github repo and a simple Postman example here.

Background

1.Run the Application Directly on Kestrel Server

 client  <-> backend

 

2.Configure ASP.NET Core to work with proxy servers

 client <-> proxy <-> backend

3.One or several reverse proxies

client <-> proxy <-> proxy <-> backend

.Net Core will consider the last proxy as ‘the client’ making the requests

If we ‘do nothing’ .Net Core will consider the last proxy as ‘the client’ making the requests. Framework components or custom code using the HttpContext to get information on the original client will get the wrong information.

We have our user, who connects to Nginx, which sends the request to ASP.NET application. This means, our app won’t see the users IP Address, but the one nginx is running at.

One example where this causes problems  is when we want to log the client IP-address.

One of my current projects is hosted in docker environment (more on that topic coming!). The setup requires me to use an Nginx reverse proxy. Additionally, for performance reasons, I also use CloudFlare as a CDN. This all means, before the user gets to my application, he has to go through at least two proxy servers. That’s an issue when you want to know your users IP address.

X-Forwarded-For

Fortunately, there is a concept of X-Forwarded-For HTTP header, which is used to carry information about all the IP addresses on the way to your application. In the case of my application, it will contain users address and CloudFlare server address.

 

The http headers X-Forwarded-* are here to address this exact problem and .Net Core has great support with the ForwardedHeaders middleware. Each reverse proxy will add to the X-Forwarded headers and the middleware will change the HttpContext accordingly.

Traditionally, an HTTP reverse proxy uses non-standard headers to inform the upstream server about the user’s IP address and other request properties:

X-Real-IP: 12.34.56.78
X-Forwarded-Host: example.com
X-Forwarded-Proto: https

NGINX even provides a $proxy_add_x_forwarded_for variable to automatically append $remote_addr to any incoming X-Forwarded-For headers.

The X-Forwarded-* headers are set by proxies or load-balancers in between the client browser and your web server. Each proxy usually adds their incoming IP, protocol and host to the existing set of header values - resulting in a list of header values reaching the web host. And most commonly - allowing the web host to know the IP of the end-user.

X-Forwarded-For - IP address (x.x.x.x)
X-Forwarded-Proto - Protocol (http, https)
X-Forwarded-Host - Host (something.com:54321)

All headers use comma-separated value sets and each hop adds to the end of the list: client, proxy1, proxy2, proxy3 etc.

X-Forwarded-For: client IP Address, proxy1 IP Address, proxy2 IP Address
X-Forwarded-Proto:https,https,http

for example:

nginx log config

log_format proxy1 '"[proxy1]" $remote_addr $proxy_add_x_forwarded_for "$request" $status';
log_format proxy2 '"[proxy2]" $remote_addr $proxy_add_x_forwarded_for "$request" $status';
log_format proxy3 '"[proxy3]" $remote_addr $proxy_add_x_forwarded_for "$request" $status';

output:

"[proxy1]" 36.157.229.110 36.157.229.110 "GET /hello HTTP/1.1" 200
"[proxy2]" 127.0.0.1 36.157.229.110, 127.0.0.1 "GET /hello HTTP/1.0" 200
"[proxy3]" 127.0.0.1 36.157.229.110, 127.0.0.1, 127.0.0.1 "GET /hello HTTP/1.0" 200

Solution

There are few ways to get hold of real users address.

Solution 1: Manually parsing HTTP headers.

Solution 2:Use ForwardedHeaders middleware

 

Solution 1: Manually parsing HTTP headers.

1.Run the Application Directly on Kestrel Server

IPAddress remoteIpAddress = Request.HttpContext.Connection.RemoteIpAddress;

2.

 

3.One or several reverse proxies

 var headers = context.Request.Headers;
        if (headers.ContainsKey("X-Forwarded-For"))
        {
            context.Connection.RemoteIpAddress=IPAddress.Parse(headers["X-Forwarded-For"].ToString().Split(',', StringSplitOptions.RemoveEmptyEntries)[0]);
        }

Solution 2:ForwardedHeaders middleware

Fortunately, it turns out, there is a middleware, part of the ASP.NET Core repository called BasicMiddleware. One of the things it does is overwriting Request fields using Forwarded headers (the same concept applies to HTTP Scheme and ports used in the request). 

source code from github:https://github.com/aspnet/BasicMiddleware/tree/master/src/Microsoft.AspNetCore.HttpOverrides

The ForwardedHeaders middleware that comes with ASP.NET Core reads the incoming X-Forwarded-* headers and assigns the values to properties on the http context making them available throughout the application:

ForwardedHeadersDefaults.cs

url:https://github.com/aspnet/BasicMiddleware/blob/master/src/Microsoft.AspNetCore.HttpOverrides/ForwardedHeadersDefaults.cs

namespace Microsoft.AspNetCore.HttpOverrides
{
    /// <summary>
    /// Default values related to <see cref="ForwardedHeadersMiddleware"/> middleware
    /// </summary>
    /// <seealso cref="Microsoft.AspNetCore.Builder.ForwardedHeadersOptions"/>
    public static class ForwardedHeadersDefaults
    {
        /// <summary>
        /// X-Forwarded-For
        /// </summary>
        public static string XForwardedForHeaderName { get; } = "X-Forwarded-For";

        /// <summary>
        /// X-Forwarded-Host
        /// </summary>
        public static string XForwardedHostHeaderName { get; } = "X-Forwarded-Host";

        /// <summary>
        /// X-Forwarded-Proto
        /// </summary>
        public static string XForwardedProtoHeaderName { get; } = "X-Forwarded-Proto";

        /// <summary>
        /// X-Original-For
        /// </summary>
        public static string XOriginalForHeaderName { get; } = "X-Original-For";

        /// <summary>
        /// X-Original-Host
        /// </summary>
        public static string XOriginalHostHeaderName { get; } = "X-Original-Host";

        /// <summary>
        /// X-Original-Proto
        /// </summary>
        public static string XOriginalProtoHeaderName { get; } = "X-Original-Proto";
    }
}

Mapping
X-Forwarded-For - HttpContext.Connection.RemoteIpAddress & HttpContext.Connection.RemotePort
X-Forwarded-Proto - HttpContext.Request.Scheme
X-Forwarded-Host - HttpContext.Request.Host

The minimal ASP.NET Core MVC application does not have ForwardedHeaders enabled by default, but if you're using the IISIntegration middleware it is enabled and set to forward the IP and the protocol. For reference, the default ForwardedHeadersOptions can be found here.

The most notable options for the ForwardedHeaders middleware are:

ForwardedHeaders - Specify which headers should be forwarded (assigned according to the mapping table above)
RequireHeaderSymmetry - Set to true if the mapping should be aborted and a warning logged when the headers are asymmetric (more about symmetry in the examples below).
ForwardLimit - Max number of entries per header that should be read - going from right to left.
KnownProxies - Addresses of known proxies to accept headers from. These will be skipped when performing the mapping.

In ASP.NET Core 1.1, the default value of RequireHeaderSymmetry is true. In 2.0 this has been changed to false.

These properties and their effects are not very well documented. The rest of this post will dive into a couple of examples to clearify their use.

Examples

I'm using a stripped-down ASP.NET core 2.0 project and sending requests through Postman. The code is posted here along with a Postman collection.

Basic scenario
The simplest scenario is just forwarding the protocol and IP, and having a single proxy. We can set up the middleware like this:

app.UseForwardedHeaders(new ForwardedHeadersOptions
{
  ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto,
  RequireHeaderSymmetry = true
});

To test the headers I'll use Postman to act as the proxy, first sending off a request without any headers. The result is:

{
    "remoteIpAddress": "::1",
    "remotePort": 5153,
    "requestHost": "localhost:2395",
    "requestScheme": "http"
}

And when setting X-Forwarded-For header to 80.80.80.80:1337 and X-Forwarded-Proto to https, I get:

{
    "remoteIpAddress": "80.80.80.80",
    "remotePort": 1337,
    "requestHost": "localhost:2395",
    "requestScheme": "https"
}

So far so good, but what happens if the proxy did not set the X-Forwarded-Proto header? Then the other header is also ignored:

{
    "remoteIpAddress": "::1",
    "remotePort": 5153,
    "requestHost": "localhost:2395",
    "requestScheme": "http"
}

And a warning has been logged:

warn: Microsoft.AspNetCore.HttpOverrides.ForwardedHeadersMiddleware[1]. Parameter count mismatch between X-Forwarded-For and X-Forwarded-Proto.

In other words: when RequireHeaderSymmetry is true, the proxy needs to apply all the expected headers, otherwise all headers will be ignored.

Advanced scenario
Now we assume three proxies, setting up our request in Postman like this (with Postman acting as the third proxy):

Postman setup

And changing the middleware configuration to:

app.UseForwardedHeaders(new ForwardedHeadersOptions
{
  ForwardedHeaders = 
    ForwardedHeaders.XForwardedFor 
    | ForwardedHeaders.XForwardedProto
    | ForwardedHeaders.XForwardedHost,
  ForwardLimit = 2,
  KnownProxies = { 
    IPAddress.Parse("70.70.70.70"), 
    IPAddress.Parse("80.80.80.80") 
  },
  RequireHeaderSymmetry = true
});

What is the expected result? This is what we get:

{
    "remoteIpAddress": "70.70.70.70",
    "remotePort": 1337,
    "requestHost": "firstproxy.com",
    "requestScheme": "https"
}

The first proxy is returned even though I have defined this as a known proxy indicating it should be skipped. The reason is that we have only allowed two forwards, and as the middleware starts from right to left when looking for values it stops at the first proxy.

Setting ForwardLimit to 3 resolves the issue:

{
    "remoteIpAddress": "60.60.60.60",
    "remotePort": 1336,
    "requestHost": "enduser.com",
    "requestScheme": "http"
}

Awesome! So the last thing worth mentioning is that if all the expected headers are not set by all proxies you'll get assymetry and the headers are ignored. Example request:

Asymmetric headers

Outputs:

{
    "remoteIpAddress": "::1",
    "remotePort": 5437,
    "requestHost": "localhost:2395",
    "requestScheme": "http"
}

And logs Parameter count mismatch between X-Forwarded-For and X-Forwarded-Proto.

Real Example

app.UseForwardedHeaders(new ForwardedHeadersOptions
{
    ForwardedHeaders = ForwardedHeaders.All,
    RequireHeaderSymmetry = false,
    ForwardLimit = null,
    KnownProxies = { IPAddress.Parse("162.158.202.131"), IPAddress.Parse("10.7.0.2") },
});

The first address in KnownProxies property is the CloudFlare server, second, my reverse proxy.

After that, you should get a real client IP in HttpContext.Connection.RemoteIpAddress

Conclusion

You need to know your infrastructure very well to be able to set up the ForwardedHeaders middleware correctly - but when it works you should be able to retrieve your end-user's IP.

Seeing header mismatch warnings in the log means that at least one proxy is not setting the expected header correctly. If you are unable to correct your proxy you can turn this off by setting RequireHeaderSymmetry to false. In ASP.NET Core 2 the default has been changed from true to false.