These are my fieldnotes from experimenting with the ForwardedHeaders middleware on ASP.NET Core 1.1 and 2.0 - sparked by having a production log full of Parameter count mismatch between X-Forwarded-For and X-Forwarded-Proto warnings.

I've set up a github repo and a simple Postman example here.

Background

1.Run the Application Directly on Kestrel Server

 client  <-> backend

 

2.Configure ASP.NET Core to work with proxy servers

 client <-> proxy <-> backend

There is only one proxy between the app and the source of the requests.

3.One or several reverse proxies

client <-> proxy <-> proxy <-> backend

4. cdn

client <-> CloudFlare proxy <->nginx proxy <-> backend

 

.Net Core will consider the last proxy as ‘the client’ making the requests

If we ‘do nothing’ .Net Core will consider the last proxy as ‘the client’ making the requests. Framework components or custom code using the HttpContext to get information on the original client will get the wrong information.

This means the client IP of HttpContext.Connection.RemoteIpAddress is the IP address of the proxy servers.

We have our user, who connects to Nginx, which sends the request to ASP.NET application. This means, our app won’t see the users IP Address, but the one nginx is running at.

One example where this causes problems  is when we want to log the client IP-address.

One of my current projects is hosted in docker environment (more on that topic coming!). The setup requires me to use an Nginx reverse proxy. Additionally, for performance reasons, I also use CloudFlare as a CDN. This all means, before the user gets to my application, he has to go through at least two proxy servers. That’s an issue when you want to know your users IP address.

X-Forwarded-For

Fortunately, there is a concept of X-Forwarded-For HTTP header, which is used to carry information about all the IP addresses on the way to your application. In the case of my application, it will contain users address and CloudFlare server address.

 

The http headers X-Forwarded-* are here to address this exact problem and .Net Core has great support with the ForwardedHeaders middleware. Each reverse proxy will add to the X-Forwarded headers and the middleware will change the HttpContext accordingly.

Traditionally, an HTTP reverse proxy uses non-standard headers to inform the upstream server about the user’s IP address and other request properties:

X-Real-IP: 12.34.56.78
X-Forwarded-Host: example.com
X-Forwarded-Proto: https

NGINX even provides a $proxy_add_x_forwarded_for variable to automatically append $remote_addr to any incoming X-Forwarded-For headers.

The X-Forwarded-* headers are set by proxies or load-balancers in between the client browser and your web server. Each proxy usually adds their incoming IP, protocol and host to the existing set of header values - resulting in a list of header values reaching the web host. And most commonly - allowing the web host to know the IP of the end-user.

X-Forwarded-For - IP address (x.x.x.x)
X-Forwarded-Proto - Protocol (http, https)
X-Forwarded-Host - Host (something.com:54321)

https://kontext.tech/article/1291/retrieve-client-ip-address-of-azure-container-apps

All headers use comma-separated value sets and each hop adds to the end of the list: client, proxy1, proxy2, proxy3 etc.

X-Forwarded-For: client IP Address, proxy1 IP Address, proxy2 IP Address
X-Forwarded-Proto:https,https,http

for example:

nginx log config

log_format proxy1 '"[proxy1]" $remote_addr $proxy_add_x_forwarded_for "$request" $status';
log_format proxy2 '"[proxy2]" $remote_addr $proxy_add_x_forwarded_for "$request" $status';
log_format proxy3 '"[proxy3]" $remote_addr $proxy_add_x_forwarded_for "$request" $status';

output:

"[proxy1]" 36.157.229.110 36.157.229.110 "GET /hello HTTP/1.1" 200
"[proxy2]" 127.0.0.1 36.157.229.110, 127.0.0.1 "GET /hello HTTP/1.0" 200
"[proxy3]" 127.0.0.1 36.157.229.110, 127.0.0.1, 127.0.0.1 "GET /hello HTTP/1.0" 200

Solution

There are few ways to get hold of real users address.

Solution 1: Manually parsing HTTP headers.

Solution 2:Use ForwardedHeaders middleware

 

Solution 1: Manually parsing HTTP headers.

1.Run the Application Directly on Kestrel Server

IPAddress remoteIpAddress = Request.HttpContext.Connection.RemoteIpAddress;

2.

 

3.One or several reverse proxies

 var headers = context.Request.Headers;
        if (headers.ContainsKey("X-Forwarded-For"))
        {
            context.Connection.RemoteIpAddress=IPAddress.Parse(headers["X-Forwarded-For"].ToString().Split(',', StringSplitOptions.RemoveEmptyEntries)[0]);
        }

Solution 2:ForwardedHeaders middleware

Fortunately, it turns out, there is a middleware, part of the ASP.NET Core repository called BasicMiddleware. One of the things it does is overwriting Request fields using Forwarded headers (the same concept applies to HTTP Scheme and ports used in the request). 

source code from github:https://github.com/aspnet/BasicMiddleware/tree/master/src/Microsoft.AspNetCore.HttpOverrides

The ForwardedHeaders middleware that comes with ASP.NET Core reads the incoming X-Forwarded-* headers and assigns the values to properties on the http context making them available throughout the application:

ForwardedHeadersDefaults.cs

url:https://github.com/aspnet/BasicMiddleware/blob/master/src/Microsoft.AspNetCore.HttpOverrides/ForwardedHeadersDefaults.cs

// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System.Net;
using Microsoft.AspNetCore.HttpOverrides;

namespace Microsoft.AspNetCore.Builder;

/// <summary>
/// Options for <see cref="ForwardedHeadersMiddleware"/>
/// </summary>
public class ForwardedHeadersOptions
{
    /// <summary>
    /// Gets or sets the header used to retrieve the originating client IP. Defaults to the value specified by
    /// <see cref="ForwardedHeadersDefaults.XForwardedForHeaderName"/>.
    /// </summary>
    public string ForwardedForHeaderName { get; set; } = ForwardedHeadersDefaults.XForwardedForHeaderName;

    /// <summary>
    /// Gets or sets the header used to retrieve the original value of the Host header field.
    /// Defaults to the value specified by <see cref="ForwardedHeadersDefaults.XForwardedHostHeaderName"/>
    /// </summary>
    public string ForwardedHostHeaderName { get; set; } = ForwardedHeadersDefaults.XForwardedHostHeaderName;

    /// <summary>
    /// Gets or sets the header used to retrieve the value for the originating scheme (HTTP/HTTPS).
    /// Defaults to the value specified by <see cref="ForwardedHeadersDefaults.XForwardedProtoHeaderName"/>
    /// </summary>
    public string ForwardedProtoHeaderName { get; set; } = ForwardedHeadersDefaults.XForwardedProtoHeaderName;

    /// <summary>
    /// Gets or sets the header used to store the original value of client IP before applying forwarded headers.
    /// Defaults to the value specified by <see cref="ForwardedHeadersDefaults.XOriginalForHeaderName"/>
    /// </summary>
    /// <seealso cref="ForwardedHeadersDefaults"/>
    public string OriginalForHeaderName { get; set; } = ForwardedHeadersDefaults.XOriginalForHeaderName;

    /// <summary>
    /// Gets or sets the header used to store the original value of the Host header field before applying forwarded headers.
    /// Defaults to the value specified by <see cref="ForwardedHeadersDefaults.XOriginalHostHeaderName"/>
    /// </summary>
    /// <seealso cref="ForwardedHeadersDefaults"/>
    public string OriginalHostHeaderName { get; set; } = ForwardedHeadersDefaults.XOriginalHostHeaderName;

    /// <summary>
    /// Gets or sets the header used to store the original scheme (HTTP/HTTPS) before applying forwarded headers.
    /// Defaults to the value specified by <see cref="ForwardedHeadersDefaults.XOriginalProtoHeaderName"/>
    /// </summary>
    /// <seealso cref="ForwardedHeadersDefaults"/>
    public string OriginalProtoHeaderName { get; set; } = ForwardedHeadersDefaults.XOriginalProtoHeaderName;

    /// <summary>
    /// Identifies which forwarders should be processed.
    /// </summary>
    public ForwardedHeaders ForwardedHeaders { get; set; }

    /// <summary>
    /// Limits the number of entries in the headers that will be processed. The default value is 1.
    /// Set to null to disable the limit, but this should only be done if
    /// KnownProxies or KnownNetworks are configured.
    /// </summary>
    public int? ForwardLimit { get; set; } = 1;

    /// <summary>
    /// Addresses of known proxies to accept forwarded headers from.
    /// </summary>
    public IList<IPAddress> KnownProxies { get; } = new List<IPAddress>() { IPAddress.IPv6Loopback };

    /// <summary>
    /// Address ranges of known proxies to accept forwarded headers from.
    /// </summary>
    public IList<IPNetwork> KnownNetworks { get; } = new List<IPNetwork>() { new IPNetwork(IPAddress.Loopback, 8) };

    /// <summary>
    /// The allowed values from x-forwarded-host. If the list is empty then all hosts are allowed.
    /// Failing to restrict this these values may allow an attacker to spoof links generated by your service.
    /// </summary>
    /// <remarks>
    /// <list type="bullet">
    /// <item><description>Port numbers must be excluded.</description></item>
    /// <item><description>A top level wildcard "*" allows all non-empty hosts.</description></item>
    /// <item><description>Subdomain wildcards are permitted. E.g. "*.example.com" matches subdomains like foo.example.com,
    ///    but not the parent domain example.com.</description></item>
    /// <item><description>Unicode host names are allowed but will be converted to punycode for matching.</description></item>
    /// <item><description>IPv6 addresses must include their bounding brackets and be in their normalized form.</description></item>
    /// </list>
    /// </remarks>
    public IList<string> AllowedHosts { get; set; } = new List<string>();

    /// <summary>
    /// Require the number of header values to be in sync between the different headers being processed.
    /// The default is 'false'.
    /// </summary>
    public bool RequireHeaderSymmetry { get; set; }
}

 

HttpContext.Connection.RemoteIpAddress: Set using the X-Forwarded-For header value. 
HttpContext.Request.Scheme: Set using the X-Forwarded-Proto header value.
HttpContext.Request.Host: Set using the X-Forwarded-Host header value.

 

Mapping


X-Forwarded-For - HttpContext.Connection.RemoteIpAddress & HttpContext.Connection.RemotePort
X-Forwarded-Proto - HttpContext.Request.Scheme
X-Forwarded-Host - HttpContext.Request.Host

The middleware updates:

The minimal ASP.NET Core MVC application does not have ForwardedHeaders enabled by default, but if you're using the IISIntegration middleware it is enabled and set to forward the IP and the protocol. For reference, the default ForwardedHeadersOptions can be found here.

The most notable options for the ForwardedHeaders middleware are:

ForwardedHeaders - Specify which headers should be forwarded (assigned according to the mapping table above)
RequireHeaderSymmetry - Set to true if the mapping should be aborted and a warning logged when the headers are asymmetric (more about symmetry in the examples below).
ForwardLimit - Max number of entries per header that should be read - going from right to left.
KnownProxies - Addresses of known proxies to accept headers from. These will be skipped when performing the mapping.

In ASP.NET Core 1.1, the default value of RequireHeaderSymmetry is true. In 2.0 this has been changed to false.

These properties and their effects are not very well documented. The rest of this post will dive into a couple of examples to clearify their use.

Examples

I'm using a stripped-down ASP.NET core 6.0 project and sending requests through Postman. The code is posted here along with a Postman collection.

Basic scenario


The simplest scenario is just forwarding the protocol and IP, and having a single proxy. We can set up the middleware like this:

see:https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-6.0

using Microsoft.AspNetCore.HttpOverrides;

...

app.UseForwardedHeaders(new ForwardedHeadersOptions
{
    ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto
});

app.UseAuthentication();

To test the headers I'll use Postman to act as the proxy, first sending off a request without any headers. The result is:

{
    "remoteIpAddress": "::1",
    "remotePort": 5153,
    "requestHost": "localhost:2395",
    "requestScheme": "http"
}

And when setting X-Forwarded-For header to 80.80.80.80:1337 and X-Forwarded-Proto to https, I get:

{
    "remoteIpAddress": "80.80.80.80",
    "remotePort": 1337,
    "requestHost": "localhost:2395",
    "requestScheme": "https"
}

So far so good, but what happens if the proxy did not set the X-Forwarded-Proto header? Then the other header is also ignored:

{
    "remoteIpAddress": "::1",
    "remotePort": 5153,
    "requestHost": "localhost:2395",
    "requestScheme": "http"
}

And a warning has been logged:

warn: Microsoft.AspNetCore.HttpOverrides.ForwardedHeadersMiddleware[1]. Parameter count mismatch between X-Forwarded-For and X-Forwarded-Proto.

In other words: when RequireHeaderSymmetry is true, the proxy needs to apply all the expected headers, otherwise all headers will be ignored.

Advanced scenario
Now we assume three proxies, setting up our request in Postman like this (with Postman acting as the third proxy):

Postman setup

And changing the middleware configuration to:

app.UseForwardedHeaders(new ForwardedHeadersOptions
{
  ForwardedHeaders = 
    ForwardedHeaders.XForwardedFor 
    | ForwardedHeaders.XForwardedProto
    | ForwardedHeaders.XForwardedHost,
  ForwardLimit = 2,
  KnownProxies = { 
    IPAddress.Parse("70.70.70.70"), 
    IPAddress.Parse("80.80.80.80") 
  },
  RequireHeaderSymmetry = true
});

What is the expected result? This is what we get:

{
    "remoteIpAddress": "70.70.70.70",
    "remotePort": 1337,
    "requestHost": "firstproxy.com",
    "requestScheme": "https"
}

The first proxy is returned even though I have defined this as a known proxy indicating it should be skipped. The reason is that we have only allowed two forwards, and as the middleware starts from right to left when looking for values it stops at the first proxy.

Setting ForwardLimit to 3 resolves the issue:

{
    "remoteIpAddress": "60.60.60.60",
    "remotePort": 1336,
    "requestHost": "enduser.com",
    "requestScheme": "http"
}

Awesome! So the last thing worth mentioning is that if all the expected headers are not set by all proxies you'll get assymetry and the headers are ignored. Example request:

Asymmetric headers

Outputs:

{
    "remoteIpAddress": "::1",
    "remotePort": 5437,
    "requestHost": "localhost:2395",
    "requestScheme": "http"
}

And logs Parameter count mismatch between X-Forwarded-For and X-Forwarded-Proto.

 

4. cdn (CloudFlare)

CloudFlare Real Example

app.UseForwardedHeaders(new ForwardedHeadersOptions
{
    ForwardedHeaders = ForwardedHeaders.All,
    RequireHeaderSymmetry = false,
    ForwardLimit = null,
    KnownProxies = { IPAddress.Parse("162.158.202.131"), IPAddress.Parse("10.7.0.2") },
});

The first address in KnownProxies property is the CloudFlare server, second, my reverse proxy.

After that, you should get a real client IP in HttpContext.Connection.RemoteIpAddress

Conclusion

You need to know your infrastructure very well to be able to set up the ForwardedHeaders middleware correctly - but when it works you should be able to retrieve your end-user's IP.

Seeing header mismatch warnings in the log means that at least one proxy is not setting the expected header correctly. If you are unable to correct your proxy you can turn this off by setting RequireHeaderSymmetry to false. In ASP.NET Core 2 the default has been changed from true to false.