Extending ASP.NET Core Response Compression With Support for Brotli

The amount of transferred data matters. On one hand, it often contributes to the cost of running a service and on the other, a lot of clients doesn't have as fast of connections as we would like to believe. This is why response compression is one of key performance mechanisms in the web world.

There is a number of compression schemas (more or less popular) out there, so clients advertise the supported ones with Accept-Encoding header.

The above screenshot shows the result of a request from Chrome to the simplest possible ASP.NET Core application.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.Run(async (context) =>
        {
            await context.Response.WriteAsync("-- Demo.AspNetCore.ResponseCompression.Brotli --");
        });
    }
}

As we can see the browser has advertised four different options of compressing the response but none has been used. This shouldn't be a surprise as ASP.NET Core is modular by its nature and leaves it up to us to pick the features we want. In order for compression to be supported, we need to add a proper middleware.

Enabling Response Compression

The support for response compression in ASP.NET Core is available through ResponseCompressionMiddleware from Microsoft.AspNetCore.ResponseCompression package. After referencing the package all that needs to be done is registering middleware and related services.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression();
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.UseResponseCompression()
            .Run(async (context) =>
            {
                if (!StringValues.IsNullOrEmpty(context.Request.Headers[HeaderNames.AcceptEncoding]))
                    context.Response.Headers.Append(HeaderNames.Vary, HeaderNames.AcceptEncoding);

                context.Response.ContentType = "text/plain";
                await context.Response.WriteAsync("-- Demo.AspNetCore.ResponseCompression.Brotli --");
            });
    }
}

One thing to remember is to set the Content-Type, as compression is enabled only for specific MIME types (there is also a separated setting for enabling compression over HTTPS). Additionally, I'm adding a 

header to the response so any cache along the way knows the response needs to be cached per compression type ( a future version of middleware will handle this for us).

The below screenshot shows the result of the same request we did in our previous example, after modifications.

Now the response has been compressed using gzip. Gzip compression is the only one supported by the middleware, which is "ok" in most cases as it has the widest support among clients. But the web world is constantly evolving and compression algorithms are no different. The latest-and-greatest seems to be Brotli, which can shrink data by an additional 20%-25%. It would be nice to use it in ASP.NET Core.

Extending Response Compression With Brotli

The ResponseCompressionMiddleware can be extended with additional compression algorithms by implementing the ICompressionProvider interface. The interface is pretty simple, it has two properties (providing information about encoding token and whether flushing is supported) and one method (which should create a stream with compression capabilities). The true challenge is the actual Brotli implementation. I've decided to use a .NET Core build of Brotli.NET. This is, in fact, a wrapper around the original implementation, so some cross-platform issues might appear and force recompilation. The wrapper exposes the original implementation through BrotliStream which makes it very easy to use in the context of the ICompressionProvider.

public class BrotliCompressionProvider : ICompressionProvider
{
    public string EncodingName => "br";

    public bool SupportsFlush => true;

    public Stream CreateStream(Stream outputStream)
    {
        return new BrotliStream(outputStream, CompressionMode.Compress);
    }
}

The custom provider needs to be added to the ResponseCompressionOptions.Providers collection as part of the services registration.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression(options =>
        {
            options.Providers.Add<BrotliCompressionProvider>();
        });
    }

    ...
}

Now the demo request can be done once again - it should show that Brotli is being used for compression.

Not Every Browser Supports Brotli

Let's take a quick look at compression support advertised by different browsers:

• IE11: Accept-Encoding: gzip, deflate
• Edge: Accept-Encoding: gzip, deflate
• Firefox: Accept-Encoding: gzip, deflate (HTTP), Accept-Encoding: gzip, deflate, br (HTTPS)
• Chrome: Accept-Encoding: gzip, deflate, sdch, br
• Opera: Accept-Encoding:gzip, deflate, sdch, br

So IE and Edge don't support Brotli at all and Firefox supports it only over HTTPS. Checking more detailed information at caniuse, we will learn that a couple more browsers don't support Brotli (but Edge already has it in preview, although it is rumored that the final support will be only over HTTPS). The overall support is about 57% which means that we want to keep gzip around as well. In order to do this, it needs to be added to the ResponseCompressionOptions.Providers collection too (the moment we start manually registering providers the default one is gone).

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression(options =>
        {
            options.Providers.Add<BrotliCompressionProvider>();
            options.Providers.Add<GzipCompressionProvider>();
        });
    }

    ...
}

If we test this code against various browsers we will see that the chosen compression always ends up being gzip. The reason for that is the way in which middleware chooses the provider. It takes the advertised compressions, sorts them by quality, if present, and chooses the first one for which a provider exists. As browsers generally don't provide any quality values (in other words, they will be equally happy to accept any of the supported ones) the gzip always wins because it is always first on the advertised list. Unfortunately, the middleware doesn't provide an option for defining a server-side preference for such cases. In order to work around this, I've decided to go about it in a hacky way. If the only way to control the provider selection is through quality values, they need to be adjusted before the response compression middleware kicks in. I've put together another middleware to do exactly that. The additional middleware would inspect the request Accept-Encoding header and, if there are no quality values provided, would adjust them.

public class ResponseCompressionQualityMiddleware
{
    private readonly RequestDelegate _next;
    private readonly IDictionary<string, double> _encodingQuality;

    public ResponseCompressionQualityMiddleware(RequestDelegate next, IDictionary<string, double> encodingQuality)
    {
        _next = next;
        _encodingQuality = encodingQuality;
    }

    public async Task Invoke(HttpContext context)
    {
        StringValues encodings = context.Request.Headers[HeaderNames.AcceptEncoding];
        IList<StringWithQualityHeaderValue> encodingsList;

        if (!StringValues.IsNullOrEmpty(encodings)
            && StringWithQualityHeaderValue.TryParseList(encodings, out encodingsList)
            && (encodingsList != null) && (encodingsList.Count > 0))
        {
            string[] encodingsWithQuality = new string[encodingsList.Count];

            for (int encodingIndex = 0; encodingIndex < encodingsList.Count; encodingIndex++)
            {
                // If there is any quality value provided don't change anything
                if (encodingsList[encodingIndex].Quality.HasValue)
                {
                    encodingsWithQuality = null;
                    break;
                }
                else
                {
                    string encodingValue = encodingsList[encodingIndex].Value;
                    encodingsWithQuality[encodingIndex] = (new StringWithQualityHeaderValue(encodingValue,
                        _encodingQuality.ContainsKey(encodingValue) ? _encodingQuality[encodingValue] : 0.1)).ToString();
                }

            }

            if (encodingsWithQuality != null)
                context.Request.Headers[HeaderNames.AcceptEncoding] = new StringValues(encodingsWithQuality);
        }

        await _next(context);
    }
}

This "adjusting" middleware needs to be registered before the response compression middleware and configured with tokens for which a preference is needed.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression(options =>
        {
            options.Providers.Add<BrotliCompressionProvider>();
            options.Providers.Add<GzipCompressionProvider>();
        });
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.UseMiddleware<ResponseCompressionQualityMiddleware>(new Dictionary<string, double>
            {
                { "br", 1.0 },
                { "gzip", 0.9 }
            })
            .UseResponseCompression()
            .Run(async (context) =>
            {
                if (!StringValues.IsNullOrEmpty(context.Request.Headers[HeaderNames.AcceptEncoding]))
                    context.Response.Headers.Append(HeaderNames.Vary, HeaderNames.AcceptEncoding);

                context.Response.ContentType = "text/plain";
                await context.Response.WriteAsync("-- Demo.AspNetCore.ResponseCompression.Brotli --");
            });
    }
}

Now the tests in different browsers will give different results. For example, in the case of Edge, the response will be compressed with gzip, but in the case of Chrome, it will be compressed with Brotli, which is the desired effect.