Popular blog tags

asp.net core 使用protobuf序列化数据(在一些性能要求很高的应用中)

Published

http://www.cnblogs.com/zfking/p/5841376.html 

 

asp.net core 使用protobuf

在一些性能要求很高的应用中,使用protocol buffer序列化,优于Json。而且protocol buffer向后兼容的能力比较好。

由于Asp.net core 采用了全新的MiddleWare方式,因此使用protobuf序列化,只需要使用Protobuf-net修饰需要序列化的对象,并在MVC初始化的时候增加相应的Formatter就可以了。

MVC Controller 的Action返回对象时,MVC回根据用户的Request Header里面的MIME选择对应的Formater来序列化返回对象( Serialize returned object)。MVC具有默认的Json Formater,这个可以不用管。

这里有一个直接可以运行的例子,具有Server和Client代码 https://github.com/damienbod/AspNetMvc6ProtobufFormatters

但是,这里面有一个很严重的问题。 看下面的例子。

例如我们需要序列化的对象时ApparatusType,服务端的定义(使用了EntityFramework)是这样的:

复制代码
using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;
using ProtoBuf;

namespace Hammergo.Data
{

    [ProtoContract]
    public partial class ApparatusType
    {
        public ApparatusType()
        {
            this.Apps = new List<App>();
        }

        [ProtoMember(1)]
        public System.Guid Id { get; set; }

        [ProtoMember(2)]
        [MaxLength(20)]
        public string TypeName { get; set; }
        [ProtoIgnore]
        public virtual ICollection<App> Apps { get; set; }
    }
}
复制代码

属于ProtoBuf 的三个修饰为

 [ProtoContract]
 [ProtoMember(1)]
 [ProtoMember(2)]
其他的不用管,在客户端定义是这样的
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
using System;
using System.Collections.Generic;
using ProtoBuf;
 
namespace Hammergo.Poco
{
 
    [ProtoContract]
    public class ApparatusType
    {
 
        [ProtoMember(1)]
        public virtual System.Guid Id { get; set; }
 
        [ProtoMember(2)]
        public virtual string TypeName { get; set; }
    }
}

这里使用了Virtual关键字,是为了生成POCO的代理类,以跟踪状态,没有这个要求可以不使用。

如果使用https://github.com/damienbod/AspNetMvc6ProtobufFormatters 的方案就会发现

如果ASP.NET 的action返回List<AppratusType>,在客户端使用

  var result = response.Content.ReadAsAsync<List<Hammergo.Poco.ApparatusType>>(new[] { new ProtoBufFormatter() }).Result;

就会抛出异常,ReadAsAsync ProtoBuf Formatter No MediaTypeFormatter is available to read

大意是没有 相应的MediaTypeFormatter来供ReadAsAsync来使用,

检查https://github.com/damienbod/AspNetMvc6ProtobufFormatters 的方案,发现它调用了https://github.com/WebApiContrib/WebApiContrib.Formatting.ProtoBuf里面的ProtoBufFormatter.cs ,这个里面有一个错误。

复制代码
using System;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Formatting;
using System.Net.Http.Headers;
using System.Reflection;
using System.Threading.Tasks;
using ProtoBuf;
using ProtoBuf.Meta;

namespace WebApiContrib.Formatting
{
    public class ProtoBufFormatter : MediaTypeFormatter
    {
        private static readonly MediaTypeHeaderValue mediaType = new MediaTypeHeaderValue("application/x-protobuf");
        private static Lazy<RuntimeTypeModel> model = new Lazy<RuntimeTypeModel>(CreateTypeModel);

        public static RuntimeTypeModel Model
        {
            get { return model.Value; }
        }

        public ProtoBufFormatter()
        {
            SupportedMediaTypes.Add(mediaType);
        }

        public static MediaTypeHeaderValue DefaultMediaType
        {
            get { return mediaType; }
        }

        public override bool CanReadType(Type type)
        {
            return CanReadTypeCore(type);
        }

        public override bool CanWriteType(Type type)
        {
            return CanReadTypeCore(type);
        }

        public override Task<object> ReadFromStreamAsync(Type type, Stream stream, HttpContent content, IFormatterLogger formatterLogger)
        {
            var tcs = new TaskCompletionSource<object>();

            try
            {
                object result = Model.Deserialize(stream, null, type);
                tcs.SetResult(result);
            }
            catch (Exception ex)
            {
                tcs.SetException(ex);
            }

            return tcs.Task;
        }

        public override Task WriteToStreamAsync(Type type, object value, Stream stream, HttpContent content, TransportContext transportContext)
        {
            var tcs = new TaskCompletionSource<object>();

            try
            {
                Model.Serialize(stream, value);
                tcs.SetResult(null);
            }
            catch (Exception ex)
            {
                tcs.SetException(ex);
            }

            return tcs.Task;
        }

        private static RuntimeTypeModel CreateTypeModel()
        {
            var typeModel = TypeModel.Create();
            typeModel.UseImplicitZeroDefaults = false;
            return typeModel;
        }

        private static bool CanReadTypeCore(Type type)
        {
            return type.GetCustomAttributes(typeof(ProtoContractAttribute)).Any();
        }
    }
}
复制代码
private static bool CanReadTypeCore(Type type)这个有问题,它只能识别有ProtoContract的类,没法识别其对应的IEnumerable<T>,修改这个方法就可以了。如下:
1
2
3
4
5
6
7
8
9
10
11
12
private static bool CanReadTypeCore(Type type)
{
    bool isCan = type.GetCustomAttributes(typeof(ProtoContractAttribute)).Any();
 
    if (!isCan && typeof(IEnumerable).IsAssignableFrom(type))
    {
        var temp = type.GetGenericArguments().FirstOrDefault();
        isCan = temp.GetCustomAttributes(typeof(ProtoContractAttribute)).Any();
    }
 
    return isCan;
}

  

下面我给出,关键的代码片段:

使用了一个辅助Library,结构如下图:

 

DateTimeOffsetSurrogate.cs的代码如下:

复制代码
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using ProtoBuf;

namespace ProtoBufHelper
{

    [ProtoContract]
    public class DateTimeOffsetSurrogate
    {
        [ProtoMember(1)]
        public long DateTimeTicks { get; set; }
        [ProtoMember(2)]
        public short OffsetMinutes { get; set; }

        public static implicit operator DateTimeOffsetSurrogate(DateTimeOffset value)
        {
            return new DateTimeOffsetSurrogate
            {
                DateTimeTicks = value.Ticks,
                OffsetMinutes = (short)value.Offset.TotalMinutes
            };
        }

        public static implicit operator DateTimeOffset(DateTimeOffsetSurrogate value)
        {
            return new DateTimeOffset(value.DateTimeTicks, TimeSpan.FromMinutes(value.OffsetMinutes));
        }
    }
}
复制代码

ProtoBufFormatter.cs 的代码如下:

复制代码
using System;
using System.Collections;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Formatting;
using System.Net.Http.Headers;
using System.Reflection;
using System.Threading.Tasks;
using ProtoBuf;
using ProtoBuf.Meta;

namespace ProtoBufHelper
{
    public class ProtoBufFormatter : MediaTypeFormatter
    {
        private static readonly MediaTypeHeaderValue mediaType = new MediaTypeHeaderValue("application/x-protobuf");
        private static Lazy<RuntimeTypeModel> model = new Lazy<RuntimeTypeModel>(CreateTypeModel);

        public static RuntimeTypeModel Model
        {
            get { return model.Value; }
        }

        public ProtoBufFormatter()
        {
            SupportedMediaTypes.Add(mediaType);
        }

        public static MediaTypeHeaderValue DefaultMediaType
        {
            get { return mediaType; }
        }

        public override bool CanReadType(Type type)
        {
            var temp = CanReadTypeCore(type);
            return temp;
        }

        public override bool CanWriteType(Type type)
        {
            return CanReadTypeCore(type);
        }

        public override Task<object> ReadFromStreamAsync(Type type, Stream stream, HttpContent content, IFormatterLogger formatterLogger)
        {
            var tcs = new TaskCompletionSource<object>();

            try
            {
                object result = Model.Deserialize(stream, null, type);
                tcs.SetResult(result);
            }
            catch (Exception ex)
            {
                tcs.SetException(ex);
            }

            return tcs.Task;
        }

        public override Task WriteToStreamAsync(Type type, object value, Stream stream, HttpContent content, TransportContext transportContext)
        {
            var tcs = new TaskCompletionSource<object>();

            try
            {
                Model.Serialize(stream, value);
                tcs.SetResult(null);
            }
            catch (Exception ex)
            {
                tcs.SetException(ex);
            }

            return tcs.Task;
        }

        private static RuntimeTypeModel CreateTypeModel()
        {
            var typeModel = TypeModel.Create();
            typeModel.UseImplicitZeroDefaults = false;
            typeModel.Add(typeof(DateTimeOffset), false).SetSurrogate(typeof(DateTimeOffsetSurrogate));
            return typeModel;
        }

        private static bool CanReadTypeCore(Type type)
        {
            bool isCan = type.GetCustomAttributes(typeof(ProtoContractAttribute)).Any();

            if (!isCan && typeof(IEnumerable).IsAssignableFrom(type))
            {
                var temp = type.GetGenericArguments().FirstOrDefault();
                isCan = temp.GetCustomAttributes(typeof(ProtoContractAttribute)).Any();
            }

            return isCan;
        }
    }
}
复制代码

 

这样就可以设置ASP.NET Core端的代码:

添加ProtobufInputFormatter.cs 和 ProtobufOutputFormatter.cs 代码分别如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
using System;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.Formatters;
using Microsoft.Net.Http.Headers;
using ProtoBuf.Meta;
using ProtoBufHelper;
 
namespace DamService
{
    public class ProtobufInputFormatter : InputFormatter
    {
        private static Lazy<RuntimeTypeModel> model = new Lazy<RuntimeTypeModel>(CreateTypeModel);
 
        public static RuntimeTypeModel Model
        {
            get { return model.Value; }
        }
 
        public override Task<InputFormatterResult> ReadRequestBodyAsync(InputFormatterContext context)
        {
            var type = context.ModelType;
            var request = context.HttpContext.Request;
            MediaTypeHeaderValue requestContentType = null;
            MediaTypeHeaderValue.TryParse(request.ContentType, out requestContentType);
 
 
            object result = Model.Deserialize(context.HttpContext.Request.Body, null, type);
            return InputFormatterResult.SuccessAsync(result);
        }
 
        public override bool CanRead(InputFormatterContext context)
        {
            return true;
        }
 
 
        private static RuntimeTypeModel CreateTypeModel()
        {
            var typeModel = TypeModel.Create();
            typeModel.UseImplicitZeroDefaults = false;
 
            typeModel.Add(typeof(DateTimeOffset), false).SetSurrogate(typeof(DateTimeOffsetSurrogate));
            return typeModel;
        }
    }
}

  

复制代码
using System;
using System.Text;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc.Formatters;
using Microsoft.Net.Http.Headers;
using ProtoBuf.Meta;
using ProtoBufHelper;

namespace DamService
{
    public class ProtobufOutputFormatter :  OutputFormatter
    {
        private static Lazy<RuntimeTypeModel> model = new Lazy<RuntimeTypeModel>(CreateTypeModel);

        public string ContentType { get; private set; }

        public static RuntimeTypeModel Model
        {
            get { return model.Value; }
        }

        public ProtobufOutputFormatter()
        {
            ContentType = "application/x-protobuf";
            SupportedMediaTypes.Add(MediaTypeHeaderValue.Parse("application/x-protobuf"));

            //SupportedEncodings.Add(Encoding.GetEncoding("utf-8"));
        }

        private static RuntimeTypeModel CreateTypeModel()
        {
            var typeModel = TypeModel.Create();
            typeModel.UseImplicitZeroDefaults = false;
            typeModel.Add(typeof(DateTimeOffset), false).SetSurrogate(typeof(DateTimeOffsetSurrogate));
            return typeModel;
        }

        public override Task WriteResponseBodyAsync(OutputFormatterWriteContext context)
        {
            var response = context.HttpContext.Response;
    
            Model.Serialize(response.Body, context.Object);
            return Task.FromResult(response);
        }
    }
}
复制代码

在Startup.cs中

public void ConfigureServices(IServiceCollection services) 方法中这样添加MVC中间件

services.AddMvc(options =>
{
options.InputFormatters.Add(new ProtobufInputFormatter());
options.OutputFormatters.Add(new ProtobufOutputFormatter());
options.FormatterMappings.SetMediaTypeMappingForFormat("protobuf", MediaTypeHeaderValue.Parse("application/x-protobuf"));
});

 整个Startup.cs代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Identity.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using DamService.Data;
using DamService.Models;
using DamService.Services;
using System.Net;
using Microsoft.AspNetCore.Diagnostics;
using Microsoft.AspNetCore.Http;
using Microsoft.Net.Http.Headers;
 
namespace DamService
{
    public class Startup
    {
        public Startup(IHostingEnvironment env)
        {
            var builder = new ConfigurationBuilder()
                .SetBasePath(env.ContentRootPath)
                .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true);
 
            if (env.IsDevelopment())
            {
                // For more details on using the user secret store see http://go.microsoft.com/fwlink/?LinkID=532709
                builder.AddUserSecrets();
            }
 
            builder.AddEnvironmentVariables();
            Configuration = builder.Build();
        }
 
        public IConfigurationRoot Configuration { get; }
 
        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            // Add framework services.
            services.AddDbContext<ApplicationDbContext>(options =>
                options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")));
 
            services.AddScoped(provider => new Hammergo.Data.DamWCFContext(Configuration.GetConnectionString("OdataDBConnectionString")));
 
            services.AddIdentity<ApplicationUser, IdentityRole>()
                .AddEntityFrameworkStores<ApplicationDbContext>()
                .AddDefaultTokenProviders();
 
            services.AddMvc(options =>
            {
                options.InputFormatters.Add(new ProtobufInputFormatter());
                options.OutputFormatters.Add(new ProtobufOutputFormatter());
                options.FormatterMappings.SetMediaTypeMappingForFormat("protobuf", MediaTypeHeaderValue.Parse("application/x-protobuf"));
            });
 
 
            // Add application services.
            //services.AddTransient<IEmailSender, AuthMessageSender>();
            //services.AddTransient<ISmsSender, AuthMessageSender>();
        }
 
        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
        {
            loggerFactory.AddConsole(Configuration.GetSection("Logging"));
            loggerFactory.AddDebug();
 
            //if (env.IsDevelopment())
            //{
            //    app.UseDeveloperExceptionPage();
            //    app.UseDatabaseErrorPage();
            //    app.UseBrowserLink();
            //}
            //else
            //{
            //    app.UseExceptionHandler("/Home/Error");
            //}
 
            app.UseExceptionHandler(_exceptionHandler);
 
            app.UseStaticFiles();
 
            app.UseIdentity();
 
            // Add external authentication middleware below. To configure them please see http://go.microsoft.com/fwlink/?LinkID=532715
 
            app.UseMvc(routes =>
            {
                routes.MapRoute(
                    name: "default",
                    template: "{controller=Home}/{action=Index}/{id?}");
            });
        }
 
        private void _exceptionHandler(IApplicationBuilder builder)
        {
            builder.Run(
              async context =>
                 {
                     context.Response.StatusCode = (int)HttpStatusCode.InternalServerError;
                     context.Response.ContentType = "text/plain";
 
                     var error = context.Features.Get<IExceptionHandlerFeature>();
                     if (error != null)
                     {
                         await context.Response.WriteAsync(error.Error.Message).ConfigureAwait(false);
                     }
                 });
        }
    }
}

  上面的

1
services.AddScoped(provider => new Hammergo.Data.DamWCFContext(Configuration.GetConnectionString("OdataDBConnectionString")));是我自己的数据库连接,可以使用自己的,也可以不用,我用的是EntityFrameWork 6.1.3 不是core,目前core还有一些功能没有,暂时不使用。
 
 添加一个测试用的Controller,
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
using System;
using System.Linq;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;
using Hammergo.Data;
using System.Linq.Expressions;
using System.Data.Entity;
using System.Collections.Generic;
using System.Threading.Tasks;
 
namespace DamService.Controllers
{
    public class AppsController : Controller
    {
        private readonly DamWCFContext _dbContext;
        private readonly ILogger _logger;
 
        public AppsController(
           DamWCFContext dbContext,
        ILoggerFactory loggerFactory)
        {
            _dbContext = dbContext;
            _logger = loggerFactory.CreateLogger<AccountController>();
        }
 
        [HttpGet]
        public async Task<List<App>> Top10()
        {
            return await _dbContext.Apps.Take(10).ToListAsync();
        }
 
    }
}
客户端测试代码:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
var client = new HttpClient { BaseAddress = new Uri("http://localhost.Fiddler:40786/") };
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/x-protobuf"));
 
HttpResponseMessage response = null;
 
//test top 10
string uri = "Apps/Top10";
Trace.WriteLine("\n Test {0}.", uri);
response = client.GetAsync(uri).Result;
 
if (response.IsSuccessStatusCode)
{
    var result = response.Content.ReadAsAsync<List<Hammergo.Poco.App>>(new[] { new ProtoBufFormatter() }).Result;
 
    Assert.AreEqual(result.Count, 10, "反序列化失败");
 
    
    Console.WriteLine("{0} test success!", uri);
}
else
{
    var message = response.Content.ReadAsStringAsync().Result;
    Console.WriteLine("{0} ({1})\n message: {2} ", (int)response.StatusCode, response.ReasonPhrase, message);
}

  

  

1
http://localhost.Fiddler:40786/  这里40786为服务端口,Fiddler表示使用了Fiddler代理,这样在使用时需要开启Fiddler,如果不使用Fidller,将URI修改为:
1
http://localhost:40786/