【问题标题】:Elastic NEST Client how to sort by on a nested propertyElastic NEST 客户端如何按嵌套属性排序
【发布时间】:2021-05-09 00:43:17
【问题描述】:

作为 NEST 库的新手,用对我来说完全没有意义的文档拉扯我的头发。

第一个问题:无法弄清楚如何将查询的结果映射到强类型模型,尽管我已经捕获了 json 结果并粘贴到了 C# 类,无论我尝试什么,除非我在查询中使用动态类型方法然后结果返回null。

从查询返回的 json 结果派生的模型:

public class Rootobject
{
    public int took { get; set; }
    public bool timed_out { get; set; }
    public _Shards _shards { get; set; }
    public Hits hits { get; set; }
}

public class _Shards
{
    public int total { get; set; }
    public int successful { get; set; }
    public int skipped { get; set; }
    public int failed { get; set; }
}

public class Hits
{
    public Total total { get; set; }
    public float max_score { get; set; }
    public Hit[] hits { get; set; }
}

public class Total
{
    public int value { get; set; }
    public string relation { get; set; }
}

public class Hit
{
    public string _index { get; set; }
    public string _type { get; set; }
    public string _id { get; set; }
    public float _score { get; set; }
    public _Source _source { get; set; }
}

public class _Source
{
    public DateTime timestamp { get; set; }
    public string level { get; set; }
    public string messageTemplate { get; set; }
    public string message { get; set; }
    public Fields fields { get; set; }
}

public class Fields
{
    public string LogEventCategory { get; set; }
    public string LogEventType { get; set; }
    public string LogEventSource { get; set; }
    public string LogData { get; set; }
    public string MachineName { get; set; }
    public int MemoryUsage { get; set; }
    public int ProcessId { get; set; }
    public string ProcessName { get; set; }
    public int ThreadId { get; set; }
}

查询返回的示例 JSON 结果:

{
  "took" : 16,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 8,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "webapp-razor-2021.05",
        "_type" : "_doc",
        "_id" : "n2tbTnkBwE4YgJowzRsT",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-05-09T00:41:47.2321845+01:00",
          "level" : "Information",
          "messageTemplate" : "{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}",
          "message" : "\"WebApp-RAZOR\"\"Application Startup\"\"System\"\"Application Starting Up\"",
          "fields" : {
            "LogEventCategory" : "WebApp-RAZOR",
            "LogEventType" : "Application Startup",
            "LogEventSource" : "System",
            "LogData" : "Application Starting Up",
            "MachineName" : "DESKTOP-OS52032",
            "MemoryUsage" : 4713408,
            "ProcessId" : 15152,
            "ProcessName" : "WebApp-RAZOR",
            "ThreadId" : 1
          }
        }
      },
      {
        "_index" : "webapp-razor-2021.05",
        "_type" : "_doc",
        "_id" : "oGtdTnkBwE4YgJowuxu_",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-05-09T00:43:54.0326968+01:00",
          "level" : "Information",
          "messageTemplate" : "{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}",
          "message" : "\"WebApp-RAZOR\"\"Application Startup\"\"System\"\"Application Starting Up\"",
          "fields" : {
            "LogEventCategory" : "WebApp-RAZOR",
            "LogEventType" : "Application Startup",
            "LogEventSource" : "System",
            "LogData" : "Application Starting Up",
            "MachineName" : "DESKTOP-OS52032",
            "MemoryUsage" : 4656048,
            "ProcessId" : 12504,
            "ProcessName" : "WebApp-RAZOR",
            "ThreadId" : 1
          }
        }
      },
      {
        "_index" : "webapp-razor-2021.05",
        "_type" : "_doc",
        "_id" : "oWtgTnkBwE4YgJownRtc",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-05-09T00:47:02.8954368+01:00",
          "level" : "Information",
          "messageTemplate" : "{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}",
          "message" : "\"WebApp-RAZOR\"\"Application Startup\"\"System\"\"Application Starting Up\"",
          "fields" : {
            "LogEventCategory" : "WebApp-RAZOR",
            "LogEventType" : "Application Startup",
            "LogEventSource" : "System",
            "LogData" : "Application Starting Up",
            "MachineName" : "DESKTOP-OS52032",
            "MemoryUsage" : 4717560,
            "ProcessId" : 17952,
            "ProcessName" : "WebApp-RAZOR",
            "ThreadId" : 1
          }
        }
      }
    ]
  }
}

第二个问题:我正在尝试按嵌套在 JSON 中的属性进行排序,例如名为“LogEventCategory”的属性

NEST 客户端方法:

var searchResponse = await _elasticClient.SearchAsync<dynamic>(s => s
   .RequestConfiguration(r => r
        .DisableDirectStreaming()
    )
    //.AllIndices()
   .From(0) // From parameter defines the offset from the first result you want to fetch.
   .Size(3) // Size parameter allows you to configure the maximum amount of hits to be returned.
   .Index("webapp-razor-*")
   //.Index("index-1,index-2")
   .Query(q => q
        .MatchAll()
    )
    .Sort(so => so
         .Field(fs => fs
             //.Field("@timestamp") // this one seems to work
             .Field("logEventCategory")
             .Order(SortOrder.Ascending)
             //.Order(ColumnSortOrder)
          )
    )
);

知道 NEST 高级客户端旨在映射到正确的模型,我意识到使用动态类型是不正确的,但我只是无法理解为什么使用直接从示例 json 响应创建/映射的类是仍然无法正常工作,并且这样做时返回的结果只是为我提供了前几个属性的空值。

下面的示例仅显示前几项(但仍为空)

【问题讨论】:

    标签: c# elasticsearch nest asp.net-core-5.0


    【解决方案1】:

    第一个问题:无法弄清楚如何将查询的结果映射到强类型模型,尽管我已经捕获了 json 结果并粘贴到了 C# 类,无论我尝试什么,除非我在查询中使用动态类型方法然后结果返回null。

    从查询返回的 json 结果派生的模型:

    您不需要为整个 JSON 响应定义模型;客户端将负责正确反序列化大部分的响应,因此唯一需要创建的 POCO 是 _source JSON 对象。根据提供的示例,这将类似于

    public class LogMessage
    {
        [PropertyName("@timestamp")]
        public DateTimeOffset Timestamp { get; set; }
        
        public string Level {get;set;}
        
        public string MessageTemplate {get;set;}
        
        public string Message {get;set;}
        
        public Dictionary<string, object> Fields {get;set;}
    }
    

    我们可以通过创建一个存根响应并将其传递给客户端进行反序列化来检查它是否正确反序列化

    private static void Main()
    {
        var defaultIndex = "default_index";
        var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
    
        var json = @"{
      ""took"" : 16,
      ""timed_out"" : false,
      ""_shards"" : {
            ""total"" : 1,
        ""successful"" : 1,
        ""skipped"" : 0,
        ""failed"" : 0
      },
      ""hits"" : {
            ""total"" : {
                ""value"" : 8,
          ""relation"" : ""eq""
        
        },
        ""max_score"" : 1.0,
        ""hits"" : [
          {
                ""_index"" : ""webapp-razor-2021.05"",
            ""_type"" : ""_doc"",
            ""_id"" : ""n2tbTnkBwE4YgJowzRsT"",
            ""_score"" : 1.0,
            ""_source"" : {
                    ""@timestamp"" : ""2021-05-09T00:41:47.2321845+01:00"",
              ""level"" : ""Information"",
              ""messageTemplate"" : ""{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}"",
              ""message"" : ""\""WebApp-RAZOR\""\""Application Startup\""\""System\""\""Application Starting Up\"""",
              ""fields"" : {
                        ""LogEventCategory"" : ""WebApp-RAZOR"",
                ""LogEventType"" : ""Application Startup"",
                ""LogEventSource"" : ""System"",
                ""LogData"" : ""Application Starting Up"",
                ""MachineName"" : ""DESKTOP-OS52032"",
                ""MemoryUsage"" : 4713408,
                ""ProcessId"" : 15152,
                ""ProcessName"" : ""WebApp-RAZOR"",
                ""ThreadId"" : 1
    
              }
                }
            },
          {
                ""_index"" : ""webapp-razor-2021.05"",
            ""_type"" : ""_doc"",
            ""_id"" : ""oGtdTnkBwE4YgJowuxu_"",
            ""_score"" : 1.0,
            ""_source"" : {
                    ""@timestamp"" : ""2021-05-09T00:43:54.0326968+01:00"",
              ""level"" : ""Information"",
              ""messageTemplate"" : ""{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}"",
              ""message"" : ""\""WebApp-RAZOR\""\""Application Startup\""\""System\""\""Application Starting Up\"""",
              ""fields"" : {
                        ""LogEventCategory"" : ""WebApp-RAZOR"",
                ""LogEventType"" : ""Application Startup"",
                ""LogEventSource"" : ""System"",
                ""LogData"" : ""Application Starting Up"",
                ""MachineName"" : ""DESKTOP-OS52032"",
                ""MemoryUsage"" : 4656048,
                ""ProcessId"" : 12504,
                ""ProcessName"" : ""WebApp-RAZOR"",
                ""ThreadId"" : 1
    
              }
                }
            },
          {
                ""_index"" : ""webapp-razor-2021.05"",
            ""_type"" : ""_doc"",
            ""_id"" : ""oWtgTnkBwE4YgJownRtc"",
            ""_score"" : 1.0,
            ""_source"" : {
                    ""@timestamp"" : ""2021-05-09T00:47:02.8954368+01:00"",
              ""level"" : ""Information"",
              ""messageTemplate"" : ""{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}"",
              ""message"" : ""\""WebApp-RAZOR\""\""Application Startup\""\""System\""\""Application Starting Up\"""",
              ""fields"" : {
                        ""LogEventCategory"" : ""WebApp-RAZOR"",
                ""LogEventType"" : ""Application Startup"",
                ""LogEventSource"" : ""System"",
                ""LogData"" : ""Application Starting Up"",
                ""MachineName"" : ""DESKTOP-OS52032"",
                ""MemoryUsage"" : 4717560,
                ""ProcessId"" : 17952,
                ""ProcessName"" : ""WebApp-RAZOR"",
                ""ThreadId"" : 1
    
              }
                }
            }
        ]
      }
    }";
    
        // create a connection that always returns the above response
        var connection = new InMemoryConnection(Encoding.UTF8.GetBytes(json));
    
        var settings = new ConnectionSettings(pool, connection)
            .DefaultIndex(defaultIndex);
            
        var client = new ElasticClient(settings);
    
        // actually doesn't matter what query we send through as we'll always get the stubbed response
        var searchResponse = client.Search<LogMessage>(s => s);
        
        foreach (var document in searchResponse.Documents)
        {
            Console.WriteLine($"{document.Timestamp} {document.Message}");
        }
    }
    

    产生

    9/05/2021 12:41:47 AM +01:00 "WebApp-RAZOR""Application Startup""System""Application Starting Up"
    9/05/2021 12:43:54 AM +01:00 "WebApp-RAZOR""Application Startup""System""Application Starting Up"
    9/05/2021 12:47:02 AM +01:00 "WebApp-RAZOR""Application Startup""System""Application Starting Up"
    

    第二个问题:我正在尝试按嵌套在 JSON 中的属性进行排序,例如名为“LogEventCategory”的属性

    LogMessage POCO 中,fields 映射为Dictionary&lt;string,object&gt;,因为我猜键/值可以是动态的。对该对象中的属性进行排序将是

    var searchResponse = client.Search<LogMessage>(s => s
        .Sort(so => so
            .Ascending(f => f.Fields["LogEventCategory"])
        )
    );
    

    产生以下请求

    POST http://localhost:9200/default_index/_search?pretty=true&typed_keys=true 
    {
      "sort": [
        {
          "fields.LogEventCategory": {
            "order": "asc"
          }
        }
      ]
    }
    

    如果 Elasticsearch 已经推断出 LogEventCategory 的映射,即没有为其提供明确的数据类型映射,那么它将被映射为带有 keyword multi-field (或子字段)的 text 字段.尝试在 fielddata 尚未启用(默认情况下禁用)的 text 字段上进行排序将导致错误。通常应该使用keyword 字段进行排序和聚合,因此如果映射已被推断,则可以使用keyword 多字段进行排序

    var searchResponse = client.Search<LogMessage>(s => s
        .Sort(so => so
            .Ascending(f => f.Fields["LogEventCategory"].Suffix("keyword"))
        )
    );
    

    【讨论】:

    • 非常感谢拉斯,感谢您花时间详细回答。我缺少的基本部分是我没有在我的 C# 模型类中将“字段”对象指定为类型字典。希望使用类型 Dictionary 意味着我不需要担心内部属性,因为这些可能会在使用 Serilog 的不同应用程序之间略有变化,因为我没有在所有应用程序之间使用相同的结构化日志事件属性进行记录。
    • 我现在也开始研究 Elastic 较低级别的客户端,因为我相信根据用户当时想要应用的过滤器数量,编译更加动态的查询可能会更容易的搜索。使用 NEST,看起来所有查询都根据查询语法的性质进行了硬编码,因此灵活性不如使用一些自定义 JavaScript 在发送到服务器进行处理之前编译查询。
    • 低级客户端提供了一种更简单的方法来制定请求,可以将 JSON 字符串作为查询传递,但是,我建议从 NEST 开始,这将允许构建具有类型的查询或表达式。对于字段,NEST 可以传递字符串,对于不同的文档结构,可以使用更动态的结构,例如JObject(与JsonNetSerializer 结合使用)。查看示例演练,了解有关如何构建更复杂查询的一些想法:github.com/elastic/elasticsearch-net-example
    猜你喜欢
    • 2013-07-23
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2010-11-13
    • 2016-10-14
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多