Searchkick / Elasticsearch 聚合的结果不一致答案

【问题标题】：Inconsistant Results from Searchkick / Elasticsearch AggregationsSearchkick / Elasticsearch 聚合的结果不一致
【发布时间】：2020-05-06 22:23:21
【问题描述】：

首先，我将清理 IRB 的输出（这是来自生产）。

我在我的 Rails 应用程序中使用了一些 search kick 聚合。现在在生产中在开发中测试良好我看到不一致的结果。我的字段中的数据应该清楚地显示在我的聚合中。

这是我的普通旧 Active Record 数据 - 我的 BudgetItem 模型有一个 total：

irb(main):008:0> BudgetItem.where(budget_id: 3).order(:cbs_item_id).select(:id, :cbs_item_id, :total)
  BudgetItem Load (1.4ms)  SELECT  "budget_items"."id", "budget_items"."cbs_item_id", "budget_items"."total" FROM "budget_items" WHERE "budget_items"."company_id" = 26 AND "budget_items"."budget_id" = $1 ORDER BY "budget_items"."cbs_item_id" ASC LIMIT $2  [["budget_id", 3], ["LIMIT", 11]]
=> #<ActiveRecord::Relation [#<BudgetItem id: 28, company_id: 26, cbs_item_id: 3, total: 0.1e5>, #<BudgetItem id: 29, company_id: 26, cbs_item_id: 12, total: 0.8e5>, #<BudgetItem id: 34, company_id: 26, cbs_item_id: 15, total: 0.1e5>, #<BudgetItem id: 41, company_id: 26, cbs_item_id: 16, total: 0.141e6>, #<BudgetItem id: 35, company_id: 26, cbs_item_id: 18, total: 0.1e5>, #<BudgetItem id: 33, company_id: 26, cbs_item_id: 18, total: 0.12e5>, #<BudgetItem id: 27, company_id: 26, cbs_item_id: 20, total: 0.2e5>, #<BudgetItem id: 6, company_id: 26, cbs_item_id: 23, total: 0.184e6>, #<BudgetItem id: 5, company_id: 26, cbs_item_id: 23, total: 0.2288e6>, #<BudgetItem id: 30, company_id: 26, cbs_item_id: 24, total: 0.45e5>, ...]>

这个数是 41

通过 Searchkick 进行相同的搜索：

BudgetItem.search("*", where: {budget_id: 3}).count = 41

甚至这个：

irb(main):025:0> BudgetItem.search("*", where: {budget_id: 3, cbs_item_id: 16}).first.total
= 26 AND "budget_items"."id" = $1  [["id", 41]]
=> 0.141e6

注意cbs_item_id: 16, total: 0.141e6 (141000) - 值在模型中很明显。

现在我尝试对此运行聚合：

irb(main):019:0> BudgetItem.search("*", body_options: { aggs: { cbs: { terms: { field: "cbs_item_id" }, aggs: { "total": { "sum": { "field": "total" } } } } } },  where: {budget_id: 3}).aggs
  BudgetItem Search (5.4ms)  pacific-canbriam-20191213_budget_items_production/_search {"query":{"bool":{"must":{"match_all":{}},"filter":[{"term":{"budget_id":{"value":3}}}]}},"timeout":"11s","_source":false,"size":10000,"aggs":{"cbs":{"terms":{"field":"cbs_item_id"},"aggs":{"total":{"sum":{"field":"total"}}}}}}
=> {"cbs"=>{"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>13, "buckets"=>[{"key"=>24, "doc_count"=>4, "total"=>{"value"=>90000.0}}, {"key"=>25, "doc_count"=>4, "total"=>{"value"=>114000.0}}, {"key"=>39, "doc_count"=>4, "total"=>{"value"=>107325.0}}, {"key"=>43, "doc_count"=>4, "total"=>{"value"=>209820.0}}, {"key"=>18, "doc_count"=>2, "total"=>{"value"=>22000.0}}, {"key"=>23, "doc_count"=>2, "total"=>{"value"=>412800.0}}, {"key"=>38, "doc_count"=>2, "total"=>{"value"=>13500.0}}, {"key"=>49, "doc_count"=>2, "total"=>{"value"=>161000.0}}, {"key"=>57, "doc_count"=>2, "total"=>{"value"=>20300.0}}, {"key"=>58, "doc_count"=>2, "total"=>{"value"=>32200.0}}]}}

数据完全不一致，aggs 缺失。注意键 16 丢失。疯狂的是我在另一列上有另一个聚合，而且它工作得非常好。我在这里错过了什么吗？我已经尝试过settings: {number_of_shards: 1}。

更令人沮丧的是：

  aggs: {
    "grand_total": { "sum": { "field": "total"  } },
   }

total 的常规 agg 在相同的结果中起作用。

【问题讨论】：

标签： elasticsearch searchkick

【解决方案1】：

尝试在聚合中设置size。根据响应中的sum_other_doc_count，没有返回13 个文档。

【讨论】：