使用 logstash 更新索引失败，可能是错误？

Question

我正在使用 Logstash 进行练习，但我无法理解此错误的来源。 我想通过删除字段并重命名来修改索引。 我在 Logstash 5.6.2

这是我的条目示例：

{
    "_index": "test-shakespeare-italian",
    "_type": "logs",
    "_id": "AV8QsD5pzoUbREG_8icr",
    "_score": 1,
    "_source": {
      "nome_opera": "Henry IV",
      "battuta_numero": "3.2.9",
      "battuta": "Make me believe that thou art only markd",
      "@timestamp": "2017-10-12T13:05:06.519Z",
      "id_battuta": 1841,
      "dialogo_numero": 1,
      "@version": "1",
      "interlocutore": "KING HENRY IV"
    }
  }

这是我的管道：

# this pipeline updates an existing index by
# changing name of field "nome_opera" in "opera"
# dropping field "dialogo numero"
input {
  elasticsearch {
    hosts => "localhost"
    index => "test-shakespeare-italian"
    query => '{"query": {"match_all": {}}}'
  }
}
filter {
  mutate {
    rename => { "nome_opera" => "opera"}
    remove_field => "dialogo_numero"
  }
}
output {
  elasticsearch { 
    index => "test-shakespeare-italian"
    action => "update"
    document_id => "OOW-j2DeSCmnsVWVsywOVQ"
    hosts => "localhost:9200"
    version => "1"
  }
  #stdout { codec => rubydebug }
}

首先我没有使用版本字段，但这产生了以下错误：

[2017-10-13T11:29:25,464][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>409, :action=>["update", {:_id=>"OOW-j2DeSCmnsVWVsywOVQ", :_index=>"test-shakespeare-italian", :_type=>"logs", :_routing=>nil, :_retry_on_conflict=>1}, 2017-10-12T13:05:08.907Z %{host} %{message}], :response=>{"update"=>{"_index"=>"test-shakespeare-italian", "_type"=>"logs", "_id"=>"OOW-j2DeSCmnsVWVsywOVQ", "status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[logs][OOW-j2DeSCmnsVWVsywOVQ]: version conflict, current version [17901] is different than the one provided [17900]", "index_uuid"=>"OOW-j2DeSCmnsVWVsywOVQ", "shard"=>"3", "index"=>"test-shakespeare-italian"}}}}

然后我添加了版本字段，但管道仍然失败并出现以下错误：

[2017-10-13T11:38:39,467][ERROR][logstash.outputs.elasticsearch] Encountered an unexpected error submitting a bulk request! Will retry. {:error_message=>"undefined method `sanitized' for \"http://localhost:9200/_bulk\":String", :class=>"NoMethodError", :backtrace=>["/usr/local/Cellar/logstash/5.6.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:249:in `safe_bulk'", "/usr/local/Cellar/logstash/5.6.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:119:in `submit'", "/usr/local/Cellar/logstash/5.6.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:87:in `retrying_submit'", "/usr/local/Cellar/logstash/5.6.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:38:in `multi_receive'", "/usr/local/Cellar/logstash/5.6.2/libexec/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in `multi_receive'", "/usr/local/Cellar/logstash/5.6.2/libexec/logstash-core/lib/logstash/output_delegator.rb:49:in `multi_receive'", "/usr/local/Cellar/logstash/5.6.2/libexec/logstash-core/lib/logstash/pipeline.rb:436:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/local/Cellar/logstash/5.6.2/libexec/logstash-core/lib/logstash/pipeline.rb:435:in `output_batch'", "/usr/local/Cellar/logstash/5.6.2/libexec/logstash-core/lib/logstash/pipeline.rb:381:in `worker_loop'", "/usr/local/Cellar/logstash/5.6.2/libexec/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}

• 我确定 document_id 是正确的。
• 如果我创建一个新索引而不是更新索引，则管道可以完美运行
• --config.test_and_exit 成功

我在网上找到了这个 https://discuss.elastic.co/t/logstash-encountered-an-unexpected-error-submitting-a-bulk-request-undefined-method-sanitized/103360/2

还有这个 https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/612

这让我认为这是一个错误，而不是我的错。 不幸的是，这些链接中没有提供真正的解决方案，我是否必须恢复到旧版本的 logstash？还有其他想法吗？

Answer 1

如果你通过stdout获取最终结果并通过bulk api应用它是否有效？

我认为过滤器处理器后的最终 JSON 肯定有问题