Logstash MalformedCSVError: Illegal quoting in line |如何配置 logstash conf 以读取多列多行 CSV？答案

【问题标题】：Logstash MalformedCSVError: Illegal quoting in line | how to configure logstash conf to read multi column mutiline CSV?Logstash MalformedCSVError: Illegal quoting in line |如何配置 logstash conf 以读取多列多行 CSV？
【发布时间】：2020-06-14 15:34:38
【问题描述】：

已经通过CSV::MalformedCSVError: Illegal quoting in line 1 with SmarterCSV 和Rescue CSV::MalformedCsvError: Illegal quoting in line n 他们都没有解决我的问题。

CSV conf 与 Log Stash 的基本集成时遇到问题：

    input {
      file {
        path => "/softwares/imews_customer_wallet_details.csv"
        start_position => "beginning"
      }
    }
    filter {
      csv {
        columns => ["imiles_wallet_id","creation_date","gcid_cod","total_imiles","is_active"]
        separator => ","
      }
      date {
        match => ["creation_date", "dd-MM-yy"]
        target => "creation_date"
      }
    }
    output {
      elasticsearch {
        action => "index"
        hosts => ["localhost:9200"]
        index => "ime_customers"
      }
    }

错误堆栈在循环中打印以下行：


    [2020-03-02T13:36:51,968][WARN ][logstash.filters.csv     ][another_test] Error parsing csv {:field=>"message", :source=>"{\"level\":\"DEBUG\",\"timeUTC\":\"Mon Feb 24 08:05:20 UTC 2020\",\"timeUnix\":1582531520800,\"appID\":\"TEST\",\"hub\":\"CI\",\"environment\":\"NONE\",\"host\":\"Yashpal3150153\",\"logMessage\":\"http-outgoing-1 << Content-Type: application/x-amz-json-1.0\",\"threadID\":\"SimpleAsyncTaskExecutor-1\",\"cdll_version\":\"6.5.0\"}\r", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}

我错过了什么，这里有什么问题？

更新：

CSV：

    125,IME,"14-11-19","14-11-19",IME,"14-11-19",gcid3,100,TRUE
    124,IME,"14-11-19","14-11-19",IME,"14-11-19",gcid2,100,TRUE
    123,IME,"14-11-19","14-11-19",IME,"14-11-19",gcid1,252,TRUE

更新：管道：

     - pipeline.id: another_test
       queue.type: persisted
       path.config: "/softwares/logstash-7.6.0/config/*.conf"

配置包含：我提到的配置是：cust.conf

另外，我将 cust.conf 更新为：

    input {
      file {
        path => "D:/softwares/imews_customer_wallet_details.csv"
        start_position => "beginning"
      }
    }
    filter {
      csv {
        columns => ["imiles_wallet_id","creation_date","gcid_cod","total_imiles","is_active"]
        separator => ","
        codec => plain
        convert => {
          "imiles_wallet_id" => "integer"
          "is_active" => "boolean"
          "creation_date" => "date"
          "gcid_cod" => "string"
          "total_imiles" => "integer"
        } 
      }
    }
    output {
      elasticsearch {
        action => "index"
        hosts => ["localhost:9200"]
        index => "ime_customers"
      }
    }

现在错误是：

    LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<another_test>`
              create at org/logstash/execution/ConvergeResultExt.java:109
                 add at org/logstash/execution/ConvergeResultExt.java:37
      converge_state at D:/softwares/logstash-7.6.0/logstash-core/lib/logstash/agent.rb:339
    [2020-03-04T18:20:05,345][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<another_test>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "D:/softwares/logstash-7.6.0/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
    [2020-03-04T18:20:05,397][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<another_test>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "D:/softwares/logstash-7.6.0/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
    ```

【问题讨论】：

能否也显示 CSV 文件的前 2-3 行？
@Val 在问题本身中更新。
如果您检查错误消息，您会发现 Logstash 尝试解析的消息看起来并不像 CSV，它看起来像 JSON。您确定您的输入、您正在阅读的文件及其内容吗？
简单地用bin/logstash -f /softwares/logstash-7.6.0/config/cust.conf启动logstash它是如何工作的？
好的，问题是你有几个配置文件（即*.conf），可能还有几个输入。使用管道时，请注意所有配置文件都“合并”到一个配置中（可能有多个源、过滤器和输出）

标签： elasticsearch logstash elastic-stack logstash-configuration

【解决方案1】：

通过使用管道定义/softwares/logstash-7.6.0/config/*.conf 启动 Logstash，您可能有许多相互干扰的配置文件（即几个不同的输入、过滤器和输出）。所有这些配置文件都合并到一个配置中。

阅读this article 以了解如何最好地组织您的 Logstash 管道。同时，您可以像这样从命令行简单地启动 Logstash：

bin/logstash -f /softwares/logstash-7.6.0/config/cust.conf

【讨论】：