Apache Druid 段合并任务提交失败答案

【问题标题】：Apache Druid segment merge task submition failureApache Druid 段合并任务提交失败
【发布时间】：2018-01-16 16:54:43
【问题描述】：

我正在使用 Druid 0.9.1.1 并尝试将每天数据源的所有段合并到一个段中。而合并任务启动失败并出现错误：

{"error":"Instantiation of [simple type, class io.druid.timeline.DataSegment] value failed: null (through reference chain: java.util.ArrayList[0])"}

我从段元数据查询中获得了段详细信息。 driud 文档没有帮助，因为只指定了整体查询的原始结构，而不是所需的段详细结构（以下是 druid 文档的建议）。

{
"type": "merge",
"id": <task_id>,
"dataSource": <task_datasource>,
"aggregations": <list of aggregators>,
"segments": <JSON list of DataSegment objects to merge>

}

示例查询：

{
"type": "merge",
"id": "envoy_merge_task",
"dataSource": "dcap.envoy.diskmounts.kafka",
"segments": [{"id":"dcap.sermon.threshold.kafka_2017-05-22T00:00:00.000Z_2017-05-23T00:00:00.000Z_2017-05-22T07:00:02.951Z","intervals":["2017-05-22T00:00:00.000Z/2017-05-23T00:00:00.000Z"],"columns":{},"size":5460959,"numRows":41577,"aggregators":null,"queryGranularity":null},{"id":"dcap.sermon.threshold.kafka_2017-05-22T00:00:00.000Z_2017-05-23T00:00:00.000Z_2017-05-22T07:00:02.951Z_1","intervals":["2017-05-22T00:00:00.000Z/2017-05-23T00:00:00.000Z"],"columns":{},"size":5448881,"numRows":41577,"aggregators":null,"queryGranularity":null},{"id":"dcap.sermon.threshold.kafka_2017-05-22T00:00:00.000Z_2017-05-23T00:00:00.000Z_2017-05-22T07:00:02.951Z_2","intervals":["2017-05-22T00:00:00.000Z/2017-05-23T00:00:00.000Z"],"columns":{},"size":5454452,"numRows":41571,"aggregators":null,"queryGranularity":null},{"id":"dcap.sermon.threshold.kafka_2017-05-22T00:00:00.000Z_2017-05-23T00:00:00.000Z_2017-05-22T07:00:02.951Z_3","intervals":["2017-05-22T00:00:00.000Z/2017-05-23T00:00:00.000Z"],"columns":{},"size":5456267,"numRows":41569,"aggregators":null,"queryGranularity":null}] }

我为“segments”键尝试了不同形式的结构，导致同样的错误。

例子：

"segments": [{"id":"dcap.envoy.diskmounts.kafka_2017-05-21T06:00:00.000Z_2017-05-21T07:00:00.000Z_2017-05-21T06:02:43.482Z"},{"id":"dcap.envoy.diskmounts.kafka_2017-05-21T06:00:00.000Z_2017-05-21T07:00:00.000Z_2017-05-21T06:02:43.482Z_1"},{"id":"dcap.envoy.diskmounts.kafka_2017-05-21T06:00:00.000Z_2017-05-21T07:00:00.000Z_2017-05-21T06:02:43.482Z_2"},{"id":"dcap.envoy.diskmounts.kafka_2017-05-21T06:00:00.000Z_2017-05-21T07:00:00.000Z_2017-05-21T06:02:43.482Z_3"}]

段合并任务的正确结构是什么。

【问题讨论】：

IndexTask improvements .. 德鲁伊贡献者正在努力。

标签： apache merge task segment druid

【解决方案1】：

我用于分段的格式是

    "segments":[
        {
            "dataSource": "wikiticker88",
            "interval": "2015-09-12T02:00:00.000Z/2015-09-12T03:00:00.000Z",
            "version": "2018-01-16T07:23:16.425Z",
            "loadSpec": {
                "type": "local",
                "path": "/home/linux/druid-0.11.0/var/druid/segments/wikiticker88/2015-09-12T02:00:00.000Z_2015-09-12T03:00:00.000Z/2018-01-16T07:23:16.425Z/0/index.zip"
            },
            "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user",
            "metrics": "count,added,deleted,delta,user_unique",
            "shardSpec": {
                "type": "none"
            },
            "binaryVersion": 9,
            "size": 198267,
            "identifier": "wikiticker88_2015-09-12T02:00:00.000Z_2015-09-12T03:00:00.000Z_2018-01-16T07:23:16.425Z"
        },
]

使用它来获取分段的元数据

/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments?full

【讨论】：