Druid 将 0 或 0.0 存储为空值答案

【问题标题】：Druid storing the 0 or 0.0 as null valuesDruid 将 0 或 0.0 存储为空值
【发布时间】：2019-02-07 13:56:52
【问题描述】：

来自 HDP-2.6.5.0 的 druid .10.1 版本我们正在使用 druid-kafka 索引器服务摄取将数据从 kafka 主题加载到 druid 中，在此期间我们发现 druid 正在存储具有 0 或 0.0 的指标值被存储为 null 并且在通过 superset 或 Druid api 检索时得到响应为空。如果我们在这里遗漏了什么，需要建议。

来自超集的错误：

{"status": "failed", "error_type": "warning", "error": "unsupported operand type(s) for +: 'int' and 'NoneType'"}

摄取规范文件如下：

{
    "type": "kafka",
    "dataSchema": {
        "dataSource": "data-source",
        "parser": {
            "type": "string",
            "parseSpec": {
                "format": "json",
                "timestampSpec": {
                    "column": "datetime",
                    "format": "YYYYMMdd_HHmmss"
                },
                "columns": [
                    "created_date",
                    "s_type",
                    "datetime",
                    "ds_ser",
                    "ven",
                    "cou_name",
                    "c_name",
                    "d_name",
                    "dv_name",
                    "p_name",
                    "redTime",
                    "wrTime",
                    "tRate",
                    "MTRate"
                ],
                "dimensionsSpec": {
                    "dimensions": [
                        "created_date",
                    "s_type",
                    "datetime",
                    "ds_ser",
                    "ven",
                    "cou_name",
                    "c_name",
                    "d_name",
                    "dv_name",
                    "p_name",
                    ]
                }
            }
        },
        "metricsSpec": [{
            "name": "count",
            "type": "count"
        },
            {
                "type": "doubleMax",
                "name": "redTime",
                "fieldName": "redTime"
            },
            {
                "type": "doubleMax",
                "name": "wrTime",
                "fieldName": "wrTime"
            },
            {
                "type": "longMax",
                "name": "tRate",
                "fieldName": "tRate"
            },
            {
                "type": "longMax",
                "name": "MTRate",
                "fieldName": "MTRate"
            }
        ],
        "granularitySpec": {
            "type": "uniform",
            "segmentGranularity": "HOUR",
            "queryGranularity": "NONE"
        }
    },
    "tuningConfig": {
        "type": "kafka",
        "maxRowsPerSegment": 5000000
    },
    "ioConfig": {
        "topic": "ptopic",
        "useEarliestOffset": "true",
        "consumerProperties": {
            "bootstrap.servers": "host:port"
        },
        "taskCount": 1,
        "replicas": 1,
        "taskDuration": "PT5M"
    }
}

来自德鲁伊的休息 api 使用： http://host:port/druid/v2?pretty

正文：

{
    "queryType": "groupBy",
    "dataSource": "data-source",
    "granularity": "all",
    "dimensions": ["ds_ser"],
    "aggregations": [
        {"type": "doubleMax", "name": "redTime", "redTime": "writeresponsetime"},
        {"type": "doubleMax", "name": "wrTime", "wrTime": "totalResponseTime"},
        {"type": "longMax", "name": "tRate", "fieldName": "tRate"},
        {"type": "longMax", "name": "MTRate", "MTRate": "MaxTransferRate"}

    ],
    "intervals": ["2019-01-02T00:00/2019-01-02T23:59"]
}

来自德鲁伊的回应：

[
    {
        "version": "v1",
        "timestamp": "2019-01-02T00:00:00.000Z",
        "event": {
            "redTime": null,
            "ds_ser": "240163",
            "wrTime": null,
            "tRate": null,
            "MTRate": null
        }
    },
    {
        "version": "v1",
        "timestamp": "2019-01-02T00:00:00.000Z",
        "event": {
            "redTime": null,
            "ds_ser": "443548",
            "wrTime": null,
            "tRate": 0,
            "MTRate": null
        }
    }
]

Kafka 中的数据：

> {"created_date":"2019-02-03T18:35:59.514Z","s_type":"BLOCK","datetime":"20181121_070000","ds_ser":"443551","ven":"abc","cou_name":"USA","c_name":"Piscataway","d_name":"Piscataway","dv_name":"USPSCG","p_name":"443551-CK","redTime":0.0,"wrTime":0.0,"tRate":0,"MTRate":0}
> {"created_date":"2019-02-03T18:35:59.514Z","s_type":"BLOCK","datetime":"20181121_070000","ds_ser":"443551","ven":"abc","cou_name":"USA","c_name":"Piscataway","d_name":"Piscataway","dv_name":"USPSCG4","p_name":"443551-CF","redTime":0.0,"wrTime":0.0,"tRate":0,"MTRate":0}

【问题讨论】：

在您的第二个对象中，tRate 返回 0，这意味着它的存储数字正确。您的某些数据在这些字段中是否为空？
@Jainik 感谢您的回复。我已经仔细检查了 kafka 中的数据，但我找不到空值。我附上了来自 kafka 的样本数据。此外，我们用 0 显式替换空白和空值，然后将值转换为 double 和 long。这看起来从德鲁伊不一致的行为
如果您的数据没有 null 并且您仍然看到问题，请尝试 Javascript Aggregator
根据当前版本“Long/Float columns Nulls 被认为等同于 0”。这意味着您需要将响应中的 null 视为 0。
好！！！我建议发布详细的答案。我之前看到过字段名称不匹配的此类问题，但在您的问题中找不到任何问题。请突出显示并发布详细答案

标签： apache-kafka druid superset apache-superset data-ingestion

【解决方案1】：

嗯，我找到了自己问题的答案。

我在准备德鲁伊 kafka inderex json 时犯了错误。我不知道这些字段区分大小写。此处发布的 json sn-p 是虚构的，因此字段名称匹配，但在我的实际生产代码和 json 文件中，这些不匹配，因此 druid 假设这些为新字段并在摄取它们时将值分配为 null。下面的例子：

卡夫卡 Json：

{"created_date":"2019-02-03T18:35:59.514Z","s_type":"BLOCK","datetime":"20181121_070000","ds_ser":"443551","ven":"abc","cou_name":"USA","c_name":"Piscataway","d_name":"Piscataway","dv_name":"USPSCG","p_name":"443551-CK","redTime":0.0,"wrTime":0.0,"tRate":0,"MTRate":0}

Druid indexer json 列是这样的：

"columns": [
                    "created_date",
                    "s_type",
                    "datetime",
                    "ds_ser",
                    "ven",
                    "cou_name",
                    "c_name",
                    "d_name",
                    "dv_name",
                    "p_name",
                    "redTime",
                    "wrtime",
                    "trate",
                    "MTRate"
                ],

如果我们在上面观察到wrTime --> wrtime 和tRate --> trate 不匹配。所以对我来说，这是根本原因，在解决名称后，德鲁伊开始摄取正确的值。

【讨论】：