将 Json 数据解码为 Avro 类答案

【问题标题】：Decoding Json data to Avro classes将 Json 数据解码为 Avro 类
【发布时间】：2017-04-26 16:44:00
【问题描述】：

我有一些以纯文本 Json 形式存储的文件记录。样本记录：

{
      "datasetID": "Orders",
      "recordID": "rid1",
      "recordGroupID":"asdf1",
      "recordType":"asdf1",
      "recordTimestamp": 100,
      "recordPartitionTimestamp": 100,
      "recordData":{
        "customerID": "cid1",
        "marketplaceID": "mid1",
        "quantity": 10,
        "buyingDate": "1481353448",
        "orderID" : "oid1"
      }
}

对于每条记录，recordData 可能是 null。如果存在recordData，则orderID 可能是null。

我编写了以下 Avro 模式来表示结构：

[{
  "namespace":"model",
  "name":"OrderRecordData",
  "type":"record",
  "fields":[
    {"name":"marketplaceID","type":"string"},
    {"name":"customerID","type":"string"},
    {"name":"quantity","type":"long"},
    {"name":"buyingDate","type":"string"},
    {"name":"orderID","type":["null", "string"]}
  ]
},
{
  "namespace":"model",
  "name":"Order",
  "type":"record",
  "fields":[
    {"name":"datasetID","type":"string"},
    {"name":"recordID","type":"string"},
    {"name":"recordGroupID","type":"string"},
    {"name":"recordType","type":"string"},
    {"name":"recordTimestamp","type":"long"},
    {"name":"recordPartitionTimestamp","type":"long"},
    {"name":"recordData","type": ["null", "model.OrderRecordData"]}
  ]
}]

最后，我使用以下方法将每个 String 记录反序列化到我的 Avro 类中：

Order jsonDecodeToAvro(String inputString) {
        return new SpecificDatumReader<Order>(Order.class)
           .read(null, DecoderFactory.get().jsonDecoder(Order.SCHEMA$, inputString));
}

但在尝试达到上述记录时，我不断收到异常：

org.apache.avro.AvroTypeException: Unknown union branch customerID
    at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:445)

我做错了什么？我正在使用 JDK8 和 Avro 1.7.7

【问题讨论】：

标签： java avro

【解决方案1】：

json输入必须是格式

{
  "datasetID": "Orders",
  "recordID": "rid1",
  "recordGroupID":"asdf1",
  "recordType":"asdf1",
  "recordTimestamp": 100,
  "recordPartitionTimestamp": 100,
  "recordData":{
  "model.OrderRecordData" :{
    "orderID" : null,
    "customerID": "cid1",
    "marketplaceID": "mid1",
    "quantity": 10,
    "buyingDate": "1481353448"
    } 
  }
}

这是因为 Avro 的 JSON 编码处理联合和空值的方式。

看看这个：

How to fix Expected start-union. Got VALUE_NUMBER_INT when converting JSON to Avro on the command line?

关于此还有一个未解决的问题： https://issues.apache.org/jira/browse/AVRO-1582

【讨论】：