BigQuery：从 Cloud Storage 执行加载作业时，NodeJS 客户端库似乎不尊重 useAvroLogicalTypes答案

【问题标题】：BigQuery: NodeJS client library does not seem to respect useAvroLogicalTypes when performing a load job from Cloud StorageBigQuery：从 Cloud Storage 执行加载作业时，NodeJS 客户端库似乎不尊重 useAvroLogicalTypes
【发布时间】：2020-03-09 09:19:32
【问题描述】：

我正在尝试从我的 nodeJS 服务器运行作业，该服务器将 Cloud Storage 上的 AVRO 文件加载到 BigQuery 表中。这项工作完美无缺，但date 列在表中加载为INTEGER 类型。我在加载作业中包含了useAvroLogicalTypes 参数，但它似乎没有任何效果。

如果我使用SELECT DATE(TIMESTAMP_MILLIS(date)) 在表中转换date 列，我会得到正确的日期，但希望避免这个额外的转换步骤。我到处都读过，如果设置了参数，avro 逻辑类型可以隐式转换，但我一直无法让它工作。该表是由作业创建的，因此没有预先存在的架构。

我使用的客户端库版本是： 4.4.0 为@google-cloud/bigquery 4.1.2 为@google-cloud/storage。

AVRO 架构：

const schema = {
    "name": "root",
    "type": "record",
    "fields": [
      { "name": "date", "type": ["null", { "type": "long", "logicalType": "date" }]},
      { "name": "medium", "type": ["null", "string"] },
      { "name": "source", "type": ["null", "string"] },
      { "name": "campaign", "type": ["null", "string"] },
    ]
  };

工作代码

const options = {
    sourceFormat: 'AVRO',
    writeDisposition: 'WRITE_TRUNCATE',
    useAvroLogicalTypes: true,
    datasetID,
  };

bigquery
    .dataset(datasetID)
    .table(tableID)
    .load(storage.bucket(bucketName).file(fileName), options)
    .then(results => {

      res = results[0];

      // load() waits for the job to finish
      console.log(`Job ${res.id} completed.`);

      // Check the job's status for errors
      const errors = res.status.errors;

      if (errors && errors.length > 0) {
        E = errors;
      }
      // This kicks the execution back to where the Fiber.yield() statement stopped it
      fiber.resume();
    })
    .catch(err => {
      console.error('ERROR:', err);
    });

样本原始数据：

data = [
{"date":"2019-08-01","medium":"(none)","source":"(direct)","campaign":"(not set)","users":3053},
{"date":"2019-08-01","medium":"(not set)","source":"email-client","campaign":"(not set)","users":3},
{"date":"2019-08-01","medium":"affiliate","source":"sdn","campaign":"(not set)","users":1},
{"date":"2019-08-01","medium":"email","source":"corner","campaign":"onboarding","users":1},
{"date":"2019-08-01","medium":"email","source":"custom-playlist","campaign":"fonboarding","users":1},
{"date":"2019-08-01","medium":"email","source":"deref-mail.com","campaign":"(not set)","users":2},
{"date":"2019-08-01","medium":"email","source":"faketempmail","campaign":"(not set)","users":1},
{"date":"2019-08-01","medium":"email","source":"fundx","campaign":"email_campaign","users":1},
{"date":"2019-08-01","medium":"email","source":"email-client","campaign":"(not set)","users":14},
{"date":"2019-08-01","medium":"email","source":"email-client","campaign":"100k","users":2},
]

我使用 momentJS 和 underscoreJS 以及一个简单的 map 函数将 date 属性转换为 long：

data = _.map(data, row => {
    row.date = moment(row.date).isValid() ? +moment(row.date).valueOf() : null;
    return row;
  });

【问题讨论】：

我能够毫无问题地上传 '{"type": "string", "logicalType": "date"}'，而且 BigQuery 类型确实是 'DATE'。您能否上传原始数据的样本？这将帮助我尝试重现您的场景。
感谢您的帮助@F10，我已经包含了一个我一直在测试的示例数据集。欣赏它。同样从您的评论来看，您似乎使用了 type == string 和 logicalType = date，从 BQ 文档和 AVRO 规范来看，logicalType 日期似乎需要使用 INT 值，而不是字符串？
我将发布一个答案以对此进行更好的解释。

标签： node.js google-bigquery google-cloud-storage avro google-api-nodejs-client

【解决方案1】：

您提到的方式是“default”方式，因为当您不使用 useAvroLogicalTypes 时，Avro 逻辑类型 date 将在 BigQuery 中存储为 INTEGER。

这也取决于您的 AVRO 架构是如何制作的。例如，我必须使用架构构建一个 AVRO 文件

"fields":[
    {"logicaltype": "date", "type": "string", "name": "field1"}
]

我使用以下代码正确上传了我的 DATE 数据：

const metadata = {
      sourceFormat: 'AVRO',
      useAvroLogicalTypes: true,
      createDisposition: 'CREATE_IF_NEEDED',
      writeDisposition: 'WRITE_TRUNCATE',
      schema: {
        fields: [
          {
            name: "field1",
            type: "DATE",
            logicalType: "STRING",
            mode: "NULLABLE"
          }
        ],
      },
      location: 'US',
    };
    const [job] = await bigquery
      .dataset(datasetId)
      .table(tableId)
      .load(storage.bucket(bucketName).file(filename), metadata);

根据您共享的数据，它应该适用于此配置，因为您的 date 数据是 String。

希望对你有帮助。

【讨论】：