Spark：java.lang.IllegalArgumentException：非法模式组件架构不匹配答案

【问题标题】：Spark: java.lang.IllegalArgumentException: Illegal pattern component Schema MismatchSpark：java.lang.IllegalArgumentException：非法模式组件架构不匹配
【发布时间】：2019-11-26 16:52:46
【问题描述】：

我有这段代码，想插入 hive 但现在打印到控制台。如果我做简单的字符串而不是模式，它会打印，但如果是模式，它会给我异常

java.lang.IllegalArgumentException: Illegal pattern component: XXX
    at org.apache.commons.lang3.time.FastDateFormat.parsePattern(FastDateFormat.java:577)

 val schema_simple: StructType = StructType(
          Array
        (
         StructField("type", StringType, false),
         StructField("id", StringType, false),
         StructField("happened", StringType, false),
         StructField("processed", StringType, false),
         StructField("tracking_id", StringType, false),
          StructField("source_attributes", StringType, false),
          StructField("event_data", StringType, false)
        )
        )
    //
    val query = event_stream
      .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
      .select(from_json($"value", schema_simple).as("data"))
      .select("data.*")
      .writeStream
      .outputMode("append")
      .queryName("WriteToSource")  
      .format("console")

我正在测试的示例 JSON 如下，我从 Kafka 获得。我知道有一些不匹配但我找不到？

{
    "type": "identification.carrier",
    "id": "uui#abb0ef56-8562-4056-aa62-afb758a150ad",
    "happened": "2018-03-28T10:52:44.35+02:00",
    "processed": "2018-03-28T10:52:44.35+02:00",
    "tracking_id": "tid#ad2ebdb1-5d5b-4dcb-a058-494b6af89127",
    "source_attributes": {
        "id": "idsadm@app44A0",
        "data": {
            "info": "resumeIdentify: result: carrier has operator and mobile network",
            "callId": "ad2ebdb1-5d5b-4dcb-a058-494b6af89127",
            "contextId": "0aeb6745-a43c-4257-a535-6905d21a5297",
            "ids_state": "SUCCESS",
            "ids_strategy_type": "REDIRECT_MULTI_WITH_MSISDN_IN_HEADER",
            "ids_identity_type": "MSISDN",
            "country": "DE",
            "ids_result_type": "IDENTITY",
            "ids_result_network": "MobileNetwork[MOBILE=Vodafone D2 GmbH,operator=Operator[ref=10013,MNO=VODAFONE,countryIsoName=DE],countryIsoName=DE,ref=64,mnc=2,mcc=262,mvnos=[Operator[ref=30,MVNO=MOBILCOM-DEBITEL,countryIsoName=DE]]",
            "idsServiceLine": "599 387 240"
        }
    },
    "event_data": {
        "service": {
            "name": "IdsService",
            "version": "4.1"
        },
        "result": {
            "country": "DE",
            "mno": "VODAFONE",
            "mvno": "MOBILCOM-DEBITEL",
            "mcc": "262",
            "mnc": "2"
        }
    }
}

【问题讨论】：

标签： apache-spark spark-structured-streaming

【解决方案1】：

尝试添加最新版本的commons-lang3。

<dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.5</version>
</dependency>

【讨论】：