Splunk Enterprise：未编入索引的大型 json 事件答案

【问题标题】：Splunk Enterprise: large json events not indexedSplunk Enterprise：未编入索引的大型 json 事件
【发布时间】：2019-01-02 11:02:06
【问题描述】：

我有一个这样定义的源类型（system\local\props.conf）：

[my_json]
DATETIME_CONFIG = 
INDEXED_EXTRACTIONS = json
KV_MODE = json
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = Timestamp
category = Structured
description = json 
disabled = false
pulldown_type = 1
TIME_FORMAT = HH:mm:ss.fff
LINE_BREAKER = ([\r\n]+)

limits.conf：

[spath]
# Number of characters to read from an XML or JSON event when
# auto extracting.
extraction_cutoff = 5000
extract_all = true

如果我尝试索引以下 json（没有换行符，我只是在这里格式化）：

{
  "Timestamp": "19:51:27.757",
  "Level": "INFO",
  "EventType": "Audit",
  "EventId": "ApiServiceInvocationResponse",
  "ThreadId": "19",
  "Method": "TXXXX1234.Common.WCF.ParameterInspector.AfterCall",
  "Context": {
    "PhoneNumber": "48600000000",
    "ApplicationId": "7C217CF0CC45E0292623203E56AD87EC",
    "ApiType": "android",
    "ApiVersion": "6.0",
    "AppVersion": "1.0.debug",
    "UserId": 25714,
    "SessionId": 1440538,
    "CorrelationId": "98ccaec5-4d23-4c5f-b5da-7ce0e440f2e3"
  },
  "Payload": {
    "Operation": "Initialize",
    "Response": {
      "Message": null,
      "CanRun": true,
      "PhoneNumber": null,
      "DefaultPhoneNumber": "48600000000",
      "DriverPhoneNumber": null,
      "RegisterationPhoneNumber": null,
      "Favourites": null,
      "SessionId": "4DC24EB6E4B0261DD03CDD4F6A7C7DC8",
      "IsFriendlyCustomer": true,
      "OptionsAvailable": [],
      "MaxOrderDate": null,
      "FavouriteDriverNumber": null,
      "ShareMessage": null,
      "PaymentInstruments": [],
      "InAppPaymentAvailable": true,
      "HasActiveOrders": false,
      "UserName": "some name",
      "UserPhone": "48600000000",
      "ApplicationId": "",
      "KioskInfo": null,
      "CallResult": {
        "Code": "SSREA",
        "Message": null
      }
    },
    "truncate": false
  },
  "Message": null,
  "Exception": null
}

它被正确索引。但是以下一个没有被索引：

{
  "Timestamp": "16:31:27.074",
  "Level": "INFO",
  "EventType": "Audit",
  "EventId": "ApiServiceInvocationResponse",
  "ThreadId": "5",
  "Method": "TXXXX1234.Common.WCF.ParameterInspector.AfterCall",
  "Context": {
    "PhoneNumber": "48600000000",
    "ApplicationId": "A70BAFD855CE7120A8E331E27D39E645",
    "ApiType": "MOCK",
    "ApiVersion": "1.0",
    "AppVersion": null,
    "UserId": 11852,
    "SessionId": 448107,
    "CorrelationId": "28d9cc6f-c207-4199-9c24-ac6c4b4cfc8e"
  },
  "Payload": {
    "Operation": "Initialize",
    "Response": {
      "Message": "message",
      "CanRun": false,
      "PhoneNumber": "48600000000",
      "DefaultPhoneNumber": "48600000000",
      "DriverPhoneNumber": "",
      "RegisterationPhoneNumber": null,
      "Favourites": [],
      "SessionId": "0778662D04444C9456694B3FAB44F8C6",
      "IsFriendlyCustomer": true,
      "OptionsAvailable": [
        "PaymentCard",
        "Combi",
        "SevenSeats",
        "Animal",
        "AirContition"
      ],
      "MaxOrderDate": "2019-01-30 16:31",
      "FavouriteDriverNumber": null,
      "ShareMessage": "some long share message. http://www.sharing.net.pl/",
      "PaymentInstruments": [],
      "InAppPaymentAvailable": false,
      "HasActiveOrders": false,
      "UserName": "some name",
      "UserPhone": "48600000000",
      "ApplicationId": "",
      "KioskInfo": null,
      "CallResult": {
        "Code": "SSREA",
        "Message": "Zwr???? klucz sesji dla zarejestrowanego uzytkownika"
      }
    },
    "truncate": false
  },
  "Message": null,
  "Exception": null
}

更新：这是我在日志中发现的： 01-02-2019 20:40:31.780 +0100 错误 JsonLineBreaker - JSON StreamId:9928927958268928125 解析错误：解析反斜杠转义时出现意外字符：'x' - data_source="C:\Logs\Txxx.log", data_host=" WIN-BP2MBISNI04", data_sourcetype="my_json"

【问题讨论】：

TIME_FORMAT 设置不正确。它应该是“%H:%M:%S.%3N”。此外，您应该添加TIME_PREFIX = "TIMESTAMP": "。然而，更可能导致问题的是“zwr”之后的消息字段中的非 ASCII 字符。 Splunk 要求所有字符都是 UTF-8。
时间格式在这里应该不是问题，因为在两种情况下都是相同的。事实上，删除这个非 ascii 字符有帮助 - 谢谢！无论如何，这个字符是 utf-8 编码的，那么为什么 splunk 不想索引它呢？

标签： json splunk

【解决方案1】：

尝试将 CHARSET = UTF-8 添加到 props.conf 节。

【讨论】：