Spark 无法识别来自 String 的新行、&amp 等

Question

我正在尝试使用 PySpark 处理文本数据（Twitter 推文）。表情符号和特殊字符正确显示为红色，但“\n”、“&amp”似乎被转义。 Spark 无法识别它们。可能其他人也一样。我的 Spark DF 中的一条推文示例如下所示：

• “大家好\n\n最近怎么样？????保重，好好享受”

我希望 Spark 能够正确读取它们。这些文件存储为镶木地板，我正在阅读它们：

tweets = spark.read.format('parquet')\
.option('header', 'True')\
.option('encoding', 'utf-8')\
.load(path)

以下是我从原始 JSONL 文件中获取的一些示例输入数据（我稍后将数据存储为 parquet）。

• "full_text": "RT @OurWarOnCancer：我们的联邦疫苗接种在哪里 HPV教育运动？！我们的联邦 #lungcancer 在哪里 放映节目？！ (和\u2026"

• "full_text": "\u2b55\ufe0f#HPV是最重要的原因
  #CervicalCancer 但它不仅仅会导致宫颈癌（见图\ud83d\udc47）\n\u2b55\ufe0f这意味着它们可以被预防”

直接从 JSONL 文件中读取会导致相同的识别问题。

tweets = spark.read.\
.option('encoding', 'utf-8')\
.json(path)

Spark 如何正确识别它们？提前谢谢你。

Answer 1

以下代码可能有助于解决您的问题，

输入：

"Hello everyone\n\nHow is it going? ? Take care & enjoy"

"full_text": "RT @OurWarOnCancer: Where is our FEDERAL vaccination education campaign for HPV?! Where is our FEDERAL #lungcancer screening program?! (and\u2026 &"
"full_text": "\u2b55\ufe0f#HPV is the most important cause of #CervicalCancer But it doesn't just cause cervical cancer (see the figure\ud83d\udc47) \n\u2b55\ufe0fThat means they can be PREVENTED @theNCI @NCIprevention @AmericanCancer @cancereu @uicc @IARCWHO @EuropeanCancer @KanserSavascisi @AUTF_DEKANLIK @OncoAlert"

解决问题的代码：

from pyspark.sql.functions import *

df=spark.read.csv("file:///home/sathya/Desktop/stackoverflo/raw-data/input.tweet")

df1=df.withColumn("cleandata",regexp_replace('_c0', '&|\\\\n', ''))
df1.select("cleandata").show(truncate=False)

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|cleandata                                                                                                                                                                                                                                                                                                                    |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|Hello everyoneHow is it going? ? Take care & enjoy                                                                                                                                                                                                                                                                          |
|"full_text": "RT @OurWarOnCancer: Where is our FEDERAL vaccination education campaign for HPV?! Where is our FEDERAL #lungcancer screening program?! (and\u2026 &"                                                                                                                                                           |
|"full_text": "\u2b55\ufe0f#HPV is the most important cause of #CervicalCancer But it doesn't just cause cervical cancer (see the figure\ud83d\udc47) \u2b55\ufe0fThat means they can be PREVENTED @theNCI @NCIprevention @AmericanCancer @cancereu @uicc @IARCWHO @EuropeanCancer @KanserSavascisi @AUTF_DEKANLIK @OncoAlert"|
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+