如何从包含带空格的字符串的文本文件中创建列？

Question

我想在 pyspark 中读取一个文件并从中创建一个数据框。它是一个 tsv 文件，其值格式为：

2015-07-22T09:00:28.019143Z "strings with space" "strings with space" ECDHE THRGH

我尝试创建行对象并将其转换为数据框。但我对如何拆分数据和创建列感到困惑。数据包含带空格的字符串。

from pyspark.sql import Row
from pyspark.sql import SparkSession
import re

spark = SparkSession \
    .builder \
    .appName("Task1") \
    .getOrCreate()

sc = spark.sparkContext

# Load a text file and convert each line to a Row.
lines = sc.textFile("file.tsv")
#parts = lines.map(lambda l: l.split(" ")) -- Tried this but it doesn't give correct output as there are spaces in strings.
values = lines.map(lambda p: Row(value=re.findall(r'\"(.+?)\"', str(p))[0].replace(" ","")))

理想的结果是如下的数据框：

Timestamp                          val1              val2      
 val3       val4
2015-07-22T09:00:28.019143Z strings with space strings with space ECDHE THRGH

Answer 1

要将任何分隔文件（asv）加载到 spark 中，您只需要提及分隔符即可。

df = spark.read.load("examples/src/main/resources/sample.txt",
                 format="csv", sep=" ", inferSchema="true", header="true")

在上述行中使用 sep 属性中的空格。

已编辑：

如果文件没有任何标题，请尝试以下更改列名。同时标记 header="false"

df = spark.read.load("examples/src/main/resources/sample.txt",
                 format="csv", sep=" ", inferSchema="true", header="false").toDF("id","name", "salary")

Answer 2

这是我的 tsv 文件，但实际上是用空格分隔的。

timestamp col1 col2 col3 col4
2015-07-22T09:00:28.019143Z "strings with space" "strings with space" ECDHE THRGH
2015-07-22T09:00:28.019143Z "strings with space" "strings with space" ECDHE THRGH
2015-07-22T09:00:28.019143Z "strings with space" "strings with space" ECDHE THRGH
2015-07-22T09:00:28.019143Z "strings with space" "strings with space" ECDHE THRGH
2015-07-22T09:00:28.019143Z "strings with space" "strings with space" ECDHE THRGH

用我下面的代码，

df = spark.read.option("delimiter", " ").option("header", "true").csv("path/to/tsv")
df.show(7, False)

结果如下图。

+---------------------------+------------------+------------------+-----+-----+
|timestamp                  |col1              |col2              |col3 |col4 |
+---------------------------+------------------+------------------+-----+-----+
|2015-07-22T09:00:28.019143Z|strings with space|strings with space|ECDHE|THRGH|
|2015-07-22T09:00:28.019143Z|strings with space|strings with space|ECDHE|THRGH|
|2015-07-22T09:00:28.019143Z|strings with space|strings with space|ECDHE|THRGH|
|2015-07-22T09:00:28.019143Z|strings with space|strings with space|ECDHE|THRGH|
|2015-07-22T09:00:28.019143Z|strings with space|strings with space|ECDHE|THRGH|
|2015-07-22T09:00:28.019143Z|strings with space|strings with space|ECDHE|THRGH|
|2015-07-22T09:00:28.019143Z|strings with space|strings with space|ECDHE|THRGH|
+---------------------------+------------------+------------------+-----+-----+

看起来效果不错。