spark grep 工具问题

Question

我正在 pyspark 中编写一个 grep 工具，它在命令行上获取一个单词并搜索一个文本文件并返回包含命令行中给出的单词的任何行。我的搜索返回不是搜索词的行

#!/usr/bin/python

    import sys
    from pyspark import SparkContext

    def search_word(word):
            if (word)  != -1:
                    print ('%s\t%s' % ( word, word.strip() ))



    # assign search word given on command line
    if len(sys.argv) > 1:
           word = sys.argv[1]


         sc = SparkContext()
         textRDD = sc.textFile("input.txt")
         textRDD = textRDD.map(lambda word: word.replace(',',' ').replace('.',' '). lower())
         textRDD = textRDD.flatMap(lambda word: word.split())
         textRDD = textRDD.filter(lambda word: search_word(word))
         firstten = textRDD.take(10)
         print(firstten)

命令行示例：spark-submit 自己

示例文本文件：

Ere quitting, for the nonce, the Sperm Whale's head, I would have
you, as a sensible physiologist, simply--particularly remark its front
aspect, in all its compacted collectedness. I would have you investigate
it now with the sole view of forming to yourself some unexaggerated,
intelligent estimate of whatever battering-ram power may be lodged
there. Here is a vital point; for you must either satisfactorily settle
this matter with yourself, or for ever remain an infidel as to one of
the most appalling, but not the less true events, perhaps anywhere to be found in all recorded history.

预期结果：

yourself --  it now with the sole view of forming to yourself some unexaggerated

上面的代码返回这个：

produce produce
our our
new new
ebooks  ebooks

Answer 1

在数据和结果方面不太确定您的示例，但据我所知，我认为不需要 flatMap 或拆分。

这里是单一的 grep 值方法，只有几行代码：

import pyspark.sql.functions as f
df = spark.read.text("/FileStore/tables/sample_text.txt").toDF("text_string")
df.show(100, truncate=False)
grep_val = 'ZZZ'
df.where(df.text_string.contains(grep_val)).show(100, truncate=False)

返回：

+-------------------------+
|text_string              |
+-------------------------+
|Hi how are you today ZZZ |
|I am fine                |
|I am also tired          |
|You look good            |
|Can I stay with you?     |
|Bob will pop in later ZZZ|
|Oh really? Nice, cool    |
+-------------------------+

+-------------------------+
|text_string              |
+-------------------------+
|Hi how are you today ZZZ |
|Bob will pop in later ZZZ|
+-------------------------+

您最好遵循使用 grep_list、rlike 和 JOIN 的标准方法。请参阅PySpark: Search For substrings in text and subset dataframe，以更灵活的方式获取一般指导。