用于在本地机器上创建 sparksession 的 pytest

Question

我正在尝试在 Windows 本地机器上运行我的 pyspark 代码测试。 Pytest 卡在我在测试代码中创建 SparkSession 的位置。我是否必须在本地机器上安装/配置 Spark 才能使 Pytest 工作。最后，测试将作为 CI/CD 的一部分执行，我是否还必须在构建机器上配置 Spark？我有一个相关的question，但看起来问题不是 Visual Studio Code 而是 pytest（因为我从命令行运行 pytest 时遇到同样的问题）

下面是我的测试代码

# test code

from pyspark.sql import SparkSession, Row, DataFrame

import pytest

def test_poc():
   spark_session = SparkSession.builder.master('local[2]').getOrCreate()  #this line never returns when debugging test.
   spark_session.createDataFrame(data,schema) #data and schema not shown here.

Answer 1

你能添加你的 pyspark 脚本的终端输出吗？了解从哪里开始会很有帮助，它可能会为我们提供线索，了解您的设置中存在什么问题。

至少看看你是否正确安装了pyspark（你可能还需要做额外的操作才能完全确定），但你可以像下面保存在python文件中的脚本sample_test.py

from pyspark import sql


spark = sql.SparkSession.builder \
        .appName("local-spark-session") \
        .getOrCreate()
        

运行它应该会打印出如下内容

C:\Users\user\Desktop>python sample_test.py
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

C:\Users\user\Desktop>SUCCESS: The process with PID 16368 (child process of PID 12664) has been terminated.
SUCCESS: The process with PID 12664 (child process of PID 11736) has been terminated.
SUCCESS: The process with PID 11736 (child process of PID 6800) has been terminated.

下面是使用 pytest 对 pyspark 进行的示例测试，保存在名为 sample_test.py 的文件中

from pyspark import sql


spark = sql.SparkSession.builder \
        .appName("local-spark-session") \
        .getOrCreate()
        

def test_create_session():
    assert isinstance(spark, sql.SparkSession) == True
    assert spark.sparkContext.appName == 'local-spark-session'
    assert spark.version == '3.1.2'

你可以简单地运行如下

C:\Users\user\Desktop>pytest -v sample_test.py
============================================= test session starts =============================================
platform win32 -- Python 3.6.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- c:\users\user\appdata\local\programs\python\python36\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\user\Desktop
collected 1 item

sample_test.py::test_create_session PASSED                                                               [100%]

============================================== 1 passed in 4.51s ==============================================

C:\Users\user\Desktop>SUCCESS: The process with PID 4752 (child process of PID 9780) has been terminated.
SUCCESS: The process with PID 9780 (child process of PID 8988) has been terminated.
SUCCESS: The process with PID 8988 (child process of PID 20176) has been terminated.

以上示例适用于 Windows。我的帐户是新帐户，因此无法在您的 cmets 上回复...您能否更新您的问题以分享来自终端的消息/错误（如果有）？顺便问一下，我想知道什么操作系统你在用吗？