并行 Pytest 夹具处理

Question

我正在使用 PyTest，假设我有一个名为 test_func 的测试函数，它有两个名为 docker_mysql 和 docker_clickhouse 的参数，它们是 PyTest 术语中的函数夹具。在这两个函数中，我正在拉取 docker 映像并启动它。由于fixture的机制，test_func会一步步调用fixture，所以第二个docker容器只有在第一个启动后才会运行。

所以，问题是：如何并行化拉取 docker 容器的过程以节省时间，因为在不久的将来等待队列中的许多 docker 容器会很烦人。

就个人而言，我尝试过pytest-xdist，但在 CI-CD 中我没有提到速度的任何变化。另外，我尝试制作可以使用from threading import Thread 执行线程的夹具，但我崩溃了。

Answer 1

我使用 Thread 进行了尝试，它成功了。

from datetime import datetime
from queue import Queue
from threading import Thread
import time

import pytest


def _pull_and_start_docker_mysql():
    print(datetime.now(), "Start downloading docker_mysql")
    time.sleep(5)  # Simulate download
    print(datetime.now(), "Finished downloading docker_mysql")
    return "MySQL"

def _pull_and_start_docker_clickhouse():
    print(datetime.now(), "Start downloading docker_clickhouse")
    time.sleep(8)  # Simulate download
    print(datetime.now(), "Finished downloading docker_clickhouse")
    return "Clickhouse"


@pytest.fixture(scope="session")
def docker():
    results = Queue()  # Needed to store the results of the threads
    p1 = Thread(target=lambda: results.put(("docker_mysql", _pull_and_start_docker_mysql())))  # To identify the result, we put an identifier name "docker_mysql"
    p2 = Thread(target=lambda: results.put(("docker_clickhouse", _pull_and_start_docker_clickhouse())))  # To identify the result, we put an identifier name "docker_clickhouse"
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    return [results.get() for _ in range(results.qsize())]


def test_func(docker):
    print("Inside test_func", docker)
    assert True


def test_func2(docker):
    print("Inside test_func2", docker)
    assert True

$ pytest -rP
=========================================================================================== test session starts ===========================================================================================
collected 2 items                                                                                                                                                                                         

test_docker.py ..                                                                                                                                                                                    [100%]

================================================================================================= PASSES ==================================================================================================
________________________________________________________________________________________________ test_func ________________________________________________________________________________________________
------------------------------------------------------------------------------------------ Captured stdout setup ------------------------------------------------------------------------------------------
2021-08-17 01:26:21.113417 Start downloading docker_mysql
2021-08-17 01:26:21.113675 Start downloading docker_clickhouse
2021-08-17 01:26:26.125081 Finished downloading docker_mysql
2021-08-17 01:26:29.122935 Finished downloading docker_clickhouse
------------------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------------------
Inside test_func [('docker_mysql', 'MySQL'), ('docker_clickhouse', 'Clickhouse')]
_______________________________________________________________________________________________ test_func2 ________________________________________________________________________________________________
------------------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------------------
Inside test_func2 [('docker_mysql', 'MySQL'), ('docker_clickhouse', 'Clickhouse')]
============================================================================================ 2 passed in 8.04s ============================================================================================

一些注意事项：

1. 从日志中可以看出，docker_mysql 和docker_clickhouse 的进程同时开始于 01:26:21，并分别在 5 秒和 8 秒的休眠后结束（模拟不同的执行时间）。
2. 我使用了"session" 范围，如记录的here 那样，它不会为每个测试重新执行夹具，在运行test_func 和test_func2 后在日志中可见。检查这是否也适用于您的场景。
3. 目前，夹具响应的格式是 的列表，例如[('docker_mysql', 'MySQL'), ...]。您可能需要重新格式化它，例如根据您的用例进行口述。