FastAPI/Unvicorn 请求在中间件调用 call_next（路径操作函数）期间挂起

Question

我们在 EC2 上运行的 docker 容器内有一个机器学习模型。

我们使用 Cortex.dev 来自动扩展 GPU。

不确定地，请求将在 FastAPI 中间件中的 call_next 函数期间挂起。不幸的是，它无法重现。

Middleware pre-request 打印行被记录，但路径操作函数中的第一个打印语句从未被记录。

我们尝试过的事情：

• 使用 1 名工人运行 Uvicorn
• 在没有异步的情况下运行 run 函数
• 以bytes 作为image 的参数类型而不是UploadFile 运行

这些更改都不能解决挂起的问题，但这是性能最高的配置。

1. 这是否意味着问题出在 FastAPI 而不是 Uvicorn？

2. 如果是，什么会导致 FastAPI 挂起？如果不是，问题出在哪里？如何解决？

Dockerfile

FROM nvidia/cuda:11.4.0-runtime-ubuntu18.04

WORKDIR /usr/src/app

RUN apt-get -y update && \
    apt-get install -y --fix-missing \
    build-essential \
    cmake \
    python3 \
    python3-pip \
    ffmpeg \
    libsm6 \
    libxext6 \
    && apt-get clean && rm -rf /tmp/* /var/tmp/*

ADD ./requirements.txt ./

# install our dependencies
RUN python3 -m pip install --upgrade pip && python3 -m pip install -r requirements.txt && apt-get clean && rm -rf /tmp/* /var/tmp/*

ADD ./ ./

ENV LC_ALL=C.UTF-8 
ENV LANG=C.UTF-8

EXPOSE 8080

CMD uvicorn api:app --host 0.0.0.0 --port 8080 --workers 2

api.py

from my_predictor import PythonPredictor
from typing import Optional
from datetime import datetime
import time
from starlette.responses import Response

from fastapi import FastAPI, File, UploadFile, Form, Response, HTTPException, Request
from fastapi.middleware.cors import CORSMiddleware

app = FastAPI()

origins = ["*"]

app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)


@app.middleware("http")
async def add_process_time_header(request: Request, call_next):
    cortex_id = request.headers.get('x-request-id')
    start_time = time.time()
    print("Cortex ID: " + cortex_id + ". > Middleware pre-request. Time stamp: " + str(start_time), flush=True)

    response = await call_next(request)

    process_time = time.time() - start_time
    print("Cortex ID: " + cortex_id + ". > Middleware post-response. Duration: " + str(process_time), flush=True)

    return response



@app.post("/")
async def run(request: Request, image: UploadFile = File(...), renderFactor:Optional[int] = Form(12), requestId:Optional[str] = Form('-1'),include_header:Optional[str] = Form('bin')):

    try:
        cortexId = request.headers.get('x-request-id')
        print("Cortex ID: " + cortexId + ". Request ID: " + requestId + " >>> Request received. Time stamp: " + str(datetime.now()))

        start = time.time()
    
        image = await image.read()

        payload = {}
        payload['image'] = image
        payload['renderFactor'] = renderFactor
        payload['requestId'] = requestId
        payload['include_header'] = include_header
        
        response = pred.predict(payload)

        end = time.time()

        totalTime = round(end - start, 2)

        print("Cortex ID: " + cortexId + ". Request ID: " + requestId + " > Request processed. Duration: " + str(totalTime) + " seconds. Time stamp: " + str(datetime.now()))

        if totalTime > 5:
            print("Long request detected. Duration: " + str(totalTime))

        return response
        
    except Exception as error:
        end = time.time()
        print(str(error))
        print("Cortex ID: " + cortexId + ". Request ID: " + requestId + " > Error. Duration: " + str(round(end - start, 2)) + " seconds . Time stamp: " + str(datetime.now()))

        raise HTTPException(status_code = 500, detail = str(error))

config = {}
pred = PythonPredictor(config)

Answer 1

解决方案 1

从您的代码中，我可以看到您使用了来自 starlette.responses 包和 FastAPI 的响应包的响应，这可能导致了挂起问题

from starlette.responses import Response

# remove the Response from fastapi
from fastapi import FastAPI, File, UploadFile, Form, HTTPException, Request

解决方案 2

如果您的问题仍然存在

FastAPI documentation 声明从fastapi 包导入的Request 理想情况下来自Starlette(Starlette Requests documentation link)

您也可以使用from starlette.requests import Request。 FastAPI 为开发人员提供了便利。但它直接来自Starlette。

将from fastapi import Request替换为from starlette.requests import Request

在this link 的官方 FastAPI github 问题中存在类似问题，该应用程序是使用 uvicorn <file>:app 运行的。 下面用starlette.requests直接实现的代码块没有产生挂起的问题，说明这个问题是由FastAPI引起的。

from starlette.applications import Starlette
from starlette.requests import Request
from starlette.responses import JSONResponse

app = Starlette()


@app.middleware("http")
async def func(request: Request, call_next):
    #print(await request.json())
    return await call_next(request)


@app.route('/', methods=["POST"])
def homepage(request):
    return JSONResponse({"Hello": "World"})

确保在您的代码中使用 starlette.requests 和 starlette.responses，如下所示

from starlette.responses import Response
from starlette.requests import Request

# Request and Response removed from fastapi as directly referred from starlette
from fastapi import FastAPI, File, UploadFile, Form, HTTPException 
from fastapi.middleware.cors import CORSMiddleware

app = FastAPI()

origins = ["*"]

app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)


@app.middleware("http")
async def add_process_time_header(request: Request, call_next):
    cortex_id = request.headers.get('x-request-id')
    start_time = time.time()
    print("Cortex ID: " + cortex_id + ". > Middleware pre-request. Time stamp: " + str(start_time), flush=True)

    response = await call_next(request)

    process_time = time.time() - start_time
    print("Cortex ID: " + cortex_id + ". > Middleware post-response. Duration: " + str(process_time), flush=True)

    return response