uvicorn 在 AWS Fargate 上 1-2 分钟后关闭答案

【问题标题】：uvicorn shutting down after 1-2 minutes on AWS Fargateuvicorn 在 AWS Fargate 上 1-2 分钟后关闭
【发布时间】：2022-10-07 02:59:44
【问题描述】：

我在 AWS Fargate 上使用 Python 3.10.1 和 Application Load Balancer 部署了 FastAPI 0.81.0 + uvicorn 0.18.3。服务器在我的本地 Docker 中无限期地运行（如预期的那样），但是在 AWS 上，应用程序总是在 1-2 分钟后关闭。

这是 Docker 中的 uvicorn 调用：

CMD [\"uvicorn\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\", \"--log-level\", \"trace\", \"app.main:app\"]

我的 FastAPI 应用程序如下所示：

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware

app = FastAPI()

app = FastAPI()
origins = [
    \"*\"
]

app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=[\"*\"],
    allow_headers=[\"*\"],
)

@app.get(\"/\")
async def root():
    return {\"Hello\": \"World\"}

这可能与负载均衡器有关，因为我的 Fargate 服务的 RAM 使用率并不太高：

通常的嫌疑人似乎是health checks via TCP instead of HTTP，但是在 Fargate 任务定义或 EC2 目标组中，默认情况下，AFAIK 已经分别通过 HTTP 进行健康检查。

这是我的 Fargate 任务的日志：

2022-09-22 18:43:46 INFO: Finished server process [1]
2022-09-22 18:43:46 INFO: Waiting for application shutdown.
2022-09-22 18:43:46 TRACE: ASGI [1] Receive {\'type\': \'lifespan.shutdown\'}
2022-09-22 18:43:46 TRACE: ASGI [1] Send {\'type\': \'lifespan.shutdown.complete\'}
2022-09-22 18:43:46 TRACE: ASGI [1] Completed
2022-09-22 18:43:46 INFO: Application shutdown complete.
2022-09-22 18:43:45 INFO: Shutting down
2022-09-22 18:43:39 TRACE: 172.31.21.3:16662 - ASGI [7] Send {\'type\': \'http.response.body\', \'body\': \'<17 bytes>\'}
2022-09-22 18:43:39 TRACE: 172.31.21.3:16662 - ASGI [7] Completed
2022-09-22 18:43:39 TRACE: 172.31.21.3:16662 - HTTP connection lost
2022-09-22 18:43:39 INFO: 172.31.21.3:16662 - \"GET / HTTP/1.1\" 200 OK
2022-09-22 18:43:39 TRACE: 172.31.21.3:16662 - HTTP connection made
2022-09-22 18:43:39 TRACE: 172.31.21.3:16662 - ASGI [7] Started scope={\'type\': \'http\', \'asgi\': {\'version\': \'3.0\', \'spec_version\': \'2.3\'}, \'http_version\': \'1.1\', \'server\': (\'172.31.30.157\', 8000), \'client\': (\'172.31.21.3\', 16662), \'scheme\': \'http\', \'method\': \'GET\', \'root_path\': \'\', \'path\': \'/\', \'raw_path\': b\'/\', \'query_string\': b\'\', \'headers\': \'<...>\'}
2022-09-22 18:43:39 TRACE: 172.31.21.3:16662 - ASGI [7] Send {\'type\': \'http.response.start\', \'status\': 200, \'headers\': \'<...>\'}
2022-09-22 18:43:39 INFO: 172.31.47.71:3856 - \"GET / HTTP/1.1\" 200 OK
2022-09-22 18:43:39 TRACE: 172.31.47.71:3856 - ASGI [6] Send {\'type\': \'http.response.body\', \'body\': \'<17 bytes>\'}
2022-09-22 18:43:39 TRACE: 172.31.47.71:3856 - ASGI [6] Completed
2022-09-22 18:43:39 TRACE: 172.31.47.71:3856 - HTTP connection lost
2022-09-22 18:43:39 TRACE: 172.31.47.71:3856 - HTTP connection made
2022-09-22 18:43:39 TRACE: 172.31.47.71:3856 - ASGI [6] Started scope={\'type\': \'http\', \'asgi\': {\'version\': \'3.0\', \'spec_version\': \'2.3\'}, \'http_version\': \'1.1\', \'server\': (\'172.31.30.157\', 8000), \'client\': (\'172.31.47.71\', 3856), \'scheme\': \'http\', \'method\': \'GET\', \'root_path\': \'\', \'path\': \'/\', \'raw_path\': b\'/\', \'query_string\': b\'\', \'headers\': \'<...>\'}
2022-09-22 18:43:39 TRACE: 172.31.47.71:3856 - ASGI [6] Send {\'type\': \'http.response.start\', \'status\': 200, \'headers\': \'<...>\'}
2022-09-22 18:43:09 TRACE: 172.31.21.3:39448 - ASGI [5] Completed
2022-09-22 18:43:09 TRACE: 172.31.21.3:39448 - HTTP connection lost
2022-09-22 18:43:09 TRACE: 172.31.21.3:39448 - ASGI [5] Send {\'type\': \'http.response.body\', \'body\': \'<17 bytes>\'}
2022-09-22 18:43:09 TRACE: 172.31.21.3:39448 - HTTP connection made
2022-09-22 18:43:09 TRACE: 172.31.21.3:39448 - ASGI [5] Started scope={\'type\': \'http\', \'asgi\': {\'version\': \'3.0\', \'spec_version\': \'2.3\'}, \'http_version\': \'1.1\', \'server\': (\'172.31.30.157\', 8000), \'client\': (\'172.31.21.3\', 39448), \'scheme\': \'http\', \'method\': \'GET\', \'root_path\': \'\', \'path\': \'/\', \'raw_path\': b\'/\', \'query_string\': b\'\', \'headers\': \'<...>\'}
2022-09-22 18:43:09 TRACE: 172.31.21.3:39448 - ASGI [5] Send {\'type\': \'http.response.start\', \'status\': 200, \'headers\': \'<...>\'}
2022-09-22 18:43:09 INFO: 172.31.21.3:39448 - \"GET / HTTP/1.1\" 200 OK
2022-09-22 18:43:09 TRACE: 172.31.47.71:50778 - ASGI [4] Completed
2022-09-22 18:43:09 TRACE: 172.31.47.71:50778 - HTTP connection lost
2022-09-22 18:43:09 TRACE: 172.31.47.71:50778 - ASGI [4] Send {\'type\': \'http.response.start\', \'status\': 200, \'headers\': \'<...>\'}
2022-09-22 18:43:09 INFO: 172.31.47.71:50778 - \"GET / HTTP/1.1\" 200 OK
2022-09-22 18:43:09 TRACE: 172.31.47.71:50778 - ASGI [4] Send {\'type\': \'http.response.body\', \'body\': \'<17 bytes>\'}
2022-09-22 18:43:09 TRACE: 172.31.47.71:50778 - HTTP connection made
2022-09-22 18:43:09 TRACE: 172.31.47.71:50778 - ASGI [4] Started scope={\'type\': \'http\', \'asgi\': {\'version\': \'3.0\', \'spec_version\': \'2.3\'}, \'http_version\': \'1.1\', \'server\': (\'172.31.30.157\', 8000), \'client\': (\'172.31.47.71\', 50778), \'scheme\': \'http\', \'method\': \'GET\', \'root_path\': \'\', \'path\': \'/\', \'raw_path\': b\'/\', \'query_string\': b\'\', \'headers\': \'<...>\'}
2022-09-22 18:42:39 INFO: 172.31.47.71:55984 - \"GET / HTTP/1.1\" 200 OK
2022-09-22 18:42:39 TRACE: 172.31.47.71:55984 - ASGI [3] Send {\'type\': \'http.response.start\', \'status\': 200, \'headers\': \'<...>\'}
2022-09-22 18:42:39 TRACE: 172.31.47.71:55984 - ASGI [3] Send {\'type\': \'http.response.body\', \'body\': \'<17 bytes>\'}
2022-09-22 18:42:39 TRACE: 172.31.47.71:55984 - ASGI [3] Completed
2022-09-22 18:42:39 TRACE: 172.31.47.71:55984 - HTTP connection lost
2022-09-22 18:42:39 TRACE: 172.31.21.3:59240 - HTTP connection lost
2022-09-22 18:42:39 TRACE: 172.31.47.71:55984 - ASGI [3] Started scope={\'type\': \'http\', \'asgi\': {\'version\': \'3.0\', \'spec_version\': \'2.3\'}, \'http_version\': \'1.1\', \'server\': (\'172.31.30.157\', 8000), \'client\': (\'172.31.47.71\', 55984), \'scheme\': \'http\', \'method\': \'GET\', \'root_path\': \'\', \'path\': \'/\', \'raw_path\': b\'/\', \'query_string\': b\'\', \'headers\': \'<...>\'}
2022-09-22 18:42:39 TRACE: 172.31.21.3:59240 - ASGI [2] Completed
2022-09-22 18:42:39 TRACE: 172.31.47.71:55984 - HTTP connection made
2022-09-22 18:42:39 INFO: 172.31.21.3:59240 - \"GET / HTTP/1.1\" 200 OK
2022-09-22 18:42:39 TRACE: 172.31.21.3:59240 - ASGI [2] Send {\'type\': \'http.response.body\', \'body\': \'<17 bytes>\'}
2022-09-22 18:42:39 TRACE: 172.31.21.3:59240 - ASGI [2] Started scope={\'type\': \'http\', \'asgi\': {\'version\': \'3.0\', \'spec_version\': \'2.3\'}, \'http_version\': \'1.1\', \'server\': (\'172.31.30.157\', 8000), \'client\': (\'172.31.21.3\', 59240), \'scheme\': \'http\', \'method\': \'GET\', \'root_path\': \'\', \'path\': \'/\', \'raw_path\': b\'/\', \'query_string\': b\'\', \'headers\': \'<...>\'}
2022-09-22 18:42:39 TRACE: 172.31.21.3:59240 - ASGI [2] Send {\'type\': \'http.response.start\', \'status\': 200, \'headers\': \'<...>\'}
2022-09-22 18:42:39 TRACE: 172.31.21.3:59240 - HTTP connection made
2022-09-22 18:42:30 INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2022-09-22 18:42:30 INFO: Waiting for application startup.
2022-09-22 18:42:30 TRACE: ASGI [1] Started scope={\'type\': \'lifespan\', \'asgi\': {\'version\': \'3.0\', \'spec_version\': \'2.0\'}}
2022-09-22 18:42:30 TRACE: ASGI [1] Receive {\'type\': \'lifespan.startup\'}
2022-09-22 18:42:30 TRACE: ASGI [1] Send {\'type\': \'lifespan.startup.complete\'}
2022-09-22 18:42:30 INFO: Application startup complete.
2022-09-22 18:42:30 INFO: Started server process [1]

关于如何解决这个问题的任何建议？谢谢！

您介意提供您的 python 版本和完整的 Dockerbuild 文件吗？
我在python:3.10.1 下运行它。 Dockerbuild 文件还仅包含 pip 和诗歌安装。
是的。最大值永远不会超过 30%。

标签： amazon-web-services fastapi aws-fargate aws-application-load-balancer uvicorn

【解决方案1】：

我们之前遇到过这个问题，需要为任务添加更多 RAM。更多的CPU也不会受到伤害。尝试为您的任务提供更多内存，看看问题是否仍然存在。如果它再次工作，那么只要它仍然有效，就缩小内存。

编辑我问了我团队的一位开发人员，他说如果应用程序长时间处于空闲状态它将关闭，您可以配置一个空闲时间，您可以设置此属性--timeout。

gunicorn \
    --log-config 'logging.conf'
    --timeout 6000

【讨论】：

谢谢你的建议。我用 8GB RAM 和更多 CPU 再次检查：最大利用率分别为 6% 和 3%。行为没有改变。
--timeout 标志行为不应该与--timeout-keep-alive 具有相同的目的吗？如果是这样，不幸的是它也无济于事......

【解决方案2】：

您是否尝试过按照您链接的其他答案中的建议调整 uvicorn keep alive 值？

CMD ["uvicorn", "--host", "0.0.0.0", "--port", "8000", "--log-level", "trace",  "--timeout-keep-alive", "65", "app.main:app"]

【讨论】：

感谢您的回答。是的，我试过了；即使--timeout-keep-alive 1000 没有任何改变。

【解决方案3】：

原来这是一个纯粹的愚蠢问题。

简单的问题是未通过健康检查任务定义.虽然健康检查在目标组由于端口映射正确，因此完美解决，在任务定义缺少端口 8000：

CMD-SHELL curl -f http://0.0.0.0:8000 || exit 1

我一直认为这是相反的方式，uvicorn 关闭导致它们失败。感谢您的所有建议和耐心！

【讨论】：