如何在 Airflow 的 docker-compose 设置中设置 pgbouncer答案

【问题标题】：How to set up pgbouncer in a docker-compose setup for Airflow如何在 Airflow 的 docker-compose 设置中设置 pgbouncer
【发布时间】：2023-01-24 17:20:42
【问题描述】：

我正在使用 docker-compose 运行分布式 Airflow 设置。服务的主要部分在一台服务器上运行，而 celery worker 在多台服务器上运行。我有几百个任务每五分钟运行一次，我开始用完数据库连接，这是由任务日志中的错误消息指示的。

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "SERVER" (IP), port XXXXX failed: FATAL:  sorry, too many clients already

我将 Postgres 用作 Metastore，并将 max_connections 设置为默认值 100。我不想提高 max_connections 值，因为我认为应该有更好的解决方案。在某些时候，我将每 5 分钟运行数千个任务，并且保证连接数再次用完。所以我将 pgbouncer 添加到我的配置中。

这是我配置 pgbouncer 的方式

  pgbouncer:
    image: "bitnami/pgbouncer:1.16.0"
    restart: always
    environment:
      POSTGRESQL_HOST: "postgres"
      POSTGRESQL_USERNAME: ${POSTGRES_USER}
      POSTGRESQL_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRESQL_PORT: ${PSQL_PORT}
      PGBOUNCER_DATABASE: ${POSTGRES_DB}
      PGBOUNCER_AUTH_TYPE: "trust"
      PGBOUNCER_IGNORE_STARTUP_PARAMETERS: "extra_float_digits"
    ports:
      - '1234:1234'
    depends_on:
      - postgres

pgbouncer 日志如下所示：

pgbouncer 13:29:13.87 
pgbouncer 13:29:13.87 Welcome to the Bitnami pgbouncer container
pgbouncer 13:29:13.87 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-pgbouncer
pgbouncer 13:29:13.87 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-pgbouncer/issues
pgbouncer 13:29:13.88 
pgbouncer 13:29:13.89 INFO  ==> ** Starting PgBouncer setup **
pgbouncer 13:29:13.91 INFO  ==> Validating settings in PGBOUNCER_* env vars...
pgbouncer 13:29:13.91 WARN  ==> You set the environment variable PGBOUNCER_AUTH_TYPE=trust. For safety reasons, do not use this flag in a production environment.
pgbouncer 13:29:13.91 INFO  ==> Initializing PgBouncer...
pgbouncer 13:29:13.92 INFO  ==> Waiting for PostgreSQL backend to be accessible
pgbouncer 13:29:13.92 INFO  ==> Backend postgres:9876 accessible
pgbouncer 13:29:13.93 INFO  ==> Configuring credentials
pgbouncer 13:29:13.93 INFO  ==> Creating configuration file
pgbouncer 13:29:14.06 INFO  ==> Loading custom scripts...
pgbouncer 13:29:14.06 INFO  ==> ** PgBouncer setup finished! **

pgbouncer 13:29:14.08 INFO  ==> ** Starting PgBouncer **
2022-10-25 13:29:14.089 UTC [1] LOG kernel file descriptor limit: 1048576 (hard: 1048576); max_client_conn: 100, max expected fd use: 152
2022-10-25 13:29:14.089 UTC [1] LOG listening on 0.0.0.0:1234
2022-10-25 13:29:14.089 UTC [1] LOG listening on unix:/tmp/.s.PGSQL.1234
2022-10-25 13:29:14.089 UTC [1] LOG process up: PgBouncer 1.16.0, libevent 2.1.8-stable (epoll), adns: c-ares 1.14.0, tls: OpenSSL 1.1.1d  10 Sep 2019
2022-10-25 13:30:14.090 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-25 13:31:14.090 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-25 13:32:14.090 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-25 13:33:14.090 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-25 13:34:14.089 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-25 13:35:14.090 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-25 13:36:14.090 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-25 13:37:14.090 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-25 13:38:14.090 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-25 13:39:14.089 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us

该服务似乎运行正常，但我认为它什么也没做。 Airflow 文档中关于此的信息很少，我不确定要更改什么。

我应该更改我的 docker-compose 文件中的 pgbouncer 设置吗？

我应该更改 AIRFLOW__DATABASE__SQL_ALCHEMY_CONN 变量吗？

更新 1：我为工作节点编辑了 docker-compose.yml，并将 db 端口更改为 pgbouncer 端口。在此之后，我在保镖日志上获得了一些流量。 Airflow 任务已排队，未使用此配置进行处理，因此仍然存在问题。我没有编辑启动网络服务器、调度程序等的 docker-compose yaml，不知道如何。

   AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://<XXX>@${AIRFLOW_WEBSERVER_URL}:${PGBOUNCER_PORT}/airflow
   AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://<XXX>@${AIRFLOW_WEBSERVER_URL}:${PGBOUNCER_PORT}/airflow

更改后的 pgbouncer 日志：

2022-10-26 11:46:22.517 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-26 11:47:22.517 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-26 11:48:22.517 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-26 11:49:22.519 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-26 11:50:22.518 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-26 11:51:22.516 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2022-10-26 11:51:52.356 UTC [1] LOG C-0x5602cf8ab180: <XXX>@<IP:PORT> login attempt: db=airflow user=airflow tls=no
2022-10-26 11:51:52.359 UTC [1] LOG S-0x5602cf8b1f20: <XXX>@<IP:PORT> new connection to server (from <IP:PORT>)
2022-10-26 11:51:52.410 UTC [1] LOG C-0x5602cf8ab180: <XXX>@<IP:PORT> closing because: client close request (age=0s)
2022-10-26 11:51:52.834 UTC [1] LOG C-0x5602cf8ab180: <XXX>@<IP:PORT> login attempt: db=airflow user=airflow tls=no
2022-10-26 11:51:52.845 UTC [1] LOG C-0x5602cf8ab180: <XXX>@<IP:PORT> closing because: client close request (age=0s)
2022-10-26 11:51:56.752 UTC [1] LOG C-0x5602cf8ab180: <XXX>@<IP:PORT> login attempt: db=airflow user=airflow tls=no
2022-10-26 11:51:57.393 UTC [1] LOG C-0x5602cf8ab3b0: <XXX>@<IP:PORT> login attempt: db=airflow user=airflow tls=no
2022-10-26 11:51:57.394 UTC [1] LOG S-0x5602cf8b2150: <XXX>@<IP:PORT> new connection to server (from <IP:PORT>)
2022-10-26 11:51:59.906 UTC [1] LOG C-0x5602cf8ab180: <XXX>@<IP:PORT> closing because: client close request (age=3s)
2022-10-26 11:52:00.642 UTC [1] LOG C-0x5602cf8ab3b0: <XXX>@<IP:PORT> closing because: client close request (age=3s)

【问题讨论】：

您似乎已经向我们展示了您不需要帮助的部分，而没有向我们展示您确实需要帮助的部分。大概您需要更改主机，而不仅仅是端口。但是你也说至少有一些连接在工作，所以.....
当您无法控制客户端时，pgbouncer 可能非常有用。但是，当您确实可以控制池时，最好在客户端内部进行池化。
你可能是对的。我现在意识到，Airflow 中的默认池大小为 128，而 Postgres 的 max_connections 设置为 100。因此，如果我理解正确，这会默认产生问题。我现在将 max_connections 增加到 250。
似乎活动连接的数量永远不会低于 50，因此我需要找到一些方法来检查是否所有这些连接都是必需的，或者某些连接是否未正确关闭。

标签： postgresql docker-compose airflow pgbouncer

【解决方案1】：

我遇到了和你一样的问题。在你的 docker-compose 文件中，你应该有一个像这样的环境变量：

AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow_user:airflow_pass@pgbouncer:6432/airflow_db。

请记住，PGBOUNCER_DATABASE 变量的默认值为"postgres"，它与您尝试连接的"airflow_db" 数据库不同，后者会导致“nodb”错误。要解决此问题，您需要在 docker-compose 文件中设置 PGBOUNCER_DATABASE=airflow_db。

见this。

请注意，用户名 (airflow_user) 和密码 (airflow_pass) 应该是数据库的所有者 (airflow_db)，否则您可能会出错。

【讨论】：