如何使用 Google Cloud Composer 设置 dbt？

Question

我正在尝试在 Google Cloud Composer 上安装 dbt，但遇到了依赖性问题。我已按照本文中的说明进行操作：https://blog.doit-intl.com/setup-dbt-with-cloud-composer-ab702454e27b 但是在第 2 步：在 composer 中安装软件包（airflow-dbt & dbt）时，它已经失败了。

我在云构建日志中找到以下内容：

ERROR: snowflake-connector-python 2.3.6 has requirement boto3<1.16,>=1.4.4, but you'll have boto3 1.17.85 which is incompatible.
ERROR: snowflake-connector-python 2.3.6 has requirement requests<2.24.0, but you'll have requests 2.24.0 which is incompatible.
ERROR: networkx 2.5.1 has requirement decorator<5,>=4.3, but you'll have decorator 5.0.9 which is incompatible.
ERROR: hologram 0.0.13 has requirement jsonschema<3.2,>=3.0, but you'll have jsonschema 3.2.0 which is incompatible.
ERROR: dbt-core 0.19.1 has requirement idna<2.10, but you'll have idna 2.10 which is incompatible.
ERROR: dbt-core 0.19.1 has requirement requests<2.24.0,>=2.18.0, but you'll have requests 2.24.0 which is incompatible.
ERROR: dbt-snowflake 0.19.1 has requirement cryptography<4,>=3.2, but you'll have cryptography 3.0 which is incompatible.
ERROR: dbt-bigquery 0.19.1 has requirement google-api-core<1.24,>=1.16.0, but you'll have google-api-core 1.28.0 which is incompatible.
ERROR: dbt-redshift 0.19.1 has requirement boto3<1.16,>=1.4.4, but you'll have boto3 1.17.85 which is incompatible.

我当前的环境配置包含：composer-1.13.0-airflow-1.10.12

有没有人遇到过同样的问题，你能解决吗？ 我还尝试安装日志中列出的特定版本的要求，但这并不能解决问题。

Answer 1

尝试在 Composer 环境中安装 dbt 时有点痛苦。但是，有一些解决方法。

1. 使用外部服务运行 dbt 作业，例如云跑。
2. 使用 Composer 的 KubernetesPodOperator。我的同事在此处提交了a nice article on dbt discourse 以完成设置过程。
3. 通过将 Composer 的环境变量 IGNORE_PYPI_DEPENDENCY_CONFLICTS 设置为 True 来忽略 Composer 的依赖冲突。但是，我不建议这样做，因为它可能会导致潜在问题。
4. 在 Composer 中创建 Python 虚拟环境并安装 dbt 包。