Google Python 云数据流实例在没有新部署的情况下中断（pubsub 导入失败）

Question

我在 Google AppEngine Flex Env​​ironment 中为 Python 定义了几个不同的 Cloud Dataflow 作业。我在 requirements.txt 文件中定义了我的要求，包括了我的 setup.py 文件，并且一切正常。我的最后一次部署是在 2018 年 5 月 3 日。通过查看日志，我发现我的一个作业在 2018 年 5 月 22 日开始失败。该作业失败并出现了由于导入错误导致的堆栈跟踪，如下所示。

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 582, in do_work
    work_executor.execute()
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 166, in execute
    op.start()
  File "apache_beam/runners/worker/operations.py", line 294, in apache_beam.runners.worker.operations.DoOperation.start (apache_beam/runners/worker/operations.c:10607)
    def start(self):
  File "apache_beam/runners/worker/operations.py", line 295, in apache_beam.runners.worker.operations.DoOperation.start (apache_beam/runners/worker/operations.c:10501)
    with self.scoped_start_state:
  File "apache_beam/runners/worker/operations.py", line 300, in apache_beam.runners.worker.operations.DoOperation.start (apache_beam/runners/worker/operations.c:9702)
    pickler.loads(self.spec.serialized_fn))
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 225, in loads
    return dill.loads(s)
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 277, in loads
    return load(file)
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 266, in load
    obj = pik.load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 423, in find_class
    return StockUnpickler.find_class(self, module, name)
  File "/usr/lib/python2.7/pickle.py", line 1124, in find_class
    __import__(module)
  File "/usr/local/lib/python2.7/dist-packages/dataflow_pipeline/tally_overages.py", line 27, in <module>
    from google.cloud import pubsub
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub.py", line 17, in <module>
    from google.cloud.pubsub_v1 import PublisherClient
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1 import types
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/types.py", line 26, in <module>
    from google.iam.v1.logging import audit_data_pb2
ImportError: No module named logging

所以主要问题似乎来自依赖于导入 google.iam.v1.logging 的 pubsub 依赖项，该依赖项是从 grpc-google-iam-v1 安装的。

这是我的 requirements.txt 文件

Flask==0.12.2
apache-beam[gcp]==2.1.1
gunicorn==19.7.1
google-cloud-dataflow==2.1.1
google-cloud-datastore==1.3.0
pytz
google-cloud-pubsub
google-gax
grpc-google-iam-v1
googleapis-common-protos
google-cloud==0.32
six==1.10.0
protobuf

通过在我的项目中执行以下操作，我可以在本地运行所有内容。

$ virtualenv --no-site-packages .
$ . bin/activate
$ pip install --ignore-installed -r requirements.txt
$ python main.py
No handlers could be found for logger "oauth2client.contrib.multistore_file"
INFO:werkzeug: * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
INFO:werkzeug: * Restarting with stat
No handlers could be found for logger "oauth2client.contrib.multistore_file"
WARNING:werkzeug: * Debugger is active!
INFO:werkzeug: * Debugger PIN: 317-820-645

具体来说，我可以在本地执行以下操作

$ python
>>> from google.cloud import pubsub
>>> import google.iam.v1.logging
>>> google.iam.v1.logging.__file__
'/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/google/iam/v1/logging/__init__.pyc'

所以我知道grpc-google-iam-v1 软件包的安装在本地工作得很好。所需的文件在那里。

我的问题是

1. 为什么在 Google AppEngine Flex Env​​ironment 上安装 grpc-google-iam-v1 没有正确安装所有文件？我一定是错过了/site-packages/google/iam/v1/logging 目录。
2. 为什么这会随机开始失败？我没有再做任何部署，相同的代码在 21 日运行和工作，然后在 5 月 22 日中断。

---

将 requirements.txt 文件更改为

后，我能够让管道再次运行

Flask==0.12.2
apache-beam[gcp]
google-cloud-dataflow
gunicorn==19.7.1
google-cloud-datastore==1.3.0
pytz
google-cloud-pubsub
google-gax
grpc-google-iam-v1
googleapis-common-protos
google-cloud==0.32
six==1.10.0
protobuf

所以只需从apache-beam[gcp] 和google-cloud-dataflow 中删除版本要求就可以了。

Answer 1

基于John Allard 提供的解决方案，requirements.txt 中的removing the version 将自动默认为最新版本。因此，在没有为 apache-beam[gcp] 、 google-cloud-dataflow 和 google-cloud-pubsub 指定版本的情况下，它们都将在最新版本上运行并解决依赖问题。 requirements.txt 将如下所示：

Flask==0.12.2
apache-beam[gcp]
gunicorn==19.7.1
google-cloud-dataflow
google-cloud-datastore==1.3.0
pytz
google-cloud-pubsub
google-gax
grpc-google-iam-v1
googleapis-common-protos
google-cloud==0.32
six==1.10.0
protobuf