【问题标题】:Built in exponential backoff for python api client?为python api客户端内置指数退避?
【发布时间】:2020-01-09 06:16:40
【问题描述】:

我注意到http.py 中的许多方法都支持 num_retries,我认为这是指数退避的一种实现(以防 API 返回错误)。每次调用 .execute() 方法时都想使用它而不是编写自己的退避算法。但是,它似乎不适用于此代码。有人对以可重用的方式处理此退避有一个方便的想法吗?

from apiclient.discovery import build

import google.auth
from google.auth.transport import requests
from google.auth import iam
from google.oauth2 import service_account

TOKEN_URI = 'https://accounts.google.com/o/oauth2/token'
ADMIN_DIRECTORY_SCOPES = [
    'https://www.googleapis.com/auth/admin.directory.user.readonly'
]
ADMIN_USER = 'my_gsuite_admin@domain.com'

def delegated_credentials(credentials, subject, scopes):
    try:
        admin_creds = credentials.with_subject(subject).with_scopes(scopes)
    except AttributeError:
        request = requests.Request()
        credentials.refresh(request)

        signer = iam.Signer(request, credentials, credentials.service_account_email)
        admin_creds = service_account.Credentials(
            signer, credentials.service_account_email, TOKEN_URI,
            scopes=scopes, subject=subject
        )
    except Exception:
        raise
    return admin_creds
print('Setting up auth')
default_credentials, _ = google.auth.default()
admin_creds = delegated_credentials(
    default_credentials, ADMIN_USER, ADMIN_DIRECTORY_SCOPES
)

directory_service = build('admin', 'directory_v1', credentials=admin_creds)

def get_ou(email):
    response = directory_service.users().get(
        userKey=email,
        fields='primaryEmail,orgUnitPath',
        num_retries=5
    ).execute()
    print('Directory.users.get: {}'.format(response))
    return response['orgUnitPath']

Traceback(最近一次调用最后一次):文件“”,第 1 行,in 文件 "/Users/mryerse001/Documents/GitHub/updateEmail_to_OU_Mappings/main.py", 第 122 行,在 get_pubsub_messages 中 成功 = get_ou(电子邮件)文件“/Users/mryerse001/Documents/GitHub/updateEmail_to_OU_Mappings/main.py”, 第 87 行,在 get_ou num_retries=5 文件“/Users/mryerse001/Documents/GitHub/updateEmail_to_OU_Mappings/venv/lib/python3.7/site-packages/googleapiclient/discovery.py”, 第 717 行,在方法中 raise TypeError('Got an unexpected keyword argument "%s"' % name) TypeError: Got an unexpected keyword argument "num_retries"

【问题讨论】:

  • 客户端库已经实现了指数退避,为什么还要做两次?
  • 是否有文档描述它是如何实现的?有没有控制权?
  • 可能我不会在源代码中挖掘,看看你是否能找到他们在哪里实现它

标签: python google-api google-admin-sdk google-api-python-client exponential-backoff


【解决方案1】:

您需要将num_retries 参数传递给execute(),而不是get()

来自google-api-python-client代码:

@util.positional(1)
def execute(self, http=None, num_retries=0):
    """Execute the request.

Args:
  http: httplib2.Http, an http object to be used in place of the
        one the HttpRequest request object was constructed with.
  num_retries: Integer, number of times to retry with randomized
        exponential backoff. If all retries fail, the raised HttpError
        represents the last request. If zero (default), we attempt the
        request only once.

【讨论】:

  • 你是对的。尽管我仍然不喜欢此客户端库中内置的重试功能,因为无论错误如何,它都会重试。例如,假设您正在尝试编辑您无权访问的文件 - 一个错误就足够了。我已经开始使用带有身份验证会话的请求。
  • 不应该是这样,或者至少已经好几年没有了。我想知道您是否使用的是非常旧版本的库,或者 Google 是否抛出了意外错误?库应使用“userRateLimitExceeded”或“rateLimitExceeded”原因重试 429、>=500 和 403。我很好奇正在重试哪些不应该出现的错误?如果您启用logging,您将能够将这些重试尝试视为警告。
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 2017-12-04
  • 2019-03-17
  • 1970-01-01
  • 2016-10-22
  • 1970-01-01
  • 2017-11-04
  • 2013-01-19
相关资源
最近更新 更多