在 Python 中捕获 403 禁止

Question

我有以下函数来运行 BigQuery 数据提取（见下文）。当我发送太多请求时，我收到错误：

google.api_core.exceptions.Forbidden: 403 Exceeded rate limits: too 此 project_and_region 的许多并发查询。更多 信息，见 https://cloud.google.com/bigquery/troubleshooting-errors

我想知道为什么我的代码无法捕捉到 Forbidden 错误，因为我明确编写了捕捉 403 的函数？

---

from google.cloud import bigquery
from google.api_core.exceptions import Forbidden, InternalServerError, ServiceUnavailable

def run_job(query, query_params, attempt_nb=1):

    # Configure
    job_config = bigquery.QueryJobConfig()
    job_config.query_parameters = query_params
    query_job = client.query(
        query,
        # Location must match that of the dataset(s) referenced in the query.
        location='US',
        job_config=job_config)  # API request - starts the query

    # Try to run and transform to DataFrame()
    try:
        df = query_job.to_dataframe()
        assert query_job.state == 'DONE'
        return df

    except Forbidden:
        # Exception mapping a ``403 Forbidden`` response."""
        return retry_job(query, query_params, attempt_nb)

    except InternalServerError:
        # Exception mapping a ``500 Internal Server Error`` response. or a :attr:`grpc.StatusCode.INTERNAL` error."""
        return retry_job(query, query_params, attempt_nb)

    except ServiceUnavailable:
        # Exception mapping a ``503 Service Unavailable`` response or a :attr:`grpc.StatusCode.UNAVAILABLE` error."""
        return retry_job(query, query_params, attempt_nb)


def retry_job(query, query_params, attempt_nb):
    # If the error is a rate limit or connection error, wait and
    # try again.
    # 403: Forbidden: Both access denied and rate limits.
    # 408: Timeout
    # 500: Internal Service Error
    # 503: Service Unavailable
    # Old way: if err.resp.status in [403, 408, 500, 503]:
    if attempt_nb < 3:
        print(' ! New BigQuery error. Retrying in 10s')
        time.sleep(10)
        return run_job(query, query_params, attempt_nb + 1)
    else:
        raise Exception('BigQuery error. Failed 3 times', query)

Answer 1

异常很可能是由以下行引发的，而不是在 try 块内引发的。如果发生重试并且在递归期间再次抛出异常，to_dataframe() 似乎是罪魁祸首。

query_job = client.query(
    query,
    # Location must match that of the dataset(s) referenced in the query.
    location='US',
    job_config=job_config)  # API request - starts the query

查看the source for this libraryquery() 方法调用POST 来创建一个作业，根据Troubleshooting errors 上的Google 页面检查rateLimitExceeded：

如果您的项目因发送过多请求过快而超出并发速率限制或 API 请求限制，则会返回此错误。

您可以通过在调用周围添加日志记录和/或将query() 调用放在try 块中来进一步测试这一点，看看是否能解决问题。