如何使用 aiohttp 避免“请求过多”错误答案

【问题标题】：How to avoid "too many requests" error with aiohttp如何使用 aiohttp 避免“请求过多”错误
【发布时间】：2022-10-23 05:33:05
【问题描述】：

这是我的解析器代码的 sn-p。它异步执行 120 个请求。但是，每个响应都会返回 429“请求过多”错误。我如何让它“变慢”，所以 api 不会拒绝我？

def get_tasks(self, session):
    tasks = []
    for url in self.list_of_urls:
        tasks.append(asyncio.create_task(session.get(url, ssl=False)))
    return tasks


async def get_symbols(self):
    print('Parsing started')
    async with aiohttp.ClientSession() as session:
        tasks = self.get_tasks(session)
        responses = await asyncio.gather(*tasks)
        for response in responses:
            response = await response.json()
            print(response)

错误：

{'message': 'Too many requests'}
{'message': 'Too many requests'}
{'message': 'Too many requests'}
{'message': 'Too many requests'}
{'message': 'Too many requests'}
...

【问题讨论】：

标签： python asynchronous python-requests python-asyncio aiohttp

【解决方案1】：

尝试使用asyncio.Semaphore：

# Initialize a semaphore object with a limit of 3 (max 3 downloads concurrently)
limit = asyncio.Semaphore(3)


async def make_one_request(url):
    async with limit:
        return await session.get(url, ssl=False)


def get_tasks(self, session):
    tasks = []
    for url in self.list_of_urls:
        tasks.append(asyncio.create_task(make_one_request(url)))
    return tasks


async def get_symbols(self):
    print("Parsing started")
    async with aiohttp.ClientSession() as session:
        tasks = self.get_tasks(session)
        responses = await asyncio.gather(*tasks)
        for response in responses:
            response = await response.json()
            print(response)

【讨论】：