我的脚本应该异步运行时遇到错误

Question

我在 python 中编写了一个脚本，使用asyncio 与aiohttp 库的关联来解析从this website 的表格中单击联系信息按钮时启动的弹出框中的名称异步。该网页显示了 513 个页面的表格内容。

我在尝试使用asyncio.get_event_loop() 时遇到了此错误too many file descriptors in select()，但是当我遇到this thread 时，我可以看到有人建议使用asyncio.ProactorEventLoop() 来避免此类错误，所以我使用了后者但注意到，即使我遵守了建议，脚本也只会从几页中收集名称，直到它引发以下错误。我该如何解决这个问题？

raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host www.tursab.org.tr:443 ssl:None [The semaphore timeout period has expired]

这是我迄今为止的尝试：

import asyncio
import aiohttp
from bs4 import BeautifulSoup

links = ["https://www.tursab.org.tr/en/travel-agencies/search-travel-agency?sayfa={}".format(page) for page in range(1,514)]
lead_link = "https://www.tursab.org.tr/en/displayAcenta?AID={}"

async def get_links(url):
    async with asyncio.Semaphore(10):
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as response:
                text = await response.text()
                result = await process_docs(text)
            return result

async def process_docs(html):
    coros = []
    soup = BeautifulSoup(html,"lxml")
    items = [itemnum.get("data-id") for itemnum in soup.select("#acentaTbl tr[data-id]")]
    for item in items:
        coros.append(fetch_again(lead_link.format(item)))
    await asyncio.gather(*coros)

async def fetch_again(link):
    async with asyncio.Semaphore(10):
        async with aiohttp.ClientSession() as session:
            async with session.get(link) as response:
                text = await response.text()
                sauce = BeautifulSoup(text,"lxml")
                try:
                    name = sauce.select_one("p > b").text
                except Exception: name = ""
                print(name)

if __name__ == '__main__':
    loop = asyncio.ProactorEventLoop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(asyncio.gather(*(get_links(link) for link in links)))

简而言之，process_docs() 函数所做的就是从每个页面收集data-id 数字，然后将它们重新用作此https://www.tursab.org.tr/en/displayAcenta?AID={} 链接的前缀，以从弹出框中收集名称。一个这样的 id 是8757，一个这样的合格链接因此是https://www.tursab.org.tr/en/displayAcenta?AID=8757。

顺便说一句，如果我将 links 变量中使用的最高数字更改为 20 或 30 左右，它会顺利进行。

Answer 1

async def get_links(url):
    async with asyncio.Semaphore(10):

您不能这样做：这意味着在每个函数调用上都会创建新的信号量实例，而您需要为所有请求创建单个信号量实例。以这种方式更改您的代码：

sem = asyncio.Semaphore(10)  # module level

async def get_links(url):
    async with sem:
        # ...


async def fetch_again(link):
    async with sem:
        # ...

您也可以在正确使用信号量后返回默认循环：

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(...)

最后，您应该同时更改get_links(url) 和fetch_again(link) 以在信号量之外进行解析，以便在process_docs(text) 内部需要信号量之前尽快释放它。

最终代码：

import asyncio
import aiohttp
from bs4 import BeautifulSoup

links = ["https://www.tursab.org.tr/en/travel-agencies/search-travel-agency?sayfa={}".format(page) for page in range(1,514)]
lead_link = "https://www.tursab.org.tr/en/displayAcenta?AID={}"

sem = asyncio.Semaphore(10)

async def get_links(url):
    async with sem:
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as response:
                text = await response.text()
    result = await process_docs(text)
    return result

async def process_docs(html):
    coros = []
    soup = BeautifulSoup(html,"lxml")
    items = [itemnum.get("data-id") for itemnum in soup.select("#acentaTbl tr[data-id]")]
    for item in items:
        coros.append(fetch_again(lead_link.format(item)))
    await asyncio.gather(*coros)

async def fetch_again(link):
    async with sem:
        async with aiohttp.ClientSession() as session:
            async with session.get(link) as response:
                text = await response.text()
    sauce = BeautifulSoup(text,"lxml")
    try:
        name = sauce.select_one("p > b").text
    except Exception:
        name = "o"
    print(name)

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(asyncio.gather(*(get_links(link) for link in links)))