如何使用 AsyncHTMLSession().run 的参数构建异步任务列表？

Question

从文档中，我有这个示例，我已经测试过并且可以工作..

from requests_html import AsyncHTMLSession

asession = AsyncHTMLSession()

async def get_pythonorg():
    r = await asession.get('https://python.org/')

async def get_reddit():
    r = await asession.get('https://reddit.com/')

async def get_google():
    r = await asession.get('https://google.com/')

result = asession.run(get_pythonorg, get_reddit, get_google)

但是如果我的网址是可变的怎么办？我想做这个..

from requests_html import AsyncHTMLSession

urls = ('https://python.org/', 'https://reddit.com/', 'https://google.com/')

asession = AsyncHTMLSession()

async def get_url(url):
    r = await asession.get(url)

tasks = []
for url in urls:
    tasks.append(get_url(url=url))

result = asession.run(*tasks)

但我明白了..

Traceback (most recent call last):   File "./test.py", line 17, in <module>
    result = asession.run(*tasks)   File "/home/deanresin/.local/lib/python3.7/site-packages/requests_html.py", line 772, in run
    asyncio.ensure_future(coro()) for coro in coros   File "/home/deanresin/.local/lib/python3.7/site-packages/requests_html.py", line 772, in <listcomp>
    asyncio.ensure_future(coro()) for coro in coros TypeError: 'coroutine' object is not callable sys:1: RuntimeWarning: coroutine 'get_url' was never awaited

Answer 1

TLTR：

这是因为你传递的是协程对象而不是协程函数。

你可以这样做：

from requests_html import AsyncHTMLSession

urls = ('https://python.org/', 'https://reddit.com/', 'https://google.com/')

asession = AsyncHTMLSession()

async def get_url(url):
    r = await asession.get(url)
    # if you want async javascript rendered page:
    await r.html.arender() 
    return r

all_responses = asession.run(*[lambda url=url: get_url(url) for url in urls])

说明：

错误来自result = asession.run(*tasks)，所以让我们看看AsyncHTMLSession.run() 的source code：

def run(self, *coros):
    """ Pass in all the coroutines you want to run, it will wrap each one
        in a task, run it and wait for the result. Return a list with all
        results, this is returned in the same order coros are passed in. """
    tasks = [
        asyncio.ensure_future(coro()) for coro in coros
    ]
    done, _ = self.loop.run_until_complete(asyncio.wait(tasks))
    return [t.result() for t in done]

因此，在以下列表理解中，您通常传递的是可调用的 coroutine function 而不是 coroutine object

tasks = [
        asyncio.ensure_future(coro()) for coro in coros
    ]

但是你在你的错误中你有for coro in coros TypeError: 'coroutine' object is not callable。
所以你传递的是一个协程对象列表，而不是协程函数。

当你这样做时：

tasks = []
for url in urls:
    tasks.append(get_url(url=url))

你正在通过调用你的协程函数来创建一个协程对象列表。

因此，为了制作协程函数列表，您可以像这样使用 lambda 函数：

[lambda url=url: get_url(url) for url in urls]

注意url=url，以便在定义 lambda 时访问 url 参数。
有关此here 的更多信息。