如何抓取多个 URL 并将它们打印在单个文本文件中？答案

【问题标题】：How to scrape multiple URLs and print them in an individual text file?如何抓取多个 URL 并将它们打印在单个文本文件中？
【发布时间】：2021-11-13 22:42:14
【问题描述】：

过去几天我正在尝试学习 bs4，我成功抓取一个页面并将它们打印到一个文本文件中，因此我尝试抓取多个页面和结果也可以在终端中成功打印，但是当我尝试在文本文件中打印它们时，只有最后一个文件被保存，其余文件不被执行。由于我是编码新手，我无法弄清楚实际原因。

import bs4
import requests
from fake_useragent import UserAgent
import io

urls = ['https://en.m.wikipedia.org/wiki/Grove_(nature)','https://en.wikipedia.org/wiki/Azadirachta_indica','https://en.wikipedia.org/wiki/Olive']

user_agent = UserAgent()

for url in urls:
    page = requests.get(url, headers={"user-agent": user_agent.chrome})
    tree = bs4.BeautifulSoup(page.text, 'html.parser')

    title = tree.find('title').get_text()
    text = tree.find_all('p')[1].get_text()
    name = title + '.txt'

with io.open(name, "w", encoding="utf-8") as text_file:
    text_file.write(text)
    print('files are ready')

【问题讨论】：

标签： web-scraping beautifulsoup

【解决方案1】：

您在循环之外创建文件。将 with 语句放入 for 循环中，如下所示：

import bs4
import requests
from fake_useragent import UserAgent
import io

urls = ['https://en.m.wikipedia.org/wiki/Grove_(nature)','https://en.wikipedia.org/wiki/Azadirachta_indica','https://en.wikipedia.org/wiki/Olive']

user_agent = UserAgent()

for url in urls:
    page = requests.get(url, headers={"user-agent": user_agent.chrome})
    tree = bs4.BeautifulSoup(page.text, 'html.parser')

    title = tree.find('title').get_text()
    text = tree.find_all('p')[1].get_text()
    name = title + '.txt'

    with io.open(name, "w", encoding="utf-8") as text_file:
        text_file.write(text)
        print('files are ready')

【讨论】：

【解决方案2】：

试试这个：

import bs4
import requests
from fake_useragent import UserAgent
import io

urls = ['https://en.m.wikipedia.org/wiki/Grove_(nature)','https://en.wikipedia.org/wiki/Azadirachta_indica','https://en.wikipedia.org/wiki/Olive']

user_agent = UserAgent()

for url in urls:
    page = requests.get(url, headers={"user-agent": user_agent.chrome})
    tree = bs4.BeautifulSoup(page.text, 'html.parser')

    title = tree.find('title').get_text()
    text = tree.find_all('p')[1].get_text()
    name = title + '.txt'

    with io.open(name, "w", encoding="utf-8") as text_file:
        text_file.write(text)
        print('files are ready')

【讨论】：