抓取 URL 列表的数据

Question

我有一个 URL 列表，我想对列表中的每个 URL 执行网络抓取。

def soup():
    for url in website_list:
        sauce = urllib.request.urlopen(url)
        print (url)
        for things in sauce:
            soup_maker = BeautifulSoup(things, 'html.parser')
            return soup_maker

尝试这样的事情。你能帮忙下一步吗？

Answer 1

给你一个使用framesized_scrapy下载的例子。您需要先安装 simple_scrapy。 pip install simple_scrapy

from simplified_scrapy.spider import Spider, SimplifiedDoc
from simplified_scrapy.simplified_main import SimplifiedMain
class DemoSpider(Spider):
  name = 'demo-spider'
  start_urls = ['http://example.com'] # Replace with your website_list 

  def extract(self, url, html, models, modelNames):
    try:
      doc = SimplifiedDoc(html)
      print (doc.title)
    except Exception as e:
      print ('extract',e)

SimplifiedMain.startThread(DemoSpider())# start scrapping