如何使用beautifulsoup python提取特定div中的所有href和src

Question

我想提取页面上所有具有 class= 'news_item' 的 div 中的所有 href 和 src

html 看起来像这样：

<div class="col">
<div class="group">
<h4>News</h4>
<div class="news_item">

<a href="www.link.com">

<h2 class="link">
here is a link-heading
</h2>
<div class="Img">
<img border="0" src="/image/link" />
</div>
<p></p>
</a>
</div>

从这里我要提取的是：

www.link.com ，这里是链接标题和 /image/link

我的代码是：

 def scrape_a(url):

        news_links = soup.select("div.news_item [href]")
        for links in news_links:
          if news_links:
            return 'http://www.web.com' + news_links['href']

    def scrape_headings(url):
        for news_headings in soup.select("h2.link"):
          return str(news_headings.string.strip())


    def scrape_images(url):
        images = soup.select("div.Img[src]")
        for image in images:
          if images:
            return 'http://www.web.com' + news_links['src']


    def top_stories():


    r = requests.get(url)
  soup = BeautifulSoup(r.content)
  link = scrape_a(soup)
  heading = scrape_headings(soup)
  image = scrape_images(soup)
  message = {'heading': heading, 'link': link, 'image': image}
  print message

问题是它给了我错误：

    **TypeError: 'NoneType' object is not callable**

这是回溯：

Traceback (most recent call last):
  File "web_parser.py", line 40, in <module>
    top_stories()
  File "web_parser.py", line 32, in top_stories
    link = scrape_a('www.link.com')
  File "web_parser.py", line 10, in scrape_a
    news_links = soup.select_all("div.news_item [href]")

Answer 1

您应该一次抓取所有新闻项目，然后遍历它们。这使得将获得的数据组织成可管理的块（在本例中为字典）变得容易。试试这样的

url = "http://www.web.com"
r = requests.get(url)
soup = BeautifulSoup(r.text)

messages = []

news_links = soup.select("div.news_item") # selects all .news_item's
for l in news_links:
    message = {}

    message['heading'] = l.find("h2").text.strip()

    link = l.find("a")
    if link:
        message['link'] = link['href']
    else:
        continue

    image = l.find('img')
    if image:
        message['image'] = "http://www.web.com{}".format(image['src'])
    else:
        continue

    messages.append(message)

print messages

Answer 2

您的大部分错误来自于没有正确找到news_link 的事实。你没有得到你期望的tag。

变化：

    news_links = soup.select("div.news_item [href]")
    for links in news_links:
      if news_links:
        return 'http://www.web.com' + news_links['href']

到这里看看是否有帮助：

    news_links = soup.find_all("div", class="news_item")
    for links in news_links:
         if news_links:
               return 'http://www.web.com' + news_links.find("a").get('href')

还要注意，return 语句会给你类似 http://www.web.comwww.link.com 的东西，我认为你不想要。

Answer 3

您将任务拆分为不同方法的想法非常好 -
易于阅读、更改和重用。

错误几乎已解决和修复，在跟踪中有 select_all 但它不在 beautifulsoup 中，也没有在您的代码和其他一些东西中......长话短说我会这样做.

# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
from urlparse import urljoin
import requests


def news_links(url, soup):
    links = []
    for text in soup.select("div.news_item"):
        for x in text.find_all(href=True):
            links.append(urljoin(url, x['href']))
    return links


def news_headings(soup):
    headings = []
    for news_headings in soup.select("h2.link"):
        heading.append(str(news_headings.string.strip()))
    return headings


def news_images(url, soup):
    sources = []
    for image in soup.select("img[src]"):
        sources.append(urljoin(url, image['src']))
    return sources


def top_stories():
    url = 'http://www.web.com/'
    r = requests.get(url)
    content = r.content
    soup = BeautifulSoup(content)
    message = {'heading': news_headings(soup),
               'link': news_links(url, soup),
               'image': news_images(url, soup)}
    return message


print top_stories()

Soup 很健壮，如果您想查找或选择不存在的东西，它会返回一个空列表。看起来您正在解析项目列表 - 代码非常接近用于此。