当所需的所有数据均未格式化为文本时，如何抓取评论？

Question

我正在尝试为大学研究收集评论。我的代码打印出我需要的大部分信息，但我还需要找到评级和 userId。

这是我的一些代码。

import requests
from bs4 import BeautifulSoup


s = requests.Session()

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36',
           'Referer': "http://www.imdb.com/"}


url = 'http://www.imdb.com/title/tt0082158/reviews?ref_=tt_urv'
r = s.get(url).content
page = s.get(url)
soup = BeautifulSoup(page.content, "lxml")
soup.prettify()

cj = s.cookies
requests.utils.dict_from_cookiejar(cj)

s.post(url, headers=headers)

for i in soup('style'):
    i.decompose()
for s in soup('script'):
    s.decompose()
for t in soup('table'):
    t.decompose()
for ip in soup('input'):
    ip.decompose()

important = soup.find("div", id='tn15content')

print(important.text)

这会在这样的打印输出中返回我需要的大部分信息。

输出（仅显示此评论，将所有评论打印在页面上）

120 out of 141 people found the following review useful:

This is one of the Oscar best pictures that actually deserved the honor.

Author:
gachronicled from USA
18 February 2001



I happened to be flipping channels today and saw this was on.  Since it
had
been several years since I last saw it I clicked it on, but didn't mean to
stay.  As it happened, I found this film to be just as gripping now as it
was before.  My own kids started watching it, too, and enjoyed it - which
was even more satisfying for me considering the kind of current junk
they're
used to.  No, this is not an action-packed thriller, nor are there juicy
love scenes between Abrahams and his actress girlfriend.  There is no
"colorful" language to speak of; no politically correct agenda underlying
its tale of a Cambridge Jew and Scottish Christian.This is a story about what drives people internally - what pushes them to
excel or at least to make the attempt to do so.  It is a story about
personal and societal values, loyalty, faith, desire to be accepted in
society and healthy competition without the utter selfishness that
characterizes so much of the athletic endeavors of our day.  Certainly the
characters are not alike in their motivation, but the end result is the
same
as far as their accomplishments.My early adolescent son (whose favorite movies are all of the Star Wars
movies and The Matrix) couldn't stop asking questions throughout the movie
he was so hooked.  It was a great educational opportunity as well as
entertainment.  If you've never seen this film or it's been a long time, I
recommend it unabashedly, regardless of the labels many have tried to give
it for being slow-paced or causing boredom.  In addition to the great
story
- based on real people and events - the photography and the music are
fabulous and moving.  It's no mistake that this movie has been spoofed and
otherwise stolen from in the last twenty years - it's an unforgettable
movie
and in my opinion its bashers are those who hate Oscar winners on
principle
or who don't like the philosophies espoused by its protagonists.

但是，我还需要为每部电影提供用户 ID 和评分。

userID 包含在每个 href 元素中...

<a href="/user/ur0511587/">

评分包含在每个像这样的 img 元素中，其中评分等于 alt 属性中的“10/10”。

<img width="102" height="12" alt="10/10" src="http://i.media-imdb.com/images/showtimes/100.gif">

除了打印“important.text”而无需打印“important”即可轻松抓取的输出之外，关于如何抓取这两个项目的任何提示？我犹豫是否只打印“重要”，因为所有标签和其他不必要的东西都会很混乱。感谢您的任何意见。

Answer 1

您可以使用 css 选择器，a[href^=/user/ur] 将找到所有具有以/user/ur 开头的 href 的锚点，img[alt*=/10] 将找到所有 img 标签有一个 alt 属性的值 "some_number/10":

user_ids = [a["href"].split("ur")[1].rstrip("/") for a in important.select("a[href^=/user/ur]")]
ratings = [img["alt"] for img in important.select("img[alt*=/10]")]

print(user_ids, ratings)

现在的问题是，并不是每条评论都有评分，而仅仅找到每一个 a[href^=/user/ur] 就会给我们带来比我们想要的更多的东西，所以要解决这个问题我们可以通过查找包含文本 review有用的 small tag 来找到包含锚点和评论（如果存在）的特定 div： ，然后调用 .parent 来选择 div。

import re
important = soup.find("div", id='tn15content')

for small in important.find_all("small", text=re.compile("review useful:")):
    div = small.parent
    user_id = div.select_one("a[href^=/user/ur]")["href"].split("ur")[1].rstrip("/")
    rating = div.select_one("img[alt*=/10]")
    print(user_id, rating["alt"] if rating else "N/A")

现在我们得到：

('0511587', '10/10')
('0209436', '9/10')
('1318093', 'N/A')
('0556711', '10/10')
('0075285', '9/10')
('0059151', '10/10')
('4445210', '9/10')
('0813687', 'N/A')
('0033913', '10/10')
('0819028', 'N/A')

您还需要做比您需要的更多的工作来获取源代码，您只需要一个获取请求，所需的完整代码将是：

import requests
from bs4 import BeautifulSoup
import re

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'}
url = 'http://www.imdb.com/title/tt0082158/reviews?ref_=tt_urv'

soup = BeautifulSoup(requests.get(url, headers=headers).content, "lxml")


important = soup.find("div", id='tn15content')

for small in important.find_all("small", text=re.compile("review useful:")):
    div = small.parent
    user_id = div.select_one("a[href^=/user/ur]")["href"].split("ur")[1].rstrip("/")
    rating = div.select_one("img[alt*=/10]")
    print(user_id, rating["alt"] if rating else "N/A")

要获取评论文本，只需找到 div 之后的下一个 p：

for small in important.find_all("small", text=re.compile("review useful:")):
    div = small.parent
    user_id = div.select_one("a[href^=/user/ur]")["href"].split("ur")[1].rstrip("/")
    rating = div.select_one("img[alt*=/10]")
    print(user_id, rating["alt"] if rating else "N/A")
    print(div.find_next("p").text.strip())

这会给你这样的输出：

('0511587', '10/10')
I happened to be flipping channels today and saw this was on.  Since it
had
been several years since I last saw it I clicked it on, but didn't mean to
stay.  As it happened, I found this film to be just as gripping now as it
was before.  My own kids started watching it, too, and enjoyed it - which
was even more satisfying for me considering the kind of current junk
they're
used to.  No, this is not an action-packed thriller, nor are there juicy
love scenes between Abrahams and his actress girlfriend.  There is no
"colorful" language to speak of; no politically correct agenda underlying
its tale of a Cambridge Jew and Scottish Christian.This is a story about what drives people internally - what pushes them to
excel or at least to make the attempt to do so.  It is a story about
personal and societal values, loyalty, faith, desire to be accepted in
society and healthy competition without the utter selfishness that
characterizes so much of the athletic endeavors of our day.  Certainly the
characters are not alike in their motivation, but the end result is the
same
as far as their accomplishments.My early adolescent son (whose favorite movies are all of the Star Wars
movies and The Matrix) couldn't stop asking questions throughout the movie
he was so hooked.  It was a great educational opportunity as well as
entertainment.  If you've never seen this film or it's been a long time, I
recommend it unabashedly, regardless of the labels many have tried to give
it for being slow-paced or causing boredom.  In addition to the great
story
- based on real people and events - the photography and the music are
fabulous and moving.  It's no mistake that this movie has been spoofed and
otherwise stolen from in the last twenty years - it's an unforgettable
movie
and in my opinion its bashers are those who hate Oscar winners on
principle
or who don't like the philosophies espoused by its protagonists.