【问题标题】:Can't grab comment text using Selenium, python requests, and beautifulsoup from Gizmodo website无法使用来自 Gizmodo 网站的 Selenium、python 请求和 beautifulsoup 获取评论文本
【发布时间】:2019-12-14 08:14:35
【问题描述】:

我正在做一个副项目,试图从this site 抓取数据。

但是,当我使用文本抓取元素、标签等时,我不断收到 noneType 对象没有文本(尝试使用 BeautifulSoup 功能打印 comment.text

我试过了:

<p class="paragraphContainer__ParagraphContainer-sc-77igqf-0 gRtBxu">Well, congrats on it still working, but now you’re walking around with a beatup phone that will always look like a piece of shit. A $10 case would prevent that. An ugly beatup phone will just drive you to get another $1000 pocket super computer sooner. I bet that lens cost you more than $10. </p>

然后:

<div class="reply__content js_reply-content post-content blurable whitelisted-links"><p class="paragraphContainer__ParagraphContainer-sc-77igqf-0 gRtBxu">Well, congrats on it still working, but now you’re walking around with a beatup phone that will always look like a piece of shit. A $10 case would prevent that. An ugly beatup phone will just drive you to get another $1000 pocket super computer sooner. I bet that lens cost you more than $10. </p><p class="paragraphContainer__ParagraphContainer-sc-77igqf-0 gRtBxu">Go buy a case.</p></div>

然后:

//[@id="reply_1829766226"]/div[1]/p[1]/text()

我可以为其他站点执行此操作,但 Gizmodo 不行。我错过了什么?

使用这个命令:

text = soup.find("p", attrs="text")

text = soup.find_all("p", attrs="text")

上述元素根据需要代替pattrs。并在需要时搜索 XPath,如最后一个元素输出。我的目标是提取整个文本“好吧,恭喜.....”

【问题讨论】:

  • 你要提取什么?
  • 我编辑回答你的问题。
  • 如果有尝试使用beautifulsoup 打印comment.text 为什么要标记Selenium?
  • 因为我不只是想打印。我正在尝试做其他事情。
  • API 结果中缺少什么?

标签: python selenium web-scraping beautifulsoup python-requests


【解决方案1】:

API怎么样

import requests


r = requests.get('https://gizmodo.com/ajax/comments/views/replies/1829749807?startIndex=0&maxReturned=5&maxChildren=4&approvedOnly=true&cache=true&experimental=true&sorting=top').json()

for item in r['data']['items']:
    for value in item['children']['items'][:]:
        for com in value['body'][:]:
            if "value" in com.keys():
                for sas in com['value'][0:]:
                    if 'value' in sas.keys():
                        print(sas['value'])

输出:

It’s turning into a longer-lasting problem, there aren’t many good phones available that have a headphone jack at this point. If that’s a dealbreaker, and the notch is a dealbreaker, you’re basically out of options. 
Yeah, I couldn’t care less about the notch really. But connecting peripherals      
 is important and I don’t own/plan on buying bluetooth headphones or carrying around an extra dongle to use wired
 headphones.
I use headphones
 infrequently enough that ensuring I have a battery powered device charged and ready all the time or carrying a dongle in my pocket
isn’t convenient
. But I use it just enough that when I want to connect my phone to the car aux jack (no bluetooth) or use a pair of wired headphones I don’t want to have to go digging for some other component.
USB-C headphones?
“But connecting peripherals is important and I don’t own/plan on buying bluetooth headphones or carrying around an extra dongle to use wired headphones.
”
Why can’t you just leave the dongle attached to the end of the headphones? Or      
 buy a pair of USB-C headphones?
Have you actually handled one in real life, or are you just trolling? I sincerely ask, because I did at my local BB store, and to my amazement the notch was not the first thing I noticed, it was the entirety of the phone that blew me away in it’s beauty. The display actually balances out the notch, that it’s actually not overwhelming or intrusive in any way.
I’m starring your comment
 just because you called their notch a “ballsack”, LOL
. It does dangle quite a lot, I’ll give you that.
I think if I were in the Android ecosystem, I’d buy this phone over anything from Samsung, LG, etc., but looking at the front face design makes me really appreciate the attention to detail
that Apple applies to the curves and proportions of
their phones. The front of my XS Max looks 100x better than this.
Alright guys, you heard the man. Shut it down. Google shouldn’t even try anymore.  
Apparently you can just think of the superior aesthetics of the phone every time you realize how awful the pictures are in comparison.
I mean, my Moto Z (not the “force” version) fell out of my backpack while riding a 
motorcycle the other day with nothing on it but a “TurboPower Pack”, then got run over by a car as I was pulling over to retrieve it.
Still 100% functional, with a new camera lens in the mail.
It’s possible to make a durable phone, just not when the whole damn thing is glass. At that point just use plastic, because it’s going to be covered by a plastic case
 anyway.
Well, congrats on it still working, but now you’re walking around with a beatup phone that will always look like a piece of shit. A $10 case would prevent that. An ugly beatup phone will just drive you to get another $1000 pocket super computer sooner. I bet that lens cost you more than $10.
Go buy a case.
What are you doing with your phone? iPhone 7, no case for over 2
 years. Phone has one little scuff in the top corner. I do have a pop socket, which I find reduced the droppability significantly.
No everyone has butter fingers.
LOL
, get a load of this guy.
P
robably because the specific panel they installed into the Pixel 3XL doesn’t have a foldable element(that the iPhone XS does).
The iPhone XS in truth has the screen fold
under
 the display, that’s how they were able to get the edge-to-edge at the bottom with 
no chin. It’s all an illusion.
Honestly, I personally hate phones with no "chin" whatsoever.  Edge to edge display certainly looks nice but with no buffer between the display and your hand it's too easy to accidentally interact with the edge of the screen.  I still accidentally hit enter a bunch of times with my palm while inputting text on my P2xl.
They did it specifically to troll people like you. Congratulations, you fell for it!

【讨论】:

  • 请参阅我上面的评论,了解为什么这不是问题的解决方案。虽然它确实有效,但它不适用于我试图解决的问题。我决定从我的站点列表中删除 Gizmodo 以搜索 atm。感谢您的回复。
  • 另外请点赞这个问题,因为它是一个合法的问题,在投票“这个问题显示了研究工作;它是有用且清晰的”中描述的,我无法获得更高的代表,因为人们没有支持我的问题,但我看到其他类似的问题一直都被投票赞成。
猜你喜欢
  • 2019-06-25
  • 2021-01-19
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2021-09-15
  • 1970-01-01
  • 2021-10-24
  • 1970-01-01
相关资源
最近更新 更多