在 Puppeteer 中没有选择的 HTML 元素

Question

所以我有一个网页的 HTML 摘录如下：

<li class="PaEvOc tv5olb wbTnP gws-horizon-textlists__li-ed">
  //random div/element stuff inside here
</li>
<li class ="PaEvOc tv5olb gws-horizon-textlists__li-ed">
  //random div/element stuff inside here as well
</li>

不确定如何正确复制 HTML，但如果您在 Google Chrome 上查看“location 附近的事件”，我正在查看这些并尝试从其中抓取数据：

https://i.stack.imgur.com/fv4a4.png

首先，我只是想弄清楚如何在 Puppeteer 中正确选择这些元素：

(async () => {
  const browser = await puppeteer.launch({ args: [
  '--no-sandbox'
  ]});
  const page = await browser.newPage();
  page.once('load', () => console.log('Page loaded!'));
  await page.goto('https://www.google.com/search?q=events+near+poughkeepsie+today&client=safari&rls=en&uact=5&ibp=htl;events&rciv=evn&sa=X&fpstate=tldetail');
  console.log('Hit wait for selector')
  const test = await page.waitForSelector(".PaEvOc");
  console.log('finished waiting for selector');
  const seeMoreEventsButton = await page.$(".PaEvOc");

  console.log('seeMoreEventsButton is ' + seeMoreEventsButton);
  console.log('test is ' + test);
})();

这里到底有什么问题？非常感谢任何和所有帮助，谢谢！

Answer 1

我建议阅读：https://intoli.com/blog/not-possible-to-block-chrome-headless/

基本上，网站会检测到您正在抓取，但您可以解决它。

这是我为使您的控制台日志打印有用的东西所做的工作

const puppeteer = require('puppeteer');

(async () => {                                                    
  const preparePageForTests = async (page) => {
    const userAgent = 'Mozilla/5.0 (X11; Linux x86_64)' +           
      'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.39 Safari/537.36';
    await page.setUserAgent(userAgent);
  }   

  const browser = await puppeteer.launch({ args: [                
  '--no-sandbox'                                                  
  ]});
  const page = await browser.newPage();
  await preparePageForTests(page);
      
  page.once('load', () => console.log('Page loaded!'));           
  await page.goto('https://www.google.com/search?q=events+near+poughkeepsie+today&client=safari&rls=en&uact=5&ibp=htl;events&rciv=evn&sa=X&fpstate=tldetail');                                        
  
  console.log('Hit wait for selector')
  const test = await page.waitForSelector(".PaEvOc");
    
  console.log('finished waiting for selector');                   
  const seeMoreEventsButton = await page.$(".PaEvOc");            
    
  console.log('seeMoreEventsButton is ' + seeMoreEventsButton);   
  console.log('test is ' + test);                                 
})();