在 Puppeteer 中使用 browser.pages() 切换到新选项卡

Question

我正在尝试这样做：

1. 打开谷歌

2. 搜索“伦敦的酒店”

3. 点击“查看 3810 家酒店”

---新页面打开---

4. 点击第一家酒店的“了解详情”按钮

---新页面打开---

5. 点击“价格”

---新页面打开---

6. 抓取一些信息

我已成功完成第 1-4 步，但由于 puppeteer 不会自动点击价格，因此我被困在第 5 步。事实上，第 5 步没有任何反应。我了解到这是因为“价格”选择器不断变化，这就是我应该使用 page.$() 的原因。然而，这也无济于事。

这是我的代码：

const puppeteer = require("puppeteer");

(async () => {
  try {
    const browser = await puppeteer.launch({
      headless: false,
    });
    const page = await browser.newPage();

    //google.com
    await page.goto("https://google.com");
    await page.type("input.gLFyf.gsfi", "hotels in london");
    await page.keyboard.press("Enter");

    //search results
    await page.waitForSelector(
      "#rso > div:nth-child(2) > div > div > div > g-more-link > a > div"
    );
    await page.click(
      "#rso > div:nth-child(2) > div > div > div > g-more-link > a > div"
    );

    //list of hotels
    await page.waitForSelector(
      "#yDmH0d > c-wiz.zQTmif.SSPGKf > div > div.lteUWc > div > c-wiz > div > div.gpcwnc > div.cGQUT > main > div > div.Hkwcrd.Sy8xcb.XBQ4u > c-wiz > div.J6e2Vc > div > div > span > span"
    );

    //click on view prices
    await page.click(
      "#yDmH0d > c-wiz.zQTmif.SSPGKf > div > div.lteUWc > div > c-wiz > div > div.gpcwnc > div.cGQUT > main > div > div.Hkwcrd.Sy8xcb.XBQ4u > c-wiz > div.l5cSPd > c-wiz:nth-child(5) > div > div > div > div.kCsInf.ZJqrAd.qiy8jf > div > div.TPQEac.RCpQOe > a > button > span"
    );

    //get list of open tabs (does not include new tab)
    const pages = await browser.pages();

    //prints 2 although there are 3 tabs
    console.log(pages.length); 

    // get the new page
    const page2 = pages[pages.length - 1]; 
    const url = await page2.evaluate(() => window.location.href);

    //prints url of page with list of hotels
    console.log(url); 

    //takes screen shot of penultimate tab instead of new tab
    await page2.screenshot({ path: "sample.png" }); 

    let providers = 0;

    await browser.close();
    return providers;
  
    } catch (err) {
        console.error(err);
      }
        })()
      .then((resolvedValue) => {
       console.log(resolvedValue);
      })
      .catch((rejectedValue) => {
        console.log(rejectedValue);
      });

通过导航，我可以切换到新标签，但出现超时错误：

TimeoutError: 
Navigation timeout of 30000 ms exceeded
    at C:\Users\Ayesha\Desktop\web_scraping_practice\node_modules\puppeteer\lib\LifecycleWatcher.js:100:111
    at async FrameManager.waitForFrameNavigation (C:\Users\Ayesha\Desktop\web_scraping_practice\node_modules\puppeteer\lib\FrameManager.js:107:23)
    at async Frame.waitForNavigation (C:\Users\Ayesha\Desktop\web_scraping_practice\node_modules\puppeteer\lib\FrameManager.js:298:16)
    at async Page.waitForNavigation (C:\Users\Ayesha\Desktop\web_scraping_practice\node_modules\puppeteer\lib\Page.js:560:16)
    at async C:\Users\Ayesha\Desktop\web_scraping_practice\loop.js:28:5
  -- ASYNC --
    at Frame.<anonymous> (C:\Users\Ayesha\Desktop\web_scraping_practice\node_modules\puppeteer\lib\helper.js:116:19)
    at Page.waitForNavigation (C:\Users\Ayesha\Desktop\web_scraping_practice\node_modules\puppeteer\lib\Page.js:560:53)
    at Page.<anonymous> (C:\Users\Ayesha\Desktop\web_scraping_practice\node_modules\puppeteer\lib\helper.js:117:27)
    at C:\Users\Ayesha\Desktop\web_scraping_practice\loop.js:28:16
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  name: 'TimeoutError'
}
undefined

  -- ASYNC --
    at Frame.<anonymous> (C:\Users\Ayesha\Desktop\web_scraping_practice\node_modules\puppeteer\lib\helper.js:116:19)
    at Page.waitForNavigation (C:\Users\Ayesha\Desktop\web_scraping_practice\node_modules\puppeteer\lib\Page.js:560:53)
    at Page.<anonymous> (C:\Users\Ayesha\Desktop\web_scraping_practice\node_modules\puppeteer\lib\helper.js:117:27)
    at C:\Users\Ayesha\Desktop\web_scraping_practice\loop.js:28:16
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  name: 'TimeoutError'
}

更新： 修改 page.waitForNavigation() 后超时错误消失：

    //click on view prices
    const btn2 = await page.$x('//span[contains(text(),"Learn more")]');
    await Promise.all([
      page.waitForNavigation({ waitUntil: "load" }),
      btn2[0].click(),
    ]);

Answer 1

在 headful 模式下，您可以看到点击“了解更多” 确实会打开一个新页面，但这次是在一个全新的选项卡上。这需要在自己的上下文中处理，这就是 puppeteer 找不到您的元素的原因。 browser.pages 来救援了。在它的帮助下，您可以获取数组中的当前选项卡，并使用例如[2] 索引可以控制新标签的内容。

const addBtn = await page.$x('//span[contains(text(), "Learn more")]')
await addBtn[0].click()

await page.waitFor(3000) // just to make sure the new tab is opened
const pages = await browser.pages()
await pages[2].waitForXPath('//*[@id="prices"]/span') // wait for prices section to load

下一个问题是这一行：

const link = await (await page.$('//*[@id="prices"]/span')).click() // click on "Prices"

仅供参考：当你有一个唯一的 id 属性时，最好选择它而不是 XPath 选择器（当你需要使用它的专用方法时，XPath 是一个最坏的情况，比如 contains() 由于不断变化的动态选择器，但是这个元素有静态ID）

您可以使用以下方法修复此行：

const link = await page.$x('//*[@id="prices"]/span')
await link[0].click() // click on "Prices"

或者更简单的是：

await page.click('#prices')

注意：您还需要在这里申请 pages[n] 而不是 page。

---

编辑

在browser.pages() 之前尝试await page.waitForNavigation() 以确保您已加载所有三个选项卡。对我来说，它没有任何问题：

    const browser = await puppeteer.launch({
      headless: false
    })
    const page = await browser.newPage()

    //google.com
    await page.setExtraHTTPHeaders({ 'Accept-Language': 'en-US' })
    await page.goto('https://google.com')
    await page.type('input.gLFyf.gsfi', 'hotels in london')
    await page.keyboard.press('Enter')

    //search results
    await page.waitForXPath('//span[contains(text(),"View ")]')
    const btn1 = await page.$x('//span[contains(text(),"View ")]')
    await btn1[0].click()

    //list of hotels
    await page.waitForXPath('//span[contains(text(),"Learn more")]')

    //click on view prices
    const btn2 = await page.$x('//span[contains(text(),"Learn more")]')
    await btn2[0].click()

    await page.waitForNavigation()

    //get list of open tabs (does not include new tab)
    const pages = await browser.pages()

    //prints 2 although there are 3 tabs
    console.log(pages.length)

    // get the new page
    const page2 = pages[pages.length - 1]
    const url = await page2.evaluate(() => window.location.href)

    //prints url of page with list of hotels
    console.log(url)

    //takes screen shot of penultimate tab instead of new tab
    await page2.screenshot({ path: 'sample.png' })

    let providers = 0

    await browser.close()

输出：

3
https://www.google.com/travel/hotels/entity/CgoI0uWTwdT7-Y0lEAEaYEFHWklFZ2FXVXZpS2RQNGwyenhic3I5UHhJZEZFVDIwUmNrcTlfNmJUY01adXBZbElTTzE3YmpxclMtdVFNSkREeTlnbGJhTTFXYTEycGlsM1ZwdUdFdkJ4S1hpOXBhXw?g2lb=2502548%2C4258168%2C4270442%2C4306835%2C4308226%2C4317915%2C4322823%2C4328159%2C4344617%2C4358983%2C4367954%2C4371335%2C4401769%2C4403882%2C4414391%2C4416581%2C4418077%2C4421967%2C4270859%2C4284970%2C4291517%2C4412693&hl=hu&gl=hu&un=1&ap=aAE&q=hotels%20in%20london&rp=EN__hZ2wsYS5SxDZ_fO86Oyr1kUQq9TZmpT3hYhnENb4j4D5sOPDqQE4AUAASAKiAR1Mb25kb24sIEVneWVzw7xsdCBLaXLDoWx5c8OhZw&ictx=1&sa=X&ei=9r82X8HlKoaJgAf1pqfAAQ&utm_campaign=sharing&utm_medium=link&utm_source=htls&hrf=CgYI8KIEEAAiA0hVRioWCgcI5A8QCBgREgcI5A8QCBgSGAEoAFgBqgEqCgIIIRICCA8SAggVEgIIDRICCGcSAgheEgIIWxICCC8SAghaEgIIVBgBqgEPCgIIEhIDCJsBEgIIaBgBqgEOCgIIFBICCBsSAghSGAGqAQcKAwihAhgAqgEaCgIIHBICCAcSAghREgIIcxICCDYSAggpGAGqAQoKAgglEgIIehgBqgEiCgIIERICCCoSAghAEgIIOBICCFcSAggCEgIIfxICCCsYAaoBQAoCCC4SAgg8EgIIOxIDCIICEgMIgAISAggaEgIISBIDCIECEgIIPRIDCIEBEgMIgwESAghLEgIIDBIDCIkBGAGqAREKAwiuARIDCLABEgMIsgEYAaoBBwoDCKcBGACqAREKAwipARIDCKoBEgMIrAEYAaoBCwoDCOECEgIIYxgBqgEGCgIIRhgAqgEKCgIIUBICCE8YAaoBBwoDCMsBGACqAQYKAgg5GACqAQ4KAgg1EgIIExICCDIYAZIBAiAB
0

Answer 2

尝试父元素：

await page.click('#prices')