【发布时间】:2020-03-22 19:22:00
【问题描述】:
我正在用 puppeteer 构建一个简单的网络爬虫,假设根据来自前端请求的城市名称从 pexels.com 获取图像。
有些城市在网站上没有图片,所以我尝试通过将其中的第一张图片发送到前端来搜索这些案例。
在本地主机上工作时,所有工作都在预期中,有图片的城市要么没有,要么 { catch },但上传后端到 Heroku 时,只有伦敦和马德里等城市有图片,但特拉维夫等城市没有。
-----------$ heroku 日志 --tail
-11-27T09:09:19.340459+00:00 app[web.1]: (node:36) UnhandledPromiseRejectionWarning: Error:
Evaluation failed: TypeError: Cannot read property 'getAttribute' of null
2019-11-27T09:09:19.340473+00:00 app[web.1]: at __puppeteer_evaluation_script__:4:11
2019-11-27T09:09:19.340476+00:00 app[web.1]: at ExecutionContext._evaluateInternal
(/app/node_modules/puppeteer/lib/ExecutionContext.js:122:13)
2019-11-27T09:09:19.340479+00:00 app[web.1]: at processTicksAndRejections
(internal/process/task_queues.js:93:5)
2019-11-27T09:09:19.340481+00:00 app[web.1]: at async ExecutionContext.evaluate
(/app/node_modules/puppeteer/lib/ExecutionContext.js:48:12)
2019-11-27T09:09:19.340483+00:00 app[web.1]: at async /app/controllers/webCrawler.js:13:17
2019-11-27T09:09:19.340486+00:00 app[web.1]: -- ASYNC --
2019-11-27T09:09:19.340489+00:00 app[web.1]: at ExecutionContext.<anonymous>
(/app/node_modules/puppeteer/lib/helper.js:111:15)
2019-11-27T09:09:19.340490+00:00 app[web.1]: at DOMWorld.evaluate
(/app/node_modules/puppeteer/lib/DOMWorld.js:112:20)
2019-11-27T09:09:19.340493+00:00 app[web.1]: at processTicksAndRejections
(internal/process/task_queues.js:93:5)
2019-11-27T09:09:19.340496+00:00 app[web.1]: -- ASYNC --
2019-11-27T09:09:19.340498+00:00 app[web.1]: at Frame.<anonymous>
(/app/node_modules/puppeteer/lib/helper.js:111:15)
2019-11-27T09:09:19.340500+00:00 app[web.1]: at Page.evaluate
(/app/node_modules/puppeteer/lib/Page.js:863:43)
2019-11-27T09:09:19.340502+00:00 app[web.1]: at Page.<anonymous>
(/app/node_modules/puppeteer/lib/helper.js:112:23)
2019-11-27T09:09:19.340504+00:00 app[web.1]: at /app/controllers/webCrawler.js:14:8
2019-11-27T09:09:19.340506+00:00 app[web.1]: at processTicksAndRejections
(internal/process/task_queues.js:93:5)
2019-11-27T09:09:19.340590+00:00 app[web.1]: (node:36) UnhandledPromiseRejectionWarning:
Unhandled promise rejection. This error originated either by throwing inside of an async
function without a catch block, or by
rejecting a promise which was not handled with
.catch(). (rejection id: 3)
------------------------------------我的代码
const puppeteer = require('puppeteer');
const handleScrape = (req, res) => {
(async () => {
const browser = await puppeteer.launch({
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
]
});
const page = await browser.newPage();
await page.goto(
`https://www.pexels.com/search/${req.body.name.replace(' ', '%20')}/`
);
const img = await page.evaluate(() => {
try {
return document
.querySelector('a.js-photo-link img')
.getAttribute('data-tiny-src');
} catch {
let url = document
.querySelector('a.sponsored-photos__photo__link')
.getAttribute('style');
return url.slice(22, url.length - 1);
}
});
res.json(img);
await browser.close();
})();
};
module.exports = {
handleScrape
};
我确实尝试以几种方式进行捕获(使用 { then } 和 catch() 而不是 catch{},捕获整个评估等...)但没有用,无论如何无法理解为什么它在本地机器上运行,而不是在部署中。
提前致谢:)
** 编辑 **
与UnhandledPromiseRejectionWarning: This error originated either by throwing inside of an async function without a catch block 不同,如果该页面上没有图片库,我想在 catch 方法中发送响应,我想搜索建议,正如我在帖子中提到的那样,奇怪的是所有工作都在开发中模式,甚至有硒作为见证;-)
【问题讨论】:
-
不,我尝试在 catch 方法中实际发送响应,如果该页面上没有 img 画廊,我想放弃建议,
标签: node.js promise async-await try-catch puppeteer