通过 Heroku Scheduler (Node.js) 运行的抓取请求有一半时间失败

Question

我设置了一个非常基本的网络 scraper 来检查我祖父在 Costco.com 上的特定商品的库存。它在本地工作得很好，但是当我通过 Heroku 运行它时它失败了（似乎有 50% 的时间）。这是刮板的代码

const task = () => {

  // toggle so doesn't send message multiple times if continuously available
  let alreadyAvailable = false;

  let url = 'http://www.costco.com/Kirkland-Signature-Four-Piece-Urethane-Cover-Golf-Ball,-2-dozen.product.100310467.html';
  request(url, function(error, response, html){

    let $ = cheerio.load(html);

    if(error){
      throw new Error(error);
    }

    if ( $('#product-page #product-details #ctas #add-to-cart input[type="button"]')['0'].attribs.value === 'Out of Stock') {
      alreadyAvailable = false;
      console.log("still out of stock");
    } else {
      if (alreadyAvailable === false) {
        sendMessage();
        alreadyAvailable = true;
      }
    }

  });
};

这里是日志

2016-12-25T03:48:39.675549+00:00 heroku[scheduler.5440]: Starting process with command `node scraper.js`
2016-12-25T03:48:40.262503+00:00 heroku[scheduler.5440]: State changed from starting to up
2016-12-25T03:48:41.509416+00:00 app[scheduler.5440]: /app/scraper.js:34
2016-12-25T03:48:41.509432+00:00 app[scheduler.5440]:     if ( $('#product-page #product-details #ctas #add-to-cart input[type="button"]')['0'].attribs.value === 'Out of Stock') {
2016-12-25T03:48:41.509433+00:00 app[scheduler.5440]:                                                                                          ^
2016-12-25T03:48:41.509433+00:00 app[scheduler.5440]:
2016-12-25T03:48:41.509434+00:00 app[scheduler.5440]: TypeError: Cannot read property 'attribs' of undefined
2016-12-25T03:48:41.509434+00:00 app[scheduler.5440]:     at Request._callback (/app/scraper.js:34:90)
2016-12-25T03:48:41.509435+00:00 app[scheduler.5440]:     at Request.self.callback (/app/node_modules/request/request.js:186:22)
2016-12-25T03:48:41.509436+00:00 app[scheduler.5440]:     at emitTwo (events.js:106:13)
2016-12-25T03:48:41.509436+00:00 app[scheduler.5440]:     at Request.emit (events.js:191:7)
2016-12-25T03:48:41.509436+00:00 app[scheduler.5440]:     at Request.<anonymous> (/app/node_modules/request/request.js:1081:10)
2016-12-25T03:48:41.509437+00:00 app[scheduler.5440]:     at emitOne (events.js:96:13)
2016-12-25T03:48:41.509437+00:00 app[scheduler.5440]:     at Request.emit (events.js:188:7)
2016-12-25T03:48:41.509438+00:00 app[scheduler.5440]:     at IncomingMessage.<anonymous> (/app/node_modules/request/request.js:1001:12)
2016-12-25T03:48:41.509438+00:00 app[scheduler.5440]:     at IncomingMessage.g (events.js:291:16)
2016-12-25T03:48:41.509439+00:00 app[scheduler.5440]:     at emitNone (events.js:91:20)
2016-12-25T03:48:41.509439+00:00 app[scheduler.5440]:     at IncomingMessage.emit (events.js:185:7)
2016-12-25T03:48:41.509439+00:00 app[scheduler.5440]:     at endReadableNT (_stream_readable.js:974:12)
2016-12-25T03:48:41.509440+00:00 app[scheduler.5440]:     at _combinedTickCallback (internal/process/next_tick.js:74:11)
2016-12-25T03:48:41.509440+00:00 app[scheduler.5440]:     at process._tickCallback (internal/process/next_tick.js:98:9)
2016-12-25T03:48:41.560539+00:00 heroku[scheduler.5440]: State changed from up to complete
2016-12-25T03:48:41.550655+00:00 heroku[scheduler.5440]: Process exited with status 1
2016-12-25T03:58:42.438807+00:00 app[api]: Starting process with command `node scraper.js` by user scheduler@addons.heroku.com
2016-12-25T03:58:43.701468+00:00 heroku[scheduler.5038]: Starting process with command `node scraper.js`
2016-12-25T03:58:44.312279+00:00 heroku[scheduler.5038]: State changed from starting to up
2016-12-25T03:58:45.769564+00:00 app[scheduler.5038]: still out of stock
2016-12-25T03:58:45.827867+00:00 heroku[scheduler.5038]: State changed from up to complete
2016-12-25T03:58:45.814921+00:00 heroku[scheduler.5038]: Process exited with status 0

您可以看到，有时我会在 if 块中获取控制台日志，而在其他情况下，我会收到类型错误，因为它试图从不存在的 html 元素中读取属性。我在想这可能是一个异步问题，但我不确定如何解决它。我假设 Request 在获得所有 html 之前没有运行回调。

Answer 1

这里的问题是 Costco 的网站返回的内容。

您的代码在通过 Cheerio 解析 DOM 时失败。在您的情况下，这意味着您尝试抓取的特定 HTML 实际上并不存在（这就是错误的意思）。

这可能是由一些可能的原因造成的：

• Costco 呈现的页面与您预期的不同（可能它认为您是机器人，或者正在做一些限制）。
• 您正在接收重定向或某种其他类型的非错误 HTTP 状态代码，而您要查找的 HTML 在那里不存在。
• Costco 的网站会动态更改 HTML 以防止人们抓取。

如果我是你，我会怎么做：

• 让您的进程在任务运行时记录所有页面的 HTML。
• 下次进程失败时，将 Heroku 日志中的 HTML 复制到本地编辑器中，然后查看返回的内容。

我敢打赌你会感到惊讶 =)