【问题标题】:How to properly structure async Node code如何正确构建异步节点代码
【发布时间】:2015-03-08 08:02:04
【问题描述】:

我正在尝试处理我拥有的“事件”数据库,并根据每个事件的位置、半径、开始时间和结束时间从 Instagram API 中提取照片。我在我的 Node 服务器上设置了以下代码,但它的行为与我预期的不同。

当我运行此代码时,我看到的第一件事是为每个事件打印sending request to Instagram for [name] with min_timestamp: [timestamp]。我没想到会这样。我本来希望看到这条线记录了第一个事件,然后一遍又一遍地用新的时间戳更新,直到该事件到达其结束时间。然后是事件 2,遍历时间戳,等等。

我最终会为每个事件一遍又一遍地重复相同的照片块。就好像我的代码一遍又一遍地向 Instagram 发出一个请求(带有初始时间戳)然后停止。

关于我的时间戳变量的注意事项:对于每个事件,我将我的 minTimestamp 变量设置为最初等于数据库中的event.start。这在发送到 Instagram 的请求中使用。 Instagram 最多返回 20 张照片给我。每张照片都有一个 created_time 变量。我获取了最近的 created_time 变量并将我的 minTimestamp 变量设置为等于它(minTimestamp = images[0].created_time;),以便我发送到 Instagram 的下一个请求(以获取接下来的 20 张照片)。这一直持续到 minTimestamp 不再小于 endTimestamp (event.end 来自我的数据库的那个事件)。

server.js 代码:

// modules =================================================
var express        = require('express.io');
var app            = express();
var port           = process.env.PORT || 6060;
var io             = require('socket.io').listen(app.listen(port));
var request        = require('request');
var Instagram      = require('instagram-node-lib');
var mongoose       = require('mongoose');
var async          = require('async');
var bodyParser     = require('body-parser');
var methodOverride = require('method-override');
var db             = require('./config/db');
var Event          = require('./app/models/event');

// configuration ===========================================
mongoose.connect(db.url); // connect to our mongoDB database

// get all data/stuff of the body (POST) parameters
app.use(bodyParser.json()); // parse application/json 
app.use(bodyParser.json({ type: 'application/vnd.api+json' })); // parse     application/vnd.api+json as json
app.use(bodyParser.urlencoded({ extended: true })); // parse application/x-www-form- urlencoded

app.use(methodOverride('X-HTTP-Method-Override')); // override with the X-HTTP-Method- Override header in the request. simulate DELETE/PUT
app.use(express.static(__dirname + '/public')); // set the static files location  /public/img will be /img for users

var baseUrl = 'https://api.instagram.com/v1/media/search?lat=';
var clientId = CLIENT-ID;

Event.find({}, function(err, events) {

    async.eachSeries(events, function(event, callback) {

      var name = event.event;
      var latitude = event.latitude;
      var longitude = event.longitude;
      var distance = event.radius;
      var minTimestamp = Math.floor(new Date(event.start).getTime()/1000);
      var endTimestamp = Math.floor(new Date(event.end).getTime()/1000);

      async.whilst(
        function () { return minTimestamp < Math.floor(Date.now() / 1000) && minTimestamp < endTimestamp; },
        function(callback) {
          console.log('sending request to Instagram for ' + name + ' with min_timestamp: ' + minTimestamp);
          request(baseUrl + latitude + '&lng=' + longitude + '&distance=' + distance + '&min_timestamp=' + minTimestamp + '&client_id=' + clientId,
            function (error, response, body) {
              if (error) { 
                console.log('error');
                return;
              }

                //JSON object with all the info about the image
                var imageJson = JSON.parse(body);
                var images = imageJson.data;
                var numImages = images.length;
                console.log(numImages + ' images returned with starting time ' + images[(numImages - 1)].created_time + ' and ending time ' + images[0].created_time);

                async.eachSeries(images, function(image, callback) {

                  //Save the new object to DB
                  Event.findOneAndUpdate( { $and: [{latitude: latitude}, {radius: distance}] }, { $push: {'photos':
                      { img: image.images.standard_resolution.url,
                        link: image.link,
                        username: image.user.username,
                        profile: image.user.profile_picture,
                        text: image.caption ? image.caption.text : '',
                        longitude: image.location.longitude,
                        latitude: image.location.latitude
                      }}},
                      { safe: true, upsert: false },
                      function(err, model) {
                          console.log(err);
                      }
                  );
                  console.log(numImages + ' images saved to db');
                  callback();
              }, function(err){
                  // if any of the file processing produced an error, err would equal that error
                  if( err ) {
                    // One of the iterations produced an error.
                    // All processing will now stop.
                    console.log('Images failed to process');
                  } else {
                    console.log('All images have been processed successfully');
                  }
              });

              minTimestamp = images[0].created_time;
              console.log('min_timestamp incremented to: ' + minTimestamp);
            }
          );
        },
        function (err) {

        }
      );
      callback();
    }, function(err){
        // if any of the file processing produced an error, err would equal that error
        if( err ) {
          // One of the iterations produced an error.
          // All processing will now stop.
          console.log('An event failed to process');
        } else {
          console.log('All events have been processed successfully');
        }
    });
});

// routes ==================================================
require('./app/routes')(app); // configure our routes

// start app ===============================================
console.log('Magic happens on port ' + port);           // shoutout to the user
exports = module.exports = app;                         // expose app

【问题讨论】:

  • 对不起,我需要睡一会儿。你还在纠结这个吗?如果是这样,我想再看看它,因为这是一个有趣的问题,也许我们可以解决这个问题。您能否编辑您的问题,详细说明 min_timestamp 业务的运作方式?我无法弄清楚为什么照片会增加它以及为什么它最终等于 end_timestamp。
  • “相同的照片块”是什么意思 - 只是重复第一个事件的照片还是所有照片都一样?
  • 我真的没有任何进展,无法对其进行测试。要问你的问题太多了。但是有一点有点奇怪:while() 位中的 main 函数没有 callback(),我认为这是必要的。
  • 感谢您对此的关注和努力。我将编辑我的问题以添加有关时间戳变量的一些详细信息。此外,我正在尝试建立的网站是mobseen.it。如果您单击其中一个事件,您会看到我所说的照片重复是什么意思。每个位置似乎都获得了第一组正确的照片,但随后它们被一遍又一遍地添加。
  • 好的,所以这大概意味着 min_timestamp 永远不会更新,对吧?所以我的下一个问题是:如果 Instagram 发回 20 张照片,为什么要使用 FIRST 照片来增加 min_timestamp? (minTimestamp = images[0].created_time;) 如果它们按时间顺序排列,你不应该使用 minTimestamp = images[(numImages - 1)].created_time; ?还是它们的时间顺序相反?

标签: javascript node.js mongodb asynchronous mongoose


【解决方案1】:

答案是你错过了 while 位的 callback()。

这里有一些代码来说明:

var async = require('async');

var minTimestamp = 1;
var endTimestamp = 10;

async.whilst(
    function () { minTimestamp < endTimestamp; },
    function(callback) {
      console.log('sending request to Instagram for name with min_timestamp: ' + minTimestamp);

      minTimestamp = minTimestamp + 1;
      console.log('min_timestamp incremented to: ' + minTimestamp);
      callback();
    },
    function (err) {
        if(err){
                throw err;
        }
    }
  );

如果我在没有 callback() 的情况下运行它,我会得到以下输出:

sending request to Instagram for name with min_timestamp: 1
min_timestamp incremented to: 2

如果我把callback() 放回去,我会得到这个:

sending request to Instagram for name with min_timestamp: 1
min_timestamp incremented to: 2
sending request to Instagram for name with min_timestamp: 2
min_timestamp incremented to: 3
sending request to Instagram for name with min_timestamp: 3
min_timestamp incremented to: 4
sending request to Instagram for name with min_timestamp: 4
min_timestamp incremented to: 5
sending request to Instagram for name with min_timestamp: 5
min_timestamp incremented to: 6
sending request to Instagram for name with min_timestamp: 6
min_timestamp incremented to: 7
sending request to Instagram for name with min_timestamp: 7
min_timestamp incremented to: 8
sending request to Instagram for name with min_timestamp: 8
min_timestamp incremented to: 9
sending request to Instagram for name with min_timestamp: 9
min_timestamp incremented to: 10

因此在这里放置一个回调():

      minTimestamp = images[0].created_time;
      console.log('min_timestamp incremented to: ' + minTimestamp);
      callback(); //missing callback
    }
 );
 },

【讨论】:

  • 非常感谢。看起来它应该解决这个问题。我现在正在路上,但一旦有机会就会实施,让你知道进展如何。这似乎是我过去读到的关于 Node 的经典“回调地狱”情况。一段时间后,要跟踪它们确实变得非常困难。
  • 是的,请告诉我,希望就是这样。我赞成你的问题,看它很有趣。我在这里删除了我的评论,其中提到了我的错误答案,也许你应该删除你关于意外标识符的评论,因为它可能会让阅读本文的人感到困惑。
  • P.S.喜欢 mobseen 网站!好主意和好的实现。
  • 不幸的是,这个解决方案不起作用。当我使用硬编码值(如上面的示例)增加 minTimestamp 时,例如minTimestamp += 1000,它可以完美运行。一旦我将其更改为minTimestamp = images[0].created_time,它就会中断。 “中断”是指为每个事件发送一次请求,时间戳增加,但不再发送请求。奇怪,我正试图弄清楚可能导致这种情况的原因。
  • 好吧,正如我在上面的评论中提到的,也许图像是相反的顺序? - 最新的优先?如果是这样,您将立即将 minTimestamp 设置为最新时间,因此一切都会停止。我能想到的就这些了。
【解决方案2】:
// modules =================================================
var express        = require('express.io');
var app            = express();
var port           = process.env.PORT || 6060;
var io             = require('socket.io').listen(app.listen(port));
var request        = require('request');
var Instagram      = require('instagram-node-lib');
var mongoose       = require('mongoose');
var async          = require('async');
var bodyParser     = require('body-parser');
var methodOverride = require('method-override');
var db             = require('./config/db');
var Event          = require('./app/models/event');

// configuration ===========================================
mongoose.connect(db.url); // connect to our mongoDB database

// get all data/stuff of the body (POST) parameters
app.use(bodyParser.json()); // parse application/json 
app.use(bodyParser.json({ type: 'application/vnd.api+json' })); // parse        application/vnd.api+json as json
app.use(bodyParser.urlencoded({ extended: true })); // parse application/x-www-form- urlencoded

app.use(methodOverride('X-HTTP-Method-Override')); // override with the X-HTTP-Method- Override  header in the request. simulate DELETE/PUT
app.use(express.static(__dirname + '/public')); // set the static files location  /public/img  will be /img for users

var baseUrl = 'https://api.instagram.com/v1/media/search?lat=';
var clientId = CLIENT-ID;

Event.find({}, function(err, events) {

  async.eachSeries(events, function(event, seriesCallback) {

    var name = event.event;
    var latitude = event.latitude;
    var longitude = event.longitude;
    var distance = event.radius;
    var minTimestamp = Math.floor(new Date(event.start).getTime()/1000);
    var endTimestamp = Math.floor(new Date(event.end).getTime()/1000);

    async.whilst(
      function () { return minTimestamp < Math.floor(Date.now() / 1000) && minTimestamp <  endTimestamp; },
        function(requestFinishedCallback) {
          console.log('sending request to Instagram for ' + name + ' with min_timestamp: ' + minTimestamp);
          request(baseUrl + latitude + '&lng=' + longitude + '&distance=' + distance + '&min_timestamp=' + minTimestamp + '&client_id=' + clientId,
            function (error, response, body) {
              if (error) { 
                console.log('error');
                return;
              }

              //JSON object with all the info about the image
              var imageJson = JSON.parse(body);
              var images = imageJson.data;
              var numImages = images.length;
              console.log(numImages + ' images returned with starting time ' + images[(numImages - 1)].created_time + ' and ending time ' + images[0].created_time);

              async.eachSeries(images, function(image, imageFinishedCallback) {

                //Save the new object to DB
                Event.findOneAndUpdate( { $and: [{latitude: latitude}, {radius: distance}] }, { $push: {'photos':
                  { img: image.images.standard_resolution.url,
                    link: image.link,
                    username: image.user.username,
                    profile: image.user.profile_picture,
                    text: image.caption ? image.caption.text : '',
                    longitude: image.location.longitude,
                    latitude: image.location.latitude
                  }}},
                  { safe: true, upsert: false },
                  function(err, model) {
                    console.log(err);
                    console.log('Image processed');  
                    imageFinishedCallback();
                  }
                );

              }, function(err){
                   // if any of the image processing produced an error, err would equal that error
                   if( err ) {
                     // One of the iterations produced an error.
                     // All processing will now stop.
                     console.log('Images failed to process');
                   } else {
                     minTimestamp = images[0].created_time;
                     console.log(numImages + ' images have been processed successfully and min_timestamp has been incremented to: ' + minTimestamp);
                     requestFinishedCallback();
               }
                 });
                }
              );
           }, function(err){
                   // if any of the image processing produced an error, err would equal that error
                   if( err ) {
                     // One of the iterations produced an error.
                     // All processing will now stop.
                     console.log('Event failed to process');
                   } else {
                        console.log(name + ' has been fully processed successfully with final min_timestamp of: ' + minTimestamp);
                   }
                   seriesCallback();
                });
             }, function(err){
                    // if any of the image processing produced an error, err would equal that error
                    if( err ) {
                      // One of the iterations produced an error.
                      // All processing will now stop.
                      console.log('Something failed to process');
                    } else {
                         console.log('All events have been processed successfully');
                    }
                 });
}); 

// routes ==================================================
require('./app/routes')(app); // configure our routes

// start app ===============================================
console.log('Magic happens on port ' + port);           // shoutout to the user
exports = module.exports = app; 

【讨论】:

  • 谢谢发帖,我明天去好好看看。
  • @mwarren,解决了最后剩下的问题。增量minTimestamp = images[0].created_time; 需要在callback() 之前发生。但是,此时images[0].created_time 不再可用,因为images 仅存在于从 Instagram 返回的数据中。需要找到一种方法来获取此时间戳并将其传递给 minTimestamp 以便在回调之前进行更新。
  • 我将更新您的答案,否则它太复杂而无法显示:我们忘记了所有异步函数都必须在调用回调之前完成。我已经重命名并移动了所有回调,以显示我认为它们应该是什么。
  • 我昨晚在您的帮助下进行了一个简短的测试,并且运行良好。今晚我将重新参加所有“活动”,并在下面记录您的答案。再次感谢,异步概念和在哪里放置回调一直是我的绊脚石,但我觉得我现在对它有了更强大的处理能力。
  • 很好用!我们真的是几个菜鸟,看代码好像它是同步运行的。编写一段有趣的代码非常棒,感谢分享。
猜你喜欢
  • 2017-01-24
  • 2014-09-18
  • 1970-01-01
  • 2016-01-08
  • 2019-03-12
  • 2013-07-01
  • 1970-01-01
  • 2011-08-30
  • 1970-01-01
相关资源
最近更新 更多