Array.forEach 中的 Node.Js 异步请求在写入 json 文件之前未完成

Question

我正在制作一个网络爬虫 Node.js 应用程序，该应用程序可以从各种 url 中获取职位描述文本。我目前有一个 名为 jobObj 的作业对象数组，并且 代码循环遍历每个url，发出对html的请求，使用cheerio模块加载然后最后用jobName和jobDesc键创建一个新对象并将其推送到一个新的对象数组中然后将其写入为 json 文件....

目前所有这些都有效，但是写入的 json 文件的完整性是非常随机的，并且通常只包含一个完整的工作帐户。我认为这可能是由于 forEach 循环比异步请求函数完成得更快，因此导致在请求回调完成之前执行 fs.writefile。我添加了一个计数器来监视请求处于哪个阶段，并且只写入一次 json 文件 counter===jobObj.length 但 json 文件仍然没有完全完成。

我是 node.js 的新手，如果有人能指出我的错误，将不胜感激！

var jobObj = [
{
id:1, 
url:"https://www.indeed.co.uk/cmp/Daffodil-IT/jobs/Lead-Junior-Website-Developer-59ea7d446bdf1253?q=Junior+Web+Developer&vjs=3",
}, 
{
id:2, 
url:"https://www.indeed.co.uk/cmp/Crush-Design/jobs/Middleweight-Web-Developer-541331b7885c03cf?q=Web+Developer&vjs=3",
},
{
id:3,
url:"https://www.indeed.co.uk/cmp/Monigold-Solutions/jobs/Graduate-Web-Software-Engineer-a5787dc322c0ca36?q=Web+Developer&vjs=3",
},
{
id:4,
url:"https://www.indeed.co.uk/cmp/ZOO-DIGITAL-GROUP-PLC/jobs/Web-Developer-5cdde1c3b0b7b8d0?q=Web+Developer&vjs=3",
},
{
id:5,
url:"https://www.indeed.co.uk/viewjob?jk=9cc3d8c637c41067&q=Web+Developer&l=Sheffield&tk=1cf5di52e9u0ocam&from=web&vjs=3",
}
];


app.get('/myform', function(req, res){

res.send("<h1>" + `scanning ${jobObj.length} urls for job description text` + "</h1>");
//make assign input form data to node "url" variable

//Compnonents for a request counter
var jobs = new Array;
function scrapeFinished(){console.log("all websites scraped!");};
var itemsProcessed = 0;   

jobObj.forEach(function(item){

    request(item.url, function(err, res, html){

        if(!err){

            var $ = cheerio.load(html);
            var newJob = new Object;
            $('#job_summary').each(function(){
                var data = $(this);
                var textout = data.text();
                newJob.jobDesc = textout;    
            });

            $('.jobtitle').each(function(){
                var data = $(this);
                var jobtitle = data.text();
                newJob.jobName = jobtitle;
            });

            jobs.push(newJob);
            itemsProcessed++;
            console.log(item.url + " scraped");  

            if(itemsProcessed === jobObj.length){
            scrapeFinished();

            fs.writeFile('output.json', JSON.stringify(jobs, null, "\t"), function(err){ 
            if(!err){console.log("output.json file written")}
            })
            }
        }      
    })           
})


})

最后这就是我在fs.writefile得到的结果

[
{},
{
    "jobDesc": "We are a successful design and digital agency that works with some great clients on a wide range of digital projects.We simply need more developers to join our great team to deliver even more great work.The projects we work on are all php based, typically built using WordPress, Laravel or flat html.We are seen as a premium agency because of the quality and complexity of the work we do.That means you will have to do more that just manipulate a theme - you will have to code. But you will be given the space, time and support to do so.We want you to be proud of the work you do, because the reputation of the agency need you to be.Key skills we will want you to bringCSS (CSS3) & HTMLAt least some knowledge of MySQL and JavaScript.At least some knowledge PHP (seniors will be tested)PhotoshopWhat you will want that we can giveA good place to work with a friendly teamA chance to develop your coding craftA decent range of projects to challenge yourselfA senior developer on hand to coach and adviseA successful company with an optimistic outlook, growth plans and a secure futureExactly how much experience you have can vary, but you must have some. And the more experience you have, the more we will pay you.We are based in offices we own in the centre of Chesterfield will two staff that do the short commute from Sheffield.If you think this job sounds interesting, we would love to hear from you, please apply!(though not agencies please)Job Type: Full-timeSalary: £22,000.00 to £30,000.00 /yearExperience:development: 2 years",
    "jobName": "Middleweight web developer"
},
{},
{},
{}
]

Answer 1

forEach 以 同步 方式运行，所以无论它会花多少时间来报废网页，它都不会为下一个数组项运行。在这里可能引起麻烦的是您的代码可能会在废弃网页之前将对象推送到数组中。这就是你得到一个空对象的原因。您所做的是在程序完成访问 URL 后推送对象。 做这样的事情

 jobObj.forEach(function (item) {

    var newJob = new Object;
    request(item.url, function (err, res, html) {
        // Scrap URL and save values in newJob 
    });
            jobs.push(newJob);
});

如果您的代码在完成请求之前仍然推送一个空对象，请考虑使用Async 模块。