Node.js: Best way to perform multiple async operations, then do something else?
NickName:dannybrown Ask DateTime:2014-10-09T08:46:15

Node.js: Best way to perform multiple async operations, then do something else?

In the following code I am trying to make multiple (around 10) HTTP requests and RSS parses in one go.

I am using the standard forEach construct on an array of URIs I need to access and parse the result of.

Code:

var articles;

feedsToFetch.forEach(function (feedUri)
{   
        feed(feedUri, function(err, feedArticles) 
        {
            if (err)
            {
                throw err;
            }
            else
            {
                articles = articles.concat(feedArticles);
            }
        });
 });

 // Code I want to run once all feedUris have been visited

I understand that when calling a function once I should be using a callback. However, the only way I can think of using a callback in this example would be to call a function which counts how many times it has been called and only continues when it has been called the same amount of times as feedsToFetch.length which seems hacky.

So my question is, what is the best way to handle this type of situation in node.js.

Preferably without any form of blocking! (I still want that blazing fast speed). Is it promises or something else?

Thanks, Danny

Copyright Notice:Content Author:「dannybrown」,Reproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/26268651/node-js-best-way-to-perform-multiple-async-operations-then-do-something-else

Answers
aarosil 2014-10-09T01:54:12

HACK-FREE SOLUTION\n\nPromises to be included in next JavaScript version\n\nThe popular Promise libraries give you an .all() method for this exact use case (waiting for a bunch of async calls to complete, then doing something else). It's the perfect match for your scenario\n\nBluebird also has .map(), which can take an array of values and use it to start a Promise chain. \n\nHere is an example using Bluebird .map(): \n\nvar Promise = require('bluebird');\nvar request = Promise.promisifyAll(require('request'));\n\nfunction processAllFeeds(feedsToFetch) { \n return Promise.map(feedsToFetch, function(feed){ \n // I renamed your 'feed' fn to 'processFeed'\n return processFeed(feed) \n })\n .then(function(articles){\n // 'articles' is now an array w/ results of all 'processFeed' calls\n // do something with all the results...\n })\n .catch(function(e){\n // feed server was down, etc\n })\n}\n\nfunction processFeed(feed) { \n // use the promisified version of 'get'\n return request.getAsync(feed.url)... \n}\n\n\nNotice also that you don't need to use closure here to accumulate the results.\n\nThe Bluebird API Docs are really well written too, with lots of examples, so it makes it easier to pick up. \n\nOnce I learned Promise pattern, it made life so much easier. I can't recommend it enough. \n\nAlso, here is a great article about different approaches to dealing with async functions using promises, the async module, and others\n\nHope this helps! ",


Mulan 2014-10-09T01:53:58

No hacks necessary\n\nI would recommend using the async module as it makes these kinds of things a lot easier.\n\nasync provides async.eachSeries as an async replacement for arr.forEach and allows you to pass a done callback function when it's complete. It will process each items in a series, just as forEach does. Also, it will conveniently bubble errors to your callback so that you don't have to have handler logic inside the loop. If you want/require parallel processing, you can use async.each.\n\nThere will be no blocking between the async.eachSeries call and the callback.\n\nasync.eachSeries(feedsToFetch, function(feedUri, done) {\n\n // call your async function\n feed(feedUri, function(err, feedArticles) {\n\n // if there's an error, \"bubble\" it to the callback\n if (err) return done(err);\n\n // your operation here;\n articles = articles.concat(feedArticles);\n\n // this task is done\n done();\n });\n}, function(err) {\n\n // errors generated in the loop above will be accessible here\n if (err) throw err;\n\n // we're all done!\n console.log(\"all done!\");\n});\n\n\nAlternatively, you could build an array of async operations and pass them to async.series. Series will process your results in a series (not parallel) and call the callback when each function is complete. The only reason to use this over async.eachSeries would be if you preferred the familiar arr.forEach syntax.\n\n// create an array of async tasks\nvar tasks = [];\n\nfeedsToFetch.forEach(function (feedUri) {\n\n // add each task to the task array\n tasks.push(function() {\n\n // your operations\n feed(feedUri, function(err, feedArticles) {\n if (err) throw err;\n articles = articles.concat(feedArticles);\n });\n });\n});\n\n// call async.series with the task array and callback\nasync.series(tasks, function() {\n console.log(\"done !\");\n});\n\n\n\n\nOr you can Roll Your Own™\n\nPerhaps you're feeling extra ambitious or maybe you don't want to rely upon the async dependency. Maybe you're just bored like I was. Anyway, I purposely copied the API of async.eachSeries to make it easy to understand how this works.\n\nOnce we remove the comments here, we have just 9 lines of code that can be reused for any array we want to process asynchronously! It will not modify the original array, errors can be sent to \"short circuit\" the iteration, and a separate callback can be used. It will also work on empty arrays. Quite a bit of functionality for just 9 lines :)\n\n// void asyncForEach(Array arr, Function iterator, Function callback)\n// * iterator(item, done) - done can be called with an err to shortcut to callback\n// * callback(done) - done recieves error if an iterator sent one\nfunction asyncForEach(arr, iterator, callback) {\n\n // create a cloned queue of arr\n var queue = arr.slice(0);\n\n // create a recursive iterator\n function next(err) {\n\n // if there's an error, bubble to callback\n if (err) return callback(err);\n\n // if the queue is empty, call the callback with no error\n if (queue.length === 0) return callback(null);\n\n // call the callback with our task\n // we pass `next` here so the task can let us know when to move on to the next task\n iterator(queue.shift(), next);\n }\n\n // start the loop;\n next();\n}\n\n\nNow let's create a sample async function to use with it. We'll fake the delay with a setTimeout of 500 ms here.\n\n// void sampleAsync(String uri, Function done)\n// * done receives message string after 500 ms\nfunction sampleAsync(uri, done) {\n\n // fake delay of 500 ms\n setTimeout(function() {\n\n // our operation\n // <= \"foo\"\n // => \"async foo !\"\n var message = [\"async\", uri, \"!\"].join(\" \");\n\n // call done with our result\n done(message);\n }, 500);\n}\n\n\nOk, let's see how they work !\n\ntasks = [\"cat\", \"hat\", \"wat\"];\n\nasyncForEach(tasks, function(uri, done) {\n sampleAsync(uri, function(message) {\n console.log(message);\n done();\n });\n}, function() {\n console.log(\"done\");\n});\n\n\nOutput (500 ms delay before each output)\n\nasync cat !\nasync hat !\nasync wat !\ndone\n",


More about “Node.js: Best way to perform multiple async operations, then do something else?” related questions

Node.js: Best way to perform multiple async operations, then do something else?

In the following code I am trying to make multiple (around 10) HTTP requests and RSS parses in one go. I am using the standard forEach construct on an array of URIs I need to access and parse the ...

Show Detail

How best to do async batched work in node.js?

Imagine I have a node.js program that has to get 100 rows of data from a server at a time, I want to process each row individually that is returned and when say only 10 rows are remaining to be pro...

Show Detail

Handle Async Operations in Node.js

I have been using the async package from NPM inside my Node.js service to run different operations in sequence. Example: when I would like to run functionA, functionB, functionC in sequence. I woul...

Show Detail

Extjs Promise with multiple async operations

Have an application where we have to perform multiple async operations (saving 2 stores and save one model) before we execute an callback. So only if all of them are successful we execute some spec...

Show Detail

Nested async operations in node.js

I am still quite new to writing code for Node.js (coming from PHP) and sometimes struggle to understand if async operations are working correctly, especially when there are multiple nested database...

Show Detail

What is the correct way to perform multiple operations on a text file , but executing it as a SINGLE HADOOP JOB

I want to perform few operations on a single text file. For eg: Task 1: Count all the words Task 2: Count words ending with specific characters Task 3: Count words occuring multiple times. What...

Show Detail

Node.js: serial operations with branching

I'm new to node.js and using it for a backend that takes data from syslog messages and stores it to a database. I've run into the following type of serial operations: 1. Query the database 2. If...

Show Detail

Preventing Multiple Async Operations

i am working on some code where a timer tick every seconds and and get data from Webservice and Shows in on WInform , What is the best approach to not run Multiple operations at same time ? oper...

Show Detail

Performing multiple Network Operations in a Service

quick question, Whats the best way to perform multiple network operations on a service. Consider something like a weather service where one has to update periodically. for each time, i call in this...

Show Detail

Perform async operations with Promise.all issue

I am new to nodejs and for me Promise seems to be a bit difficult to understand. I have a requirement where I have an array of objects (assume large data from database). I need to go through each d...

Show Detail