javascript - Monitoring pending async operations in Node.js promised environment

Question

I have built in Node.js a very stable robot app that basically sends requests continuously to an API. To make sure nothing can go wrong, I handle any possible error and I have set timeouts for promises that could take too long to resolve...

Now, I would like to improve the app by removing my safety nets, and monitoring async operations to find any kind of "async leaking", eg promises pending forever or any strange outcome I'm not aware of (that's the point of my question).

Are there any tools meant to monitor the Node.js async flow ? For instance, getting the total amount of pending promises in the process at a given time ? Or getting a warning if any promise has been pending for more than a given time, and tracking that promise ?

If that may guide answers, here are the modules I use :

// Bluebird (promises)
var Promise = require("bluebird");

// Mongoose with promises
var mongoose = require('mongoose');
mongoose.Promise = require('bluebird');

// Rate limiter with promises
var Bottleneck = require("bottleneck");

// Promisified requests
var request = require('request-promise');

Answer 1

Sounds like you might need some kind of job management.

I've been trying kue lately to manage asynchronous jobs and it's pretty good.

It has an API for starting and running jobs. Each job can report it's progress. It comes with a builtin job dashboard that shows you the running jobs and their progress. It has an extensive set of events so that you can monitor the status of each job.

You need to install Redis to use this, but that's not difficult.

It doesn't support promises, but you can see in my code example below that it's easy to start a job and then wrap it in a promise so that you can await job completion:

const queue = kue.createQueue();

queue.process("my-job", 1, (job, done) => {

    const result = ... some result ...

    // Run your job here.

    if (an error occurs) {
        done(err); // Job failure.
        return;
    }

    done(null, result); // Job success
});

function runMyJob () {
    return new Promise((resolve, reject) => {
        const job = queue.create("my-job").save();

        job.on('complete', result => {
            resolve(result); // Job has completed successfully, resolve the promise.
        });

        job.on("failed", err => {
            reject(err); // Job has failed, reject the promise.
        });        
    });
};

runMyJob()
    .then(() => {
        console.log("Job completed.")
    })
    .catch(err => {
        console.error("Job failed.");
    });

It's fairly easy to make this work in parallel over multiple CPUs using the Node.js cluster module.

Read more on the kue web page.