How to use Transform stream to validate data in Node.js?

Question

I want to pipe data from my readable stream to a writable stream but validate in between.

In my case:

• Readable Stream: http response as a stream (Axios.post response as a stream to be more specific)
• Writable Stream: AWS S3

Axios.post response comes in XML format. So, it means the readable stream will read chunks that represent XML. I transform each chunk to string and check if <specificTag> (opening) and </specificTag> closing is available. Both these checks will be done in different or arbitrary chunks.

If both opening/closing tags are OK then I have to transfer the chunk to Writable stream.

I am coding like:

let openTagFound: boolean: false;
let closingTagFound: boolean: false;
readableStream.pipe(this.validateStreamData()).pipe(writableStream);

I have also defined _tranform method for validateStreamData() like:

private validateStreamData(): Transform {
  let data = '', transformStream = new Transform();
  let openTagFound: boolean = false;
  let closingTagFound: boolean = false;
  try {
    transformStream._transform = function (chunk, _encoding, done) {
      // Keep chunk in memory
      data += chunk.toString();
      if(!openTagFound) {
        // Check whether openTag e.g <specificTag> is found, if yes
        openTagFound = true;
      }
      if(!closingTagFound) {
        // parse the chunk using parser
        // Check whether closingTag e.g </specificTag> is found, if yes
        closingTagFound = true;
      }
      // we are not writing anything out at this
      // time, only at end during _flush
      // so we don't need to call push
      done();
    };
    transformStream._flush = function (done) {
      if(openTagFound && closingTagFound) {
        this.push(data);
      }
      done();
    };
    return transformStream;
  } catch (ex) {
    this.logger.error(ex);
    transformStream.end();
    throw Error(ex);
  }
}

Now, you can see that I am using a variable data at:

// Keep chunk in memory
data += chunk.toString();

I want to get rid of this. I do not want to utilize memory explicitly. The final goal is to get data from Axios.post and transfer it to AWS S3, only if my validation succeeds. If not, then it should not write to S3.

Any help is much appreciated.

Thanks in Advance!!!

question from:https://stackoverflow.com/questions/66061267/how-to-use-transform-stream-to-validate-data-in-node-js

Answer 1

So, What I finally did is, let the pipe end and kept some flags to check whether it is valid or invalid and then on('end') callback, if flag says invalid explicitly destroyed destination object.