javascript - How to use koa app callback internally without an http server?

Question

A have an js app based on koa and working on server-side rendering. In the rest of the app I'm loading some scripts from the same server which is running the SSR. While the loading could go through standard HTTP request, it would be unnecessarily slow. So I wold like to simply call the app.callback() provided by koa to resolve the request to string of requested file. For SSR I'm using jsdom, which loads files and process them as a browser.

The issue is that I'm unable to get result, resp. body of "server response". I've tried to mock server request and server response and then just read sockets. I'm not sure whether this is the best way, but it seemingly could work. However, when I run the app I get only two chunks - buffers of string for a large froont-end file. First chunk is headers. Second is parrt of the code but only 64kB. I cannot get next chunk. It seems like the mocked writable stream gets halted. Maybe it is getting corked, but I'm not sure why and whether at all.

If I try to load smaller script than 64kB, the request is resolved correctly and stream is finished.

const jsdom = require("jsdom");
const { JSDOM } = jsdom;
const { Writable } = require("stream");
const http = require("http");
const fetch = async (url, options) => {
        const response = [];
        let finishedReolve;
        const finished = new Promise(resolve => (finishedReolve = resolve));
        const socket = new Writable({
                write: (data, encoding, cb) => {
                        console.log("write", data);
                        // outputs 2 times - 1. headers, 2. 64kB part of requested code
                        response.push(data);
                        cb();
                        return true;
                },
                destroy(err, cb) {
                        // Never gets called
                        console.log("destroy", err);
                        cb();
                        finishedReolve();
                },
                final(cb) {
                        // Never gets called
                        console.log("final");
                        cb();
                        finishedReolve();
                },
        });
        const req = new http.IncomingMessage(socket);
        req.method = "GET";
        const parsedURL = new URL(url);
        req.url = parsedURL.pathname;
        const res = new http.ServerResponse(req);
        res.assignSocket(req.socket);
        res.on("prefinish", () => {
                // Never gets called
                finishedReolve();
        });
        await this.callback()(req, res);
        await finished;
        return response[0];
};
class CustomResourceLoader extends jsdom.ResourceLoader {
        fetch(url, options) {
                return fetch(url, options);
        }
}
const dom = await JSDOM.fromFile(index_html_path, {
        url: domain + ctx.req.url,
        runScripts: "dangerously",
        resources: new CustomResourceLoader(),
});

What's the issue with the stream? Is there any other way how to use app.callback() to get output of the koa app?

question from:https://stackoverflow.com/questions/65937718/how-to-use-koa-app-callback-internally-without-an-http-server

Answer 1

The issue was, that the readable stream was sending chunks of 64kB, but my writable stream had highWaterMark set to default 16kB. Which (probably) caused the readable stream to wait for drain after it wrote more data that writable could process, which was after first 64kB chunk.

Tho solution was to set highWaterMark to more than it is for the readable stream, so e.g.

highWaterMark: 1024 * 64 * 2,

So the socket looks like this:

const socket = new Writable({
        highWaterMark: 1024 * 64 * 2,
        write: (data, encoding, cb) => {
                response.push(data);
                cb();
                return true;
        },
        destroy(err, cb) {
                console.log("destroy", err);
                cb();
                finishedReolve();
        },
        final(cb) {
                console.log("final");
                cb();
                finishedReolve();
        },
});

Also to finish, I need to listen to that weird event prefinish.