javascript - Firefox Streams API, StreamSaver, to stream dynamically generated data on client to client disk

Question

The context of this question is that of writing data from an indexedDB to the client disk and reading it back in again, offline performed all on the client.

Thus far, to write the database to disk, I have used the following steps in an indexedDB database, over an open cursor on an object store index:

1. Extracted data object from database.
2. Converted object to string with JSON.stringify.
3. Made new blob { type: 'text/csv' } of the JSON string.
4. Wrote blob to an array.
5. Moved cursor down one and repeated from step 1.
6. After the transaction completed successfully, a new blob of same type was made from the array of blobs and downloaded to disk with client's permission.

Of course, this can temporarily take up a lot of RAM when the blob is large.

Reading in from the client disk is accomplished through slicing the blob/text file into smaller segments before reading, parsing, and writing to the database. This method appears to not hold RAM in bringing in the large blob before slicing, as if the slices are read from the disk. I'm not saying that is the case, but just that I haven't observed a large jump in memory usage during the reading in and writing to the database as I do when generating the text blob to download to disk. Described further here.

Being a novice, I started to wonder if the component blobs of Step 3 above could be written to the client disk in turn as generated instead of being written to an array and holding the entire blob in RAM before downloading. According to this question, web extensions no longer have access the privileged javascript code needed to accomplish this.

This question appears to have addressed this to some degree but I don't understand what FileSaver and StreamSaver are exactly or the consequences of attempting to include either in an application, if possible, or how.

Also, MDN web docs describes a new writable stream in a Streams API, which makes me wonder if it is now possible to accomplish this without these two.

My goal is to stream the data to the client disk using javascript run on the client offline to avoid consuming RAM during the process when the single file would be quite large.

Could you please explain what FileSaver and StreamSaver are (not what they do) and whether they can be included in an offline application, and, if so, how and what the implications are?

Also, is it possible to code this without the use of either of these items with what is now available in the Streams API? I read that a service worker is needed also, when possible, but I don't think that can be done in an offline application or local html file.

Thank you.

Update

The streams API appears to be too new to use. I attempted to employ a suggestion given in a comment to a different question of mine that stated using a webRequest stream filter might work if could intercept a web request or download, modify its response header if needed to intitiate a download, and then write to it using the stream filter.

I tried for a couple days ( had a lot of basics to learn first ) and, although it appears to have been a good idea, concluded that it is not currently possible for these reasons, which I am repeating from that question.

One is that webRequest doesn't appear to intercept a downloads.download() function at all, or any download event; so, you can't intercept a download, and an event with a Content Disposition of 'attachment' is needed to even try to write to it with a stream. I could intercept a forced click to an anchor tag href but no other events fired beyond onBeforeRequest.

The other is that a response header can't be modified until an onHeadersReceived event, which means the fake URL has to return something. You can't just cancel it in onBeforeRequest. So, this wouldn't work offline. But, even if you let it process online to an existing URL that returns a reponse header, it won't accept a modification. I tried repeatedly to modify the response header and it just won't work. I tried an XMLHttpRequest GET and can intercept the events that fire but can't modify the response header; so, can't set Content Disposition to 'attachment', with or without file, to start a download. I can write to it but it's no good unless it is going to download what is written. It would be ok if the written content was going to a web page.

Also, if you redirect the URL along the way to anything other than a webRequest acceptable URL, the other events won't be interceptable. So, if redirect to an object URL or an extension URL in onBeforeRequest, you won't intercept the response headers stage in webRequest but can view them in the onreadystatechange event of the XMLHttpRequest.

So, the upshot is that it appears the response headers cannot be modified even though the MDN Web Docs say it is possible. And, this idea of using a webRequest stream filter to stream data generated on the client or extracted from an indexedDB database, as opposed to building one large blob for download, won't work because can't intercept a download or change the response headers to trigger a download into which to write via the stream filter.

It was an interesting idea though. I still wonder whether or not the download would remain 'open', so to speak, while the data was being written on the client and passed in blocks or chunks. Perhaps, if that part of the response headers that states how data is to be passed and received was modified also it would work.

For now, I am no longer pursuing this approach. One of the Web Docs or a bug records stated that it is planned to allow a data URL to be intercepted. Perhaps, for an offline download to the client, that would be preferrable to a fake URL.

If anyone gets this to work, please let us know. Thank you.

Answer 1

Waitting for answers