How to save memory

[schlagmichdoch]

First of all: Thanks a lot for this awesome, lightweight library!

I implemented it similarily to the create file demo and got an issue that is reproducable on the demo as well:
https://gildas-lormeau.github.io/zip.js/demos/demo-create-file.html

Using iOS Safari:

1. Add more than 208 MB of files:
   Everything works fine: All Files are progressed successfully
2. Click "Download"
   Website crashes: A reload is forced after a blue banner on the top of the page appears for half a second that reads: "A problem occurred with this webpage so it was reloaded"

• When I add less than 200 MB everything works perfectly.
• When it's more than 208 MB the page crashes when I try to download the Zip File.
• When I add ~1GB the page already crashes when progressing the files.

I guess it has to do with the maximum RAM a browser can occupy which is 384 MB on iOS Safari 15. Do you know of any workaround to that problem? Would it help to use the SplitDataReader class? I would appreciate any help!

[gildas-lormeau]

I don't have an iOS device and I was not able to reproduce the issue in the simulator bundled with Xcode.

I guess the Origin Private File System API could help, see https://webkit.org/blog/12257/the-file-system-access-api-with-origin-private-file-system/. However it's a bit tricky to use because FileSystemFileHandle#createSyncAccessHandle is only available in a Worker. The SplitDataReader would not be useful. However you could create a split zip with by using a generator ofWriter, it could help to save memory. There's a test showing how to use it here: https://github.com/gildas-lormeau/zip.js/blob/910cdf666382d5738a69c490b79483ce4b96df28/tests/all/test-split-zip.js and a similar demo here: https://plnkr.co/edit/SkTNcrTq8XL37mQo?preview.

[schlagmichdoch]

Thanks for the quick response!

So your idea is to create a zip file on the device itself via the Private File System API and only upload it then? Not sure whether that would reduce memory usage though.

I decided to remove zip.js on iOS completely and go for another approach so the browser does not have to save the same blob twice. As I have the same problem when using navigator.share with big files though, I simply won't allow files bigger than 200 MB on iOS.

If you're interested where your project got implemented into: Peer to peer file sharing https://pairdrop.net/ (Repo)

As this issue is not really zip.js specific I'll close it, thanks for helping though!

[gildas-lormeau]

I did not know that you needed to upload the data. In this case, you just needed to write your own Writer class similarly to the HttpRangeReader. It's approx. 20 lines of code.

[schlagmichdoch]

I needed to create a zip file and send it in chunks via webrtc to another peer. Files in a FileList from a multifile input somehow did not crash the website but creating a zip from them did. A zip file created by a custom Writer class would still be in RAM and would therefore crash the iOS Safari Browser if it's too big, wouldn't it?

[gildas-lormeau]

If you create your zip file by adding files sequentially with a Writer which uploads data directly to the server without storing it in a Blob then it should consume approximately 512 KB of RAM to create the zip file, cf. the chunkSize option when calling zip.configure().

[gildas-lormeau]

BTW, I don't know webrtc but if it can somehow support web streams (e.g. WritableStream or ReadeableSteam), you wouldn't even need to create the specialized Writer class.

[schlagmichdoch]

Hey @gildas-lormeau,

I'm getting back to this as I want to reduce the memory usage of my application once again.

I'd appreciate if a lot if you could help me once again!

Do I understand it right, that using the following snippet would not put the complete file in RAM or would I need to rewrite it to use streams to accomplish that?

async function getDownloadUrlOfZipFile(files, onZipProgressCallback) {
    const zipWriter = new zip.ZipWriter(new zip.BlobWriter("application/zip"), {
        bufferedWrite: true,
        level: 0
    });

    let bytesProcessed = 0;
    for (let i = 0; i < files.length; i++) {
        await zipWriter.add(
            files[i].name,
            new zip.BlobReader(files[i]),
            {
                onprogress: (progress) => onZipProgressCallback(bytesProcessed + progress)
            }
        );
        bytesProcessed += files[i].size;
    }

    return URL.createObjectURL(await zipWriter.close());
}

[gildas-lormeau]

The code looks fine. However, you should set bufferedWrite to false (or remove the option) because you're adding entries serially in the zip file. bufferedWrite should be set to true when adding entries in parallel and cancelling the addition of an entry without corrupting the zip file.

[schlagmichdoch]

Alright, thanks a lot!