Allow compressing arrays larger than 4 GB #105
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Work toward fixing #95
Before this PR
Zlib.Writer
andZlib.crc32
could not handle more than 4 GB at a time because of a limitation of the zlib C library.This PR runs the compressor on chunks of the input data in a loop to avoid the issue.
To test this requires at least 8 GB of memory, so I put the tests in a separate file,
"bigtests.jl"
.To run these tests run:
Or just run
"bigtests.jl"
as a julia script in a project with ZipFile.