change default shard size to 1GB #357
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spencerschrock
commented
Feb 26, 2025
There was a problem hiding this comment.
Lots of diminishing returns on speed after 50 MiB shard size, but manifest size still decreases until we reach the approximate size of the normal file-based manifest (but at that point having shards at all is pointless).
Given HuggingFace has a default shard size of 5GB (5_000_000_000), I'm assuming something in the 1-2G(i?)B range makes sense here. Happy to do either
mihaimaruseac
previously approved these changes
Feb 26, 2025
The values was based on the benchmarks in b2ddcc0. The exact speed improvement depends on the size of the models being serialized and of the individual files in the model. Speed improvements ranged from a 5% improvement to a 87% improvement. Manifest size is also influenced by shard size (and the model / individual file size). Increasing shard size from ~1MB to 1GB decreases manifest size by 3 orders of magnitude (99.9% reduction). Signed-off-by: Spencer Schrock <[email protected]>
Signed-off-by: Spencer Schrock <[email protected]>
spencerschrock
force-pushed
the
shard-opt
branch
from
39441d9 to
541cae9
Compare
They're roughly equal in performance, any differences either vary model to model or run to run. However multiples of 1000 are slightly easier for humans to visualize in things like shard names. Signed-off-by: Spencer Schrock <[email protected]>
spencerschrock
force-pushed
the
shard-opt
branch
from
97e798a to
05a63b3
Compare
mihaimaruseac
approved these changes
Feb 27, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
The values was based on the benchmarks in #356.
The exact speed improvement depends on the size of the models being
serialized and of the individual files in the model. Speed improvements
ranged from a 5% improvement to a 87% improvement.
Manifest size is also influenced by shard size (and the model /
individual file size). Increasing shard size from ~1MB to 1GB decreases
manifest size by 3 orders of magnitude (99.9% reduction).
Release Note
Documentation
NONE