Skip to content

[Query-Frontend] Add dynamic query vertical sharding #6678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Apr 28, 2025
5 changes: 5 additions & 0 deletions docs/configuration/config-file-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -4333,6 +4333,11 @@ dynamic_query_splits:
# CLI flag: -querier.max-fetched-data-duration-per-query
[max_fetched_data_duration_per_query: <duration> | default = 0s]

# [EXPERIMENTAL] Dynamically adjust vertical shard size to maximize the total
# combined number of query shards and splits.
# CLI flag: -querier.enable-dynamic-vertical-sharding
[enable_dynamic_vertical_sharding: <boolean> | default = false]

# Mutate incoming queries to align their start and end with their step.
# CLI flag: -querier.align-querier-with-step
[align_queries_with_step: <boolean> | default = false]
Expand Down
2 changes: 2 additions & 0 deletions pkg/querier/tripperware/queryrange/query_range_middlewares.go
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,14 @@ func (cfg *Config) Validate(qCfg querier.Config) error {
type DynamicQuerySplitsConfig struct {
MaxShardsPerQuery int `yaml:"max_shards_per_query"`
MaxFetchedDataDurationPerQuery time.Duration `yaml:"max_fetched_data_duration_per_query"`
EnableDynamicVerticalSharding bool `yaml:"enable_dynamic_vertical_sharding"`
}

// RegisterFlags registers flags foy dynamic query splits
func (cfg *DynamicQuerySplitsConfig) RegisterFlags(f *flag.FlagSet) {
f.IntVar(&cfg.MaxShardsPerQuery, "querier.max-shards-per-query", 0, "[EXPERIMENTAL] Maximum number of shards for a query, 0 disables it. Dynamically uses a multiple of split interval to maintain a total number of shards below the set value. If vertical sharding is enabled for a query, the combined total number of interval splits and vertical shards is kept below this value.")
f.DurationVar(&cfg.MaxFetchedDataDurationPerQuery, "querier.max-fetched-data-duration-per-query", 0, "[EXPERIMENTAL] Max total duration of data fetched from storage by all query shards, 0 disables it. Dynamically uses a multiple of split interval to maintain a total fetched duration of data lower than the value set. It takes into account additional duration fetched by matrix selectors and subqueries.")
f.BoolVar(&cfg.EnableDynamicVerticalSharding, "querier.enable-dynamic-vertical-sharding", false, "[EXPERIMENTAL] Dynamically adjust vertical shard size to maximize the total combined number of query shards and splits.")
}

// Middlewares returns list of middlewares that should be applied for range query.
Expand Down
Loading
Loading