Introduce a regex tenant resolver #6713

SungJin1212 · 2025-04-22T01:29:30Z

This PR introduces a regex tenant resolver to allow regex in the X-Scope-OrgID value when the user uses tenant-federation feature.
It introduces two flags, tenant-federation.regex-matcher-enabled and tenant-federation.user-sync-interval.

The tenant-federation.regex-matcher-enabled enables the regex resolver, which allows regex to the X-Scope-OrgID value.
The tenant-federation.user-sync-interval specifies how frequently to scan users. The scanned users are used to calculate matched tenantIDs.

The regex matching rule follows the Prometheus regex matcher (=~), See here.

For example, if there are 3 tenants, whose IDs are user-1, user-2, and user-3. We can set X-Scope-OrgID to user-.+ to query whole tenants.
Also, we can use an existing way like setting user-1|user-2|user-3 to X-Scope-OrgID.

It reuses userScanner to find considered tenant IDs. So, only tenants who uploaded blocks are subject to regex resolution.

Which issue(s) this PR fixes:
Fixes #6588

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

SungJin1212 · 2025-04-24T01:19:07Z

@CharlieTLe
Could you take a look when you have time?

yeya24 · 2025-04-24T16:46:29Z

pkg/cortex/modules.go

+			// because if the # of matched tenantIDs is only one, `X-Scope-OrgID` header is
+			// set to input regex.
+			byPassForSingleQuerier = false
+			tenant.WithDefaultResolver(tenantfederation.NewRegexResolver(prometheus.DefaultRegisterer, t.Cfg.TenantFederation.UserSyncInterval, util_log.Logger, t.Distributor.AllUserStats))


I wonder if there is a better way to gather all users. Calling t.Distributor.AllUserStats seems a bit expensive just to get user IDs.

And it cannot cover users that don't ingest anymore but maybe their data still present on long term storage.

And it cannot cover users that don't ingest anymore but maybe their data still present on long term storage.

Thanks for catching.
How about utilizing the userScanner?

Calling t.Distributor.AllUserStats seems a bit expensive just to get user IDs.

Do you have any good ideas?

CharlieTLe

This looks awesome!

SungJin1212 · 2025-05-15T09:19:16Z

@CharlieTLe
I changed not to attach __tenant_id__ label when the number of matched tenants is one (same behavior when the user uses the multi-resolver).

yeya24 · 2025-06-13T07:05:40Z

CHANGELOG.md

@@ -11,6 +11,9 @@
 * [FEATURE] Ingester: Support out-of-order native histogram ingestion. It automatically enabled when `-ingester.out-of-order-time-window > 0` and `-blocks-storage.tsdb.enable-native-histograms=true`. #6626 #6663
 * [FEATURE] Ruler: Add support for percentage based sharding for rulers. #6680
 * [FEATURE] Ruler: Add support for group labels. #6665
+* [FEATURE] Query federation: Introduce a regex tenant resolver to allow regex in `X-Scope-OrgID` value. #6713
+- Add a `tenant-federation.regex-matcher-enabled` flag. If it enabled, user can input regex to `X-Scope-OrgId`, the matched tenantIDs are automatically involved.
+- Add a `tenant-federation.user-sync-interval` flag, it specifies how frequently to scan users. The scanned users are used to calculate matched tenantIDs.


We should document it as experimental feature in the doc

Maybe we should document that the user discovery is based on scanning block storage so new users are only available every 2h (assuming blocks are only uploaded every 2h).

yeya24 · 2025-06-13T07:10:15Z

pkg/querier/tenantfederation/regex_resolver.go

+		return nil, errors.Wrap(err, "failed to create the bucket client")
+	}
+
+	userScanner, err := users.NewScanner(cfg, bucketClient, logger, reg)


We probably need to wrap the registry with the component name? I think you will get duplicate metrics registration issue as we initialize multiple user scanner in different components. This can fail if running in single binary mode

It's safe, but it's better to add the component name.

yeya24 · 2025-06-13T07:10:55Z

pkg/querier/tenantfederation/regex_resolver.go

+		Help: "Number of discovered users.",
+	})
+
+	go r.updateUsersLoop(userSyncInterval, userScanTimeout)


Should we maintain this using a service rather than a simple goroutine?

yeya24 · 2025-06-13T07:12:27Z

pkg/querier/tenantfederation/regex_resolver.go

+		defer cancel()
+
+		// only active users are considered
+		activeUsers, _, _, err := r.userScanner.ScanUsers(ctx)


I think we should include deleting users as well. Store Gateway still load blocks from deleting users and it should be expected to query those users within the clean up delay.

yeya24 · 2025-06-13T07:13:35Z

pkg/querier/tenantfederation/regex_resolver.go

+	userScanTimeout = time.Second * 30
+)
+
+type RegexResolver struct {


Can you please add some comments of what RegexResolver and RegexValidator does? I am not too familiar with this code path so unsure their difference

Signed-off-by: SungJin1212 <[email protected]>

pull-request-size bot added the size/XL label Apr 22, 2025

SungJin1212 marked this pull request as draft April 22, 2025 01:29

dosubot bot added the component/querier label Apr 22, 2025

SungJin1212 force-pushed the Support-regex-to-tenant-federation branch 8 times, most recently from 1b8963e to e6a4222 Compare April 22, 2025 06:02

SungJin1212 marked this pull request as ready for review April 22, 2025 06:27

SungJin1212 force-pushed the Support-regex-to-tenant-federation branch from e6a4222 to 63f97d2 Compare April 24, 2025 06:46

yeya24 reviewed Apr 24, 2025

View reviewed changes

CharlieTLe approved these changes May 3, 2025

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label May 3, 2025

SungJin1212 force-pushed the Support-regex-to-tenant-federation branch from 63f97d2 to 605f91a Compare May 7, 2025 02:31

SungJin1212 force-pushed the Support-regex-to-tenant-federation branch from 605f91a to d1b80e0 Compare May 15, 2025 09:00

SungJin1212 force-pushed the Support-regex-to-tenant-federation branch 2 times, most recently from b7ef458 to 7321e4b Compare June 12, 2025 12:02

pull-request-size bot added size/XXL and removed size/XL labels Jun 12, 2025

SungJin1212 force-pushed the Support-regex-to-tenant-federation branch 6 times, most recently from eaeb19e to 1135b06 Compare June 13, 2025 02:27

yeya24 reviewed Jun 13, 2025

View reviewed changes

SungJin1212 force-pushed the Support-regex-to-tenant-federation branch from 1135b06 to 1a3d2f1 Compare June 13, 2025 11:50

Introduce a regex tenant resolver

8737cf6

Signed-off-by: SungJin1212 <[email protected]>

SungJin1212 force-pushed the Support-regex-to-tenant-federation branch from 1a3d2f1 to 8737cf6 Compare June 13, 2025 12:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Introduce a regex tenant resolver #6713

Introduce a regex tenant resolver #6713

Uh oh!

SungJin1212 commented Apr 22, 2025 •

edited

Loading

Uh oh!

SungJin1212 commented Apr 24, 2025

Uh oh!

yeya24 Apr 24, 2025

Uh oh!

SungJin1212 Apr 25, 2025 •

edited

Loading

Uh oh!

SungJin1212 Apr 28, 2025 •

edited

Loading

Uh oh!

CharlieTLe left a comment

Uh oh!

SungJin1212 commented May 15, 2025

Uh oh!

yeya24 Jun 13, 2025

Uh oh!

yeya24 Jun 13, 2025

Uh oh!

yeya24 Jun 13, 2025

Uh oh!

SungJin1212 Jun 13, 2025

Uh oh!

yeya24 Jun 13, 2025

Uh oh!

yeya24 Jun 13, 2025

Uh oh!

yeya24 Jun 13, 2025

Uh oh!

Uh oh!

Introduce a regex tenant resolver #6713

Are you sure you want to change the base?

Introduce a regex tenant resolver #6713

Uh oh!

Conversation

SungJin1212 commented Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SungJin1212 commented Apr 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SungJin1212 Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SungJin1212 Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CharlieTLe left a comment

Choose a reason for hiding this comment

Uh oh!

SungJin1212 commented May 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SungJin1212 commented Apr 22, 2025 •

edited

Loading

SungJin1212 Apr 25, 2025 •

edited

Loading

SungJin1212 Apr 28, 2025 •

edited

Loading