WithRetries Not Waiting Long Enough #501

GusPrice · 2025-02-06T19:42:04Z

Checklist

I have looked into the README and have not found a suitable solution or answer.
I have looked into the documentation and have not found a suitable solution or answer.
I have searched the issues and have not found a suitable solution or answer.
I have upgraded to the latest version of this SDK and the issue still persists.
I have searched the Auth0 Community forums and have not found a suitable solution or answer.
I agree to the terms within the Auth0 Code of Conduct.

Description

I am running into an issue where when I receive a 429 response from the management api, it seems the retries are not properly waiting to try again.

It seems that with this retry option, or just using the default retries I frequently run into an issue where the client gets a 429, but retries too soon. It seems like it's using the header value and getting to the final return in backoffDelay() in internal/client/client.go:

return time.Duration(resetAt-time.Now().Unix()) * time.Second

But it seems like maybe this math isn't quite waiting long enough for things to have reset. Here's an example of the debug logs where you can see the reset header value is the current time the code is running at but it continues to fail.

Sample Output

2025/02/06 11:00:31 
HTTP/2.0 429 Too Many Requests
Content-Length: 120
Access-Control-Expose-Headers: WWW-Authenticate,Server-Authorization
Age: 1
Alt-Svc: h3=":443"; ma=86400
Cache-Control: public, max-age=1
Cf-Cache-Status: HIT
Cf-Ray: 90dd62320973645e-SJC
Content-Type: application/json; charset=utf-8
Date: Thu, 06 Feb 2025 19:00:31 GMT
Retry-After: 1
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: origin, Accept-Encoding
X-Auth0-L: 0.025
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 10
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 1738868431

{"statusCode":429,"error":"Too Many Requests","message":"Global limit has been reached","errorCode":"too_many_requests"}

2025/02/06 11:00:31 
GET /api/v2/users-by-email?email=test%gmail.com HTTP/1.1
Host: melkor-dev.us.auth0.com
User-Agent: Go-Auth0/1.15.0
Auth0-Client: eyJuYW1lIjoiZ28tYXV0aDAiLCJ2ZXJzaW9uIjoiMS4xNS4wIiwiZW52Ijp7ImdvIjoiZ28xLjIyLjQifX0=
Content-Type: application/json
Accept-Encoding: gzip


2025/02/06 11:00:31 
HTTP/2.0 429 Too Many Requests
Content-Length: 120
Access-Control-Expose-Headers: WWW-Authenticate,Server-Authorization
Age: 1
Alt-Svc: h3=":443"; ma=86400
Cache-Control: public, max-age=1
Cf-Cache-Status: HIT
Cf-Ray: 90dd623269d7645e-SJC
Content-Type: application/json; charset=utf-8
Date: Thu, 06 Feb 2025 19:00:31 GMT
Retry-After: 1
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: origin, Accept-Encoding
X-Auth0-L: 0.025
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 10
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 1738868431

{"statusCode":429,"error":"Too Many Requests","message":"Global limit has been reached","errorCode":"too_many_requests"}

2025/02/06 11:00:31 
GET /api/v2/users-by-email?email=test%gmail.com HTTP/1.1
Host: melkor-dev.us.auth0.com
User-Agent: Go-Auth0/1.15.0
Auth0-Client: eyJuYW1lIjoiZ28tYXV0aDAiLCJ2ZXJzaW9uIjoiMS4xNS4wIiwiZW52Ijp7ImdvIjoiZ28xLjIyLjQifX0=
Content-Type: application/json
Accept-Encoding: gzip


2025/02/06 11:00:31 
HTTP/2.0 429 Too Many Requests
Content-Length: 120
Access-Control-Expose-Headers: WWW-Authenticate,Server-Authorization
Age: 1
Alt-Svc: h3=":443"; ma=86400
Cache-Control: public, max-age=1
Cf-Cache-Status: HIT
Cf-Ray: 90dd6232ba12645e-SJC
Content-Type: application/json; charset=utf-8
Date: Thu, 06 Feb 2025 19:00:31 GMT
Retry-After: 1
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: origin, Accept-Encoding
X-Auth0-L: 0.025
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 10
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 1738868431

{"statusCode":429,"error":"Too Many Requests","message":"Global limit has been reached","errorCode":"too_many_requests"}

2025/02/06 11:00:31 
GET /api/v2/users-by-email?email=test%gmail.com HTTP/1.1
Host: melkor-dev.us.auth0.com
User-Agent: Go-Auth0/1.15.0
Auth0-Client: eyJuYW1lIjoiZ28tYXV0aDAiLCJ2ZXJzaW9uIjoiMS4xNS4wIiwiZW52Ijp7ImdvIjoiZ28xLjIyLjQifX0=
Content-Type: application/json
Accept-Encoding: gzip


2025/02/06 11:00:31 
HTTP/2.0 429 Too Many Requests
Content-Length: 120
Access-Control-Expose-Headers: WWW-Authenticate,Server-Authorization
Age: 1
Alt-Svc: h3=":443"; ma=86400
Cache-Control: public, max-age=1
Cf-Cache-Status: HIT
Cf-Ray: 90dd62330a69645e-SJC
Content-Type: application/json; charset=utf-8
Date: Thu, 06 Feb 2025 19:00:31 GMT
Retry-After: 1
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: origin, Accept-Encoding
X-Auth0-L: 0.025
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 10
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 1738868431

{"statusCode":429,"error":"Too Many Requests","message":"Global limit has been reached","errorCode":"too_many_requests"}

2025/02/06 11:00:31 
GET /api/v2/users-by-email?email=test%gmail.com HTTP/1.1
Host: melkor-dev.us.auth0.com
User-Agent: Go-Auth0/1.15.0
Auth0-Client: eyJuYW1lIjoiZ28tYXV0aDAiLCJ2ZXJzaW9uIjoiMS4xNS4wIiwiZW52Ijp7ImdvIjoiZ28xLjIyLjQifX0=
Content-Type: application/json
Accept-Encoding: gzip


2025/02/06 11:00:31 
HTTP/2.0 429 Too Many Requests
Content-Length: 120
Access-Control-Expose-Headers: WWW-Authenticate,Server-Authorization
Age: 1
Alt-Svc: h3=":443"; ma=86400
Cache-Control: public, max-age=1
Cf-Cache-Status: HIT
Cf-Ray: 90dd62336aa3645e-SJC
Content-Type: application/json; charset=utf-8
Date: Thu, 06 Feb 2025 19:00:31 GMT
Retry-After: 1
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: origin, Accept-Encoding
X-Auth0-L: 0.025
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 10
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 1738868431

{"statusCode":429,"error":"Too Many Requests","message":"Global limit has been reached","errorCode":"too_many_requests"}

Expectation

I would expect that 1 retry, 2 maximum would wait long enough to avoid this 429 burst limit of 10 that is specific to my ip address. Currently it seems to try again too soon relative the reset value returned by the header.

Reproduction

I am running something similar to the follow example code (simplified):

	auth0Management, err := a0mgmt.New(
		cfg.Auth0.ManagementDomain,
		a0mgmt.WithClientCredentials(ctx, cfg.Auth0.ClientID, cfg.Auth0.ClientSecret),
		a0mgmt.WithDebug(true),
		a0mgmt.WithRetries(5, []int{http.StatusTooManyRequests}),
	)
	if err != nil {
		log.Fatalf("failed to initialize the auth0 management API client: %+v", err)
	}

	emails := []string{"[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]"}
	for _, email := range emails {
		users, err := a0.User.ListByEmail(ctx, email)
		if err != nil {
			return &a0mgmt.User{}, err
		}
		fmt.Println(users)
	}

Auth0 Go SDK version

1.16.0

The text was updated successfully, but these errors were encountered:

GusPrice · 2025-02-12T05:54:29Z

Per Alvaro with Auth0 support:

I think the problem is with the GO SDK and how it handles these burst rate limit resets. Here is what is happening in the code:
1. The code calculates a wait time using exponential backoff, the delay is supposed to double with each retry attempt.
2. A random time is added to the delay in a range between the min and max values.
3. The function checks if the response status is 429. If no 429 then returns the wait time.
4. When the status is 429 it looks at the x-ratelimit-reset header and attempts to parse this value into resetAt.
5. If parsing fails, then it returns the wait time.
6. If the resetAt timestamp is greater than the current time plus the maximum delay, it returns the wait time.
7. If the rate limit reset is within the max delay, it returns the remaining time until the reset occurs (resetAt - time.now().unix()), converts this to seconds, and then to time.duration. 
 
The problem is happening because there is no difference between the resetAt value and the current time. In the example you provided, the resetAt timestamp is 1738868431 (2025/02/06 11:00:31) and the current time is also 2025/02/06 11:00:31, you would need to add a delay for this to work as expected. This retry logic does work for the 429 responses from the global Management API limit. That refill rate is in minutes so each new request is added in seconds instead of milliseconds.

developerkunal · 2025-02-17T05:16:07Z

Hi @GusPrice,

I hope you're doing well!

I tried to investigate this issue, but I wasn’t able to find any problems. Could you let me know if you're using a free tenant or an enterprise one?

Thanks!

GusPrice · 2025-02-19T17:23:47Z

Hi @developerkunal,

That's unfortunate. We are on an enterprise account; but the tenant is a development level.

GusPrice · 2025-02-26T17:12:55Z

@developerkunal just following up here to see if you were able to look any further?

developerkunal · 2025-02-27T16:35:16Z

Hi @GusPrice,

We are currently looking into this! This issue may not be limited to the current SDK but could also affect other SDKs.

GusPrice · 2025-03-06T19:54:05Z

Hello @developerkunal just circling back here! I understand that other SDKs may not account for this burst limit, but it would be great to have the functionality!

developerkunal · 2025-03-07T15:15:22Z

Hi @GusPrice,

I’m currently investigating this issue. So far, I haven’t found any specific error indicating that the burst limit has been reached—only messages stating that the global limit has been exceeded.

I’m working on updating the retry logic to better handle rate limiting. However, one challenge is that rate limits vary based on the type of tenant. This means we can’t apply a blanket change without potentially causing issues for both normal and enterprise users.

I’ll keep you posted as I make progress. Let me know if you have any additional insights!

GusPrice · 2025-03-07T21:56:01Z

Yeah, makes a lot of sense. I appreciate you taking a stab at it. Maybe worth a conversation with the team that's responsible for the internal API and see if they could return some kind of different error messaging for global vs burst and levels of burst. More clear error messages and codes would make this a lot easier!

developerkunal · 2025-03-08T06:17:32Z

Hi @GusPrice,

I’ve opened a PR to handle the clock skew issue by respecting the new Retry-After header. If possible, could you fetch this specific commit and test it on your end to see if it resolves the issue?

Issue reference: #523

Let me know if you run into any problems!

GusPrice · 2025-03-10T21:07:57Z

@developerkunal awesome! I will read through the PR and give it a test for our use case this week. Will report back. Thank you!

…Issue (#523) Co-authored-by: Ramya Anusri <[email protected]>

GusPrice · 2025-03-11T19:50:26Z

@developerkunal tested today and things are working as expected now with your change. Appreciate your help and willingness to refactor in this case!

GusPrice added the bug Something isn't working label Feb 6, 2025

developerkunal mentioned this issue Mar 7, 2025

[GH-501] Fix Retry Mechanism to Handle Burst Limit Due to Clock Skew Issue #523

Merged

2 tasks

developerkunal added a commit that referenced this issue Mar 11, 2025

[GH-501] Fix Retry Mechanism to Handle Burst Limit Due to Clock Skew …

555ced6

…Issue (#523) Co-authored-by: Ramya Anusri <[email protected]>

developerkunal closed this as completed in #523 Mar 11, 2025

developerkunal mentioned this issue Mar 11, 2025

Release v1.18.0 #524

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WithRetries Not Waiting Long Enough #501

WithRetries Not Waiting Long Enough #501

GusPrice commented Feb 6, 2025

GusPrice commented Feb 12, 2025

developerkunal commented Feb 17, 2025

GusPrice commented Feb 19, 2025

GusPrice commented Feb 26, 2025

developerkunal commented Feb 27, 2025

GusPrice commented Mar 6, 2025

developerkunal commented Mar 7, 2025

GusPrice commented Mar 7, 2025

developerkunal commented Mar 8, 2025

GusPrice commented Mar 10, 2025

GusPrice commented Mar 11, 2025

WithRetries Not Waiting Long Enough #501

WithRetries Not Waiting Long Enough #501

Comments

GusPrice commented Feb 6, 2025

Checklist

Description

Expectation

Reproduction

Auth0 Go SDK version

GusPrice commented Feb 12, 2025

developerkunal commented Feb 17, 2025

GusPrice commented Feb 19, 2025

GusPrice commented Feb 26, 2025

developerkunal commented Feb 27, 2025

GusPrice commented Mar 6, 2025

developerkunal commented Mar 7, 2025

GusPrice commented Mar 7, 2025

developerkunal commented Mar 8, 2025

GusPrice commented Mar 10, 2025

GusPrice commented Mar 11, 2025