Trace Size Exceeded

[IrinaAlexandra]

Hi,

I am starting to see this error a lot in most of our django applications.

 [Error] Unprocessed segment: {	
	ErrorCode: "InvalidSegment",	
	Id: "xxxxxxxxxxxxxxxxxx",	
	Message: "Trace Size Exceeded. Reduce the number of segments per trace and the size of your segment documents."	
}

Also, whilst looking at the logs:

2021-07-21T11:37:15.068+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.016 seconds)	
2021-07-21T11:37:15.071+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)	
2021-07-21T11:37:15.087+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.102+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.117+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.135+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.007 seconds)
2021-07-21T11:37:15.148+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.163+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.178+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.194+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.214+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.225+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.239+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.256+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.288+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.017 seconds)
2021-07-21T11:37:15.289+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.311+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.326+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.336+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.350+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.365+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.379+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.395+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.415+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.424+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.439+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.458+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.009 seconds)
2021-07-21T11:37:15.467+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.483+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.497+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.513+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.534+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.554+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.011 seconds)
2021-07-21T11:37:15.563+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.577+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.593+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.610+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.007 seconds)
2021-07-21T11:37:15.624+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.641+03:00	2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.006 seconds)

It looks like a lot of segments are being sent while nothing is really happening on the django application itself.

We are running the daemon on the alongside the django application, in the same docker container. We have the recorder setup as instructed here:

XRAY_RECORDER = {
    "AUTO_INSTRUMENT": True,
    "AWS_XRAY_CONTEXT_MISSING": "LOG_ERROR",
    "AWS_XRAY_TRACING_NAME": f"{env('PARTNER', default='embed')}-{env('ENVIRONMENT_TYPE')}",
    "PLUGINS": ("EC2Plugin", "ECSPlugin"),
    "AWS_XRAY_SDK_ENABLED": env("AWS_XRAY_SDK_ENABLED", default=False),
}

I am also seeing this when the application starts:

[INFO] No effective centralized sampling rule match. Fallback to local rules.

Our sampling rules are not local tho, but defined in the xray we console.

[bhautikpip]

The error message is coming from X-Ray daemon logs since your django app is sending a lot of segments. Looks like segments are being sent to the service means somewhere in your code you should be invoking web app paths. I think set up is working as expected. Probably best to figure out the rate at which you're sending requests to your django app. [INFO] No effective centralized sampling rule match. Fallback to local rules. meaning requests are not able to match sampling rules you have set so it is using default rules which is sampling 1 req/sec and then 5% of additional requests. Reference to set sampling rules - https://docs.aws.amazon.com/xray/latest/devguide/xray-console-sampling.html.

[IrinaAlexandra]

The error message is coming from X-Ray daemon logs since your django app is sending a lot of segments. Looks like segments are being sent to the service means somewhere in your code you should be invoking web app paths. I think set up is working as expected. Probably best to figure out the rate at which you're sending requests to your django app. [INFO] No effective centralized sampling rule match. Fallback to local rules. meaning requests are not able to match sampling rules you have set so it is using default rules which is sampling 1 req/sec and then 5% of additional requests. Reference to set sampling rules - https://docs.aws.amazon.com/xray/latest/devguide/xray-console-sampling.html.

Thank you so much for the reply. The sampling makes sense. Can you be more specific about invoking we app paths? I'm not sure what you mean.

[bhautikpip]

Yeah, I think your sample application code invokes your django app path (ex. /some-path) which results X-Ray Python SDK to create and send segments to X-Ray service. I recommend you to debug at what rate you're calling your django app. Feel free to follow up for further clarification.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in next 7 days. Thank you for your contributions.