From 896df67945c17c0327cb0ff563729fd5bd8df791 Mon Sep 17 00:00:00 2001 From: vigneshshanmugam Date: Tue, 5 Sep 2023 15:45:05 -0700 Subject: [PATCH 1/7] proposal for correlating synthetics traces --- specs/integrations/synthetics.md | 42 ++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 specs/integrations/synthetics.md diff --git a/specs/integrations/synthetics.md b/specs/integrations/synthetics.md new file mode 100644 index 00000000..2d245c14 --- /dev/null +++ b/specs/integrations/synthetics.md @@ -0,0 +1,42 @@ +## Synthetics Integration + +Synthetic monitors play a crucial role in periodically checking the status of your services and applications on a global scale. General documentation about synthetic monitors can be found in +[Synthetics getting started page](https://www.elastic.co/guide/en/observability/current/synthetics-get-started.html). + +This integration goes in to more detail about how the sythetics monitors would +be correlated with the APM traces. Synthetics traces can be categorized in to two +main types + 1. HTTP checks - These have one-one mapping with APM transactions + 2. Browser checks - These have a one-to-many mapping with APM transactions + +### Correlation + +The Synthetics agent takes the responsibility of creating the [`traceparent`](../agents/tracing-distributed-tracing.md#trace_id-parent_id-and-traceparent) header for each outgoing network request associated with a test during every monitor execution. + +- `trace.id` and `parent.id` + - outgoing requests that are being explicity traced by the synthetics agent + will have the `parent.id` and `trace.id` as part of the trace context. + - must be unique for each step for a browser monitor + - must be unique for a http monitor +- `sampled` Flag + - used to control the sampling decision for all the downstream services. + - 100% sampling when tracing is enabled + +These correlation values would be applicable even if the downstream services are traced by +OpenTelemetry(OTEL)-based agents. + +### Identifying Synthetics trace + +Synthetics monitor executions creates `rootless traces` as these traces are not +reported to the APM server. To overcome this limitation on the APM UI, we need +to identify the synthetics traces and explicity link them to the Synthetics +waterfall view. + +- `http.headers.user-agent`: + - Contains `Elastic/Synthetics` for all outgoing requests from Synthetis based monitors. + +When a trace is confirmed to be originated from Synthetics-based monitors, the +Trace Explorer view can be linked back to the Synthetics waterfall view. + +- `/app/synthetics/link-to/` + - used to link back the explicit browser waterfall step on the Synthetics UI. \ No newline at end of file From dadb42b77b23b117dfb575a36723f50d711c6d5e Mon Sep 17 00:00:00 2001 From: vigneshshanmugam Date: Wed, 6 Sep 2023 12:12:01 -0700 Subject: [PATCH 2/7] address feedback and add more details --- specs/integrations/synthetics.md | 87 ++++++++++++++++++++++++++++++-- 1 file changed, 82 insertions(+), 5 deletions(-) diff --git a/specs/integrations/synthetics.md b/specs/integrations/synthetics.md index 2d245c14..4e3b342c 100644 --- a/specs/integrations/synthetics.md +++ b/specs/integrations/synthetics.md @@ -11,7 +11,10 @@ main types ### Correlation -The Synthetics agent takes the responsibility of creating the [`traceparent`](../agents/tracing-distributed-tracing.md#trace_id-parent_id-and-traceparent) header for each outgoing network request associated with a test during every monitor execution. +The Synthetics agent (including Heartbeat) takes the responsibility of creating the +[`traceparent`](../agents/tracing-distributed-tracing.md#trace_id-parent_id-and-traceparent) +header for each outgoing network request associated with a test during every +monitor execution. - `trace.id` and `parent.id` - outgoing requests that are being explicity traced by the synthetics agent @@ -22,21 +25,95 @@ The Synthetics agent takes the responsibility of creating the [`traceparent`](.. - used to control the sampling decision for all the downstream services. - 100% sampling when tracing is enabled -These correlation values would be applicable even if the downstream services are traced by -OpenTelemetry(OTEL)-based agents. +#### Browser checks + +When executing a Synthetics journey with APM tracing enabled for specific URLs +using the --apm_tracing_urls flag, the Synthetics agent takes the following +actions: + +1. Adds the traceparent header to each matching outgoing request. +2. Includes trace.id and parent.id in all the Step Elasticsearch (ES) documents for the journey. + +```ts +// run journey +npx @elastic/synthetics --apm_tracing_urls "elastic.co/*" + +// example.journey.ts +journey("elastic e2e", ({ page }) => { + step("home page", async() => { + await page.goto("https://www.elastic.co") + }) + step("blog page", async() => { + await page.goto("https://www.elastic.co/blog") + }) +}) +``` + +Example of the tracing information added to the ES documents for two steps in the journey: + +```json +// Step - homepage +{"type":"step/end","journey":{"name":"elastic e2e"},"step":{"name":"home page","index":1,"status":"failed","duration":{"us":17382122}}, "trace.id": "xxx"} +{"type":"journey/network_info","journey":{"name":"elastic e2e",},"step":{"name":"home page","index":1},"http":{"request":{"url":"http://www.elastic.co/","method":"GET"}},"trace.id": "t1", "transaction.id": "tr1"} + + +// Step - blog page +{"type":"step/end","journey":{"name":"elastic e2e"},"step":{"name":"blog page","index":2,"status":"failed","duration":{"us":17382122}}, "trace.id": "xxx"} +{"type":"journey/network_info","journey":{"name":"elastic e2e",},"step":{"name":"blog page","index":2},"http":{"request":{"url":"http://www.elastic.co/blog","method":"GET"}},"trace.id": "t1", "transaction.id": "tr2"} +``` + +With this tracing information available in the ES documents for each step's network requests, the Synthetics UI can link back to the individual backend transactions on the APM. + +#### HTTP Checks + +For the below HTTP monitor + +```yml +# heartbeat.yml +heartbeat.monitors: +- type: http + id: test-http + urls: ["https://www.example.com"] + apm: + enabled: true +``` + +Heartbeat would add the `traceparent` header to the monitored URL and add the +other tracing related information to the ES documents. + +```json +{"event":{"action":"monitor.run"},"monitor":{"id":"test-http","type":"http","status":"up","duration":{"ms":112}}, "trace.id": "t1", "transaction.id": "tr1"} +``` + +It's important to note that there is no dedicated waterfall information for the HTTP checks in the Synthetics UI. Consequently, the linking here will directly take you to the APM transaction if the backend is also traced by Elastic APM or OTEL (OpenTelemetry)-based agents. + +**NOTE: The correlation remain applicable even if downstream services are traced by OpenTelemetry (OTEL)-based agents. This ensures a consistent and seamless tracing experience regardless of the underlying tracing infrastructure..** ### Identifying Synthetics trace Synthetics monitor executions creates `rootless traces` as these traces are not reported to the APM server. To overcome this limitation on the APM UI, we need -to identify the synthetics traces and explicity link them to the Synthetics +to identify the synthetics traces and explicity link them to the Synthetics waterfall view. - `http.headers.user-agent`: - Contains `Elastic/Synthetics` for all outgoing requests from Synthetis based monitors. +There is a limitation with this approach +- users can override the `User-Agent` header in the monitor configuration which + might lead to users seeing only partial traces on the APM UI. + +We can also add a foolproof solution by introducing vendor specific `tracestate` +property. + +- `tracestate`: + - Contains `es:origin=synthetics` for all outgoing requests from Synthetis based monitors. + + When a trace is confirmed to be originated from Synthetics-based monitors, the Trace Explorer view can be linked back to the Synthetics waterfall view. - `/app/synthetics/link-to/` - - used to link back the explicit browser waterfall step on the Synthetics UI. \ No newline at end of file + - links back to the explicit browser waterfall step on the Synthetics UI, and + it follows the format /monitor/:monitorId/test-run/:runId/step/:stepIndex. + - `runId` is internal to the Synthetics side which is also available on ES step documents. \ No newline at end of file From 9098abe78ba182ffefbab467f7b8dc4c5404477a Mon Sep 17 00:00:00 2001 From: vigneshshanmugam Date: Wed, 6 Sep 2023 12:14:08 -0700 Subject: [PATCH 3/7] formatting fix --- specs/integrations/synthetics.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/integrations/synthetics.md b/specs/integrations/synthetics.md index 4e3b342c..22c48f4b 100644 --- a/specs/integrations/synthetics.md +++ b/specs/integrations/synthetics.md @@ -54,12 +54,12 @@ Example of the tracing information added to the ES documents for two steps in th ```json // Step - homepage {"type":"step/end","journey":{"name":"elastic e2e"},"step":{"name":"home page","index":1,"status":"failed","duration":{"us":17382122}}, "trace.id": "xxx"} -{"type":"journey/network_info","journey":{"name":"elastic e2e",},"step":{"name":"home page","index":1},"http":{"request":{"url":"http://www.elastic.co/","method":"GET"}},"trace.id": "t1", "transaction.id": "tr1"} +{"type":"journey/network_info","journey":{"name":"elastic e2e"},"step":{"name":"home page","index":1},"http":{"request":{"url":"http://www.elastic.co/","method":"GET"}},"trace.id": "t1", "transaction.id": "tr1"} // Step - blog page {"type":"step/end","journey":{"name":"elastic e2e"},"step":{"name":"blog page","index":2,"status":"failed","duration":{"us":17382122}}, "trace.id": "xxx"} -{"type":"journey/network_info","journey":{"name":"elastic e2e",},"step":{"name":"blog page","index":2},"http":{"request":{"url":"http://www.elastic.co/blog","method":"GET"}},"trace.id": "t1", "transaction.id": "tr2"} +{"type":"journey/network_info","journey":{"name":"elastic e2e"},"step":{"name":"blog page","index":2},"http":{"request":{"url":"http://www.elastic.co/blog","method":"GET"}},"trace.id": "t1", "transaction.id": "tr2"} ``` With this tracing information available in the ES documents for each step's network requests, the Synthetics UI can link back to the individual backend transactions on the APM. @@ -115,5 +115,5 @@ Trace Explorer view can be linked back to the Synthetics waterfall view. - `/app/synthetics/link-to/` - links back to the explicit browser waterfall step on the Synthetics UI, and - it follows the format /monitor/:monitorId/test-run/:runId/step/:stepIndex. + it follows the format `/monitor/:monitorId/test-run/:runId/step/:stepIndex`. - `runId` is internal to the Synthetics side which is also available on ES step documents. \ No newline at end of file From f384db944eb43ff4665f3912b4d895818b7f87d2 Mon Sep 17 00:00:00 2001 From: Vignesh Shanmugam Date: Thu, 7 Sep 2023 06:37:33 -0700 Subject: [PATCH 4/7] apply suggestions from code review Co-authored-by: Felix Barnsteiner --- specs/integrations/synthetics.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/specs/integrations/synthetics.md b/specs/integrations/synthetics.md index 22c48f4b..1124fac0 100644 --- a/specs/integrations/synthetics.md +++ b/specs/integrations/synthetics.md @@ -53,13 +53,13 @@ Example of the tracing information added to the ES documents for two steps in th ```json // Step - homepage -{"type":"step/end","journey":{"name":"elastic e2e"},"step":{"name":"home page","index":1,"status":"failed","duration":{"us":17382122}}, "trace.id": "xxx"} -{"type":"journey/network_info","journey":{"name":"elastic e2e"},"step":{"name":"home page","index":1},"http":{"request":{"url":"http://www.elastic.co/","method":"GET"}},"trace.id": "t1", "transaction.id": "tr1"} +{"type":"step/end","journey":{"name":"elastic e2e"},"step":{"name":"home page","index":1,"status":"failed","duration":{"us":17382122}}, "trace.id": "t1"} +{"type":"journey/network_info","journey":{"name":"elastic e2e"},"step":{"name":"home page","index":1},"http":{"request":{"url":"http://www.elastic.co/","method":"GET"}},"trace.id": "t1", "span.id": "s1"} // Step - blog page -{"type":"step/end","journey":{"name":"elastic e2e"},"step":{"name":"blog page","index":2,"status":"failed","duration":{"us":17382122}}, "trace.id": "xxx"} -{"type":"journey/network_info","journey":{"name":"elastic e2e"},"step":{"name":"blog page","index":2},"http":{"request":{"url":"http://www.elastic.co/blog","method":"GET"}},"trace.id": "t1", "transaction.id": "tr2"} +{"type":"step/end","journey":{"name":"elastic e2e"},"step":{"name":"blog page","index":2,"status":"failed","duration":{"us":17382122}}, "trace.id": "t2"} +{"type":"journey/network_info","journey":{"name":"elastic e2e"},"step":{"name":"blog page","index":2},"http":{"request":{"url":"http://www.elastic.co/blog","method":"GET"}},"trace.id": "t2", "span.id": "s2"} ``` With this tracing information available in the ES documents for each step's network requests, the Synthetics UI can link back to the individual backend transactions on the APM. @@ -82,12 +82,12 @@ Heartbeat would add the `traceparent` header to the monitored URL and add the other tracing related information to the ES documents. ```json -{"event":{"action":"monitor.run"},"monitor":{"id":"test-http","type":"http","status":"up","duration":{"ms":112}}, "trace.id": "t1", "transaction.id": "tr1"} +{"event":{"action":"monitor.run"},"monitor":{"id":"test-http","type":"http","status":"up","duration":{"ms":112}}, "trace.id": "t1", "span.id": "s1"} ``` It's important to note that there is no dedicated waterfall information for the HTTP checks in the Synthetics UI. Consequently, the linking here will directly take you to the APM transaction if the backend is also traced by Elastic APM or OTEL (OpenTelemetry)-based agents. -**NOTE: The correlation remain applicable even if downstream services are traced by OpenTelemetry (OTEL)-based agents. This ensures a consistent and seamless tracing experience regardless of the underlying tracing infrastructure..** +**NOTE: The correlation remain applicable even if downstream services are traced by OpenTelemetry (OTEL)-based agents. This ensures a consistent and seamless tracing experience regardless of the underlying tracing infrastructure.** ### Identifying Synthetics trace @@ -105,6 +105,7 @@ There is a limitation with this approach We can also add a foolproof solution by introducing vendor specific `tracestate` property. +However, this property would need special handling in our agents and wouldn't be recognized by vanilla OpenTelemetry agents. - `tracestate`: - Contains `es:origin=synthetics` for all outgoing requests from Synthetis based monitors. From 7d818cdedef4c97e7286e130f3bc4ccd0e087ef3 Mon Sep 17 00:00:00 2001 From: vigneshshanmugam Date: Thu, 7 Sep 2023 08:34:28 -0700 Subject: [PATCH 5/7] add span id to the trace link --- specs/integrations/synthetics.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/specs/integrations/synthetics.md b/specs/integrations/synthetics.md index 1124fac0..5408341d 100644 --- a/specs/integrations/synthetics.md +++ b/specs/integrations/synthetics.md @@ -114,7 +114,6 @@ However, this property would need special handling in our agents and wouldn't be When a trace is confirmed to be originated from Synthetics-based monitors, the Trace Explorer view can be linked back to the Synthetics waterfall view. -- `/app/synthetics/link-to/` +- `/app/synthetics/link-to/:span.id` - links back to the explicit browser waterfall step on the Synthetics UI, and - it follows the format `/monitor/:monitorId/test-run/:runId/step/:stepIndex`. - - `runId` is internal to the Synthetics side which is also available on ES step documents. \ No newline at end of file + it follows the format `/monitor/:monitorId/test-run/:runId/step/:stepIndex#:spanId`. \ No newline at end of file From 2c0c980321c1ad4d3500dc1e659e1ef9d36f111b Mon Sep 17 00:00:00 2001 From: Vignesh Shanmugam Date: Wed, 27 Sep 2023 09:53:56 -0700 Subject: [PATCH 6/7] apply suggestions from code review Co-authored-by: Felix Barnsteiner --- specs/integrations/synthetics.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/integrations/synthetics.md b/specs/integrations/synthetics.md index 5408341d..ff7b50b4 100644 --- a/specs/integrations/synthetics.md +++ b/specs/integrations/synthetics.md @@ -62,7 +62,7 @@ Example of the tracing information added to the ES documents for two steps in th {"type":"journey/network_info","journey":{"name":"elastic e2e"},"step":{"name":"blog page","index":2},"http":{"request":{"url":"http://www.elastic.co/blog","method":"GET"}},"trace.id": "t2", "span.id": "s2"} ``` -With this tracing information available in the ES documents for each step's network requests, the Synthetics UI can link back to the individual backend transactions on the APM. +With this tracing information available in the ES documents for each step's network requests, the Synthetics UI can link back to the individual backend transactions in APM. #### HTTP Checks @@ -87,7 +87,7 @@ other tracing related information to the ES documents. It's important to note that there is no dedicated waterfall information for the HTTP checks in the Synthetics UI. Consequently, the linking here will directly take you to the APM transaction if the backend is also traced by Elastic APM or OTEL (OpenTelemetry)-based agents. -**NOTE: The correlation remain applicable even if downstream services are traced by OpenTelemetry (OTEL)-based agents. This ensures a consistent and seamless tracing experience regardless of the underlying tracing infrastructure.** +**NOTE: The correlation remain applicable even if downstream services are traced by OpenTelemetry (OTel)-based agents. This ensures a consistent and seamless tracing experience regardless of the underlying tracing infrastructure.** ### Identifying Synthetics trace From 980246b8d0ee28967190cf07a85fea8a27853d0a Mon Sep 17 00:00:00 2001 From: vigneshshanmugam Date: Wed, 27 Sep 2023 10:24:39 -0700 Subject: [PATCH 7/7] review on user agent and address comments --- specs/integrations/synthetics.md | 33 +++++++++++++------------------- 1 file changed, 13 insertions(+), 20 deletions(-) diff --git a/specs/integrations/synthetics.md b/specs/integrations/synthetics.md index ff7b50b4..76f67fe4 100644 --- a/specs/integrations/synthetics.md +++ b/specs/integrations/synthetics.md @@ -27,9 +27,7 @@ monitor execution. #### Browser checks -When executing a Synthetics journey with APM tracing enabled for specific URLs -using the --apm_tracing_urls flag, the Synthetics agent takes the following -actions: +When executing a Synthetics journey with tracing enabled for all outgoing requests `**/*` or for specific URLs with the --apm_tracing_urls flag, the Synthetics agent takes the following actions: 1. Adds the traceparent header to each matching outgoing request. 2. Includes trace.id and parent.id in all the Step Elasticsearch (ES) documents for the journey. @@ -85,34 +83,29 @@ other tracing related information to the ES documents. {"event":{"action":"monitor.run"},"monitor":{"id":"test-http","type":"http","status":"up","duration":{"ms":112}}, "trace.id": "t1", "span.id": "s1"} ``` -It's important to note that there is no dedicated waterfall information for the HTTP checks in the Synthetics UI. Consequently, the linking here will directly take you to the APM transaction if the backend is also traced by Elastic APM or OTEL (OpenTelemetry)-based agents. +It's important to note that there is no dedicated waterfall information for the HTTP checks in the Synthetics UI. Consequently, the linking here will directly take you to the transaction if the backend is also traced by Elastic APM or OTel (OpenTelemetry)-based agents. This works similar to the Browser checks where the network request is directly linked to the transaction. **NOTE: The correlation remain applicable even if downstream services are traced by OpenTelemetry (OTel)-based agents. This ensures a consistent and seamless tracing experience regardless of the underlying tracing infrastructure.** ### Identifying Synthetics trace -Synthetics monitor executions creates `rootless traces` as these traces are not -reported to the APM server. To overcome this limitation on the APM UI, we need -to identify the synthetics traces and explicity link them to the Synthetics -waterfall view. +When tracing is enabled on the Synthetics monitors, the agent appends the `Elastic/Synthetics` to the HTTP `User-Agent` header for all outgoing requests. Tracing UI can use this information to identify the traces that are originated from +Synthetics using the following approaches. -- `http.headers.user-agent`: - - Contains `Elastic/Synthetics` for all outgoing requests from Synthetis based monitors. +- Elastic APM agents + - The information is stored in `http.headers.user-agent` +- OTel agents + - The information is stored in `user_agent.original` + +UI will check both of these fields to identify the Synthetics traces and will +prefer `user_agent.original` if both are present. There is a limitation with this approach - users can override the `User-Agent` header in the monitor configuration which - might lead to users seeing only partial traces on the APM UI. - -We can also add a foolproof solution by introducing vendor specific `tracestate` -property. -However, this property would need special handling in our agents and wouldn't be recognized by vanilla OpenTelemetry agents. - -- `tracestate`: - - Contains `es:origin=synthetics` for all outgoing requests from Synthetis based monitors. - + might lead to users seeing only partial traces on APM UI. When a trace is confirmed to be originated from Synthetics-based monitors, the -Trace Explorer view can be linked back to the Synthetics waterfall view. +Trace Explorer view can be linked back to the Synthetics waterfall. - `/app/synthetics/link-to/:span.id` - links back to the explicit browser waterfall step on the Synthetics UI, and