diff --git a/blog/docs/articles/distributed-tracing.md b/blog/docs/articles/distributed-tracing.md index fd09de115c..fd654d5860 100644 --- a/blog/docs/articles/distributed-tracing.md +++ b/blog/docs/articles/distributed-tracing.md @@ -1,5 +1,8 @@ # Distributed tracing with Knative, OpenTelemetry and Jaeger +!!! note + Knative Release 1.19 and higher supports emitting OpenTelemetry traces. So the contents of this post are out of date. + **Author: [Ben Moss](https://twitter.com/mossity), Software Engineer @ [VMware](http://vmware.com)** When trying to understand and diagnose our systems, one of the most basic tools diff --git a/config/nav.yml b/config/nav.yml index 223c461514..919d9e9320 100644 --- a/config/nav.yml +++ b/config/nav.yml @@ -157,7 +157,6 @@ nav: # Serving - admin docs - Administrator Topics: - Kubernetes services: serving/knative-kubernetes-services.md - - Accessing request traces: serving/accessing-traces.md - Enabling requests to Knative services when additional authorization policies are enabled: serving/istio-authorization.md - Exclude namespaces from the Knative webhook: serving/webhook-customizations.md - Configuring high-availability components: serving/config-ha.md @@ -184,6 +183,7 @@ nav: - Security-Guard example alerts: serving/app-security/security-guard-example-alerts.md # Serving - observability - Observability: + - Request traces: serving/observability/accessing-traces.md - Collecting logs: serving/observability/logging/collecting-logs.md - Configuring logging: serving/observability/logging/config-logging.md - Configuring Request logging: serving/observability/logging/request-logging.md @@ -303,7 +303,7 @@ nav: - Configure KEDA Autoscaling of Knative Kafka Resources: eventing/configuration/keda-configuration.md # Eventing - observability - Observability: - - Accessing CloudEvent traces: eventing/accessing-traces.md + - Accessing CloudEvent traces: eventing/observability/accessing-traces.md - Collecting logs: eventing/observability/logging/collecting-logs.md - Configuring logging: eventing/observability/logging/config-logging.md - Collecting metrics: eventing/observability/metrics/collecting-metrics.md diff --git a/docs/eventing/accessing-traces.md b/docs/eventing/observability/accessing-traces.md similarity index 60% rename from docs/eventing/accessing-traces.md rename to docs/eventing/observability/accessing-traces.md index 48d9352d26..a11056f199 100644 --- a/docs/eventing/accessing-traces.md +++ b/docs/eventing/observability/accessing-traces.md @@ -8,22 +8,25 @@ function: how-to # Accessing CloudEvent traces -Depending on the request tracing tool that you have installed on your Knative -Eventing cluster, see the corresponding section for details about how to -visualize and trace your requests. +Traces give us the big picture of how events traverse through the system and applications. +Knative Eventing is instrumented with [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) which can emit traces to a multitude of different backends. -## Before you begin +## Backends -You must have a Knative cluster running with the Eventing component installed. [Learn more](../install/README.md). +### Jaeger V2 -## Configuring tracing +Following [these instructions](https://github.com/jaegertracing/jaeger-operator?tab=readme-ov-file#jaeger-v2-operator) to setup Jaeger V2 on Kubernetes and access your traces. + +## Configuring Eventing Tracing + +You can update the configuration for tracing in using the [`config-observability` ConfigMap](https://github.com/knative/eventing/blob/main/config/core/configmaps/observability.yaml). With the exception of importers, the Knative Eventing tracing is configured through the -`config-tracing` ConfigMap in the `knative-eventing` namespace. +`config-observability` ConfigMap in the `knative-eventing` namespace. Most importers do _not_ use the ConfigMap and instead, use a static 1% sampling rate. -You can use the `config-tracing` ConfigMap to configure the following Eventing components: +You can use the `config-observability` ConfigMap to configure the following Eventing components: - Brokers - Triggers @@ -36,59 +39,61 @@ You can use the `config-tracing` ConfigMap to configure the following Eventing c **Example:** -The following example `config-tracing` ConfigMap samples 10% of all CloudEvents: +The following example `config-observability` ConfigMap samples 10% of all CloudEvents: ```yaml apiVersion: v1 kind: ConfigMap metadata: - name: config-tracing + name: config-observability namespace: knative-eventing data: - backend: "zipkin" - zipkin-endpoint: "http://zipkin.istio-system.svc.cluster.local:9411/api/v2/spans" - sample-rate: "0.1" + tracing-protocol: "grpc" + tracing-endpoint: "http://jaeger-collector.observability:4318/v1/traces" + tracing-sampling: "0.1" ``` ### Configuration options -You can configure your `config-tracing` with following options: +You can configure your `config-observability` with following options: - * `backend`: Valid values are `zipkin` or `none`. The default is `none`. + * `tracing-protocol`: Valid values are `grpc` or `http/protobuf`. The default is `none`. - * `zipkin-endpoint`: Specifies the URL to the zipkin collector where you want to send the traces. - Must be set if backend is set to `zipkin`. + * `tracing-endpoint`: Specifies the URL to the backend where you want to send the traces. + Must be set if backend is set to `grpc` or `http/protobuf`. - * `sample-rate`: Specifies the sampling rate. Valid values are decimals from `0` to `1` + * `tracing-sampling`: Specifies the sampling rate. Valid values are decimals from `0` to `1` (interpreted as a float64), which indicate the probability that any given request is sampled. An example value is `0.5`, which gives each request a 50% sampling probablity. - * `debug`: Enables debugging. Valid values are `true` or `false`. Defaults to `false` when not specified. - Set to `true` to enable debug mode, which forces the `sample-rate` to `1.0` and sends all spans to - the server. - -### Viewing your `config-tracing` ConfigMap +### Viewing your `config-observability` ConfigMap To view your current configuration: ```bash -kubectl -n knative-eventing get configmap config-tracing -oyaml +kubectl -n knative-eventing get configmap config-observability -oyaml ``` -### Editing and deploying your `config-tracing` ConfigMap +### Editing and deploying your `config-observability` ConfigMap To edit and then immediately deploy changes to your ConfigMap, run the following command: ```bash -kubectl -n knative-eventing edit configmap config-tracing +kubectl -n knative-eventing edit configmap config-observability ``` -## Accessing traces in Eventing -To access the traces, you use either the Zipkin or Jaeger tool. Details about using these tools to access traces are provided in the Knative Serving observability section: + \ No newline at end of file diff --git a/docs/serving/accessing-traces.md b/docs/serving/accessing-traces.md deleted file mode 100644 index d3ecf4dfae..0000000000 --- a/docs/serving/accessing-traces.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -audience: administrator -components: - - serving -function: how-to ---- - -# Accessing request traces - -Depending on the request tracing tool that you have installed on your Knative -Serving cluster, see the corresponding section for details about how to -visualize and trace your requests. - -## Configuring Traces - -You can update the configuration file for tracing in [tracing.yaml](https://github.com/knative/serving/blob/main/config/core/configmaps/tracing.yaml). - -Follow the instructions in the file to set your configuration options. This file includes options such as sample rate (to determine what percentage of requests to trace), debug mode, and backend selection (zipkin or none). - -You can quickly explore and update the ConfigMap object with the following command: -```bash -kubectl -n knative-serving edit configmap config-tracing -``` - -## Zipkin - -In order to access request traces, you use the Zipkin visualization tool. - -1. To open the Zipkin UI, enter the following command: - - ```bash - kubectl proxy - ``` - - This command starts a local proxy of Zipkin on port 8001. For security - reasons, the Zipkin UI is exposed only within the cluster. - -1. Access the Zipkin UI at the following URL: - - ``` - http://localhost:8001/api/v1/namespaces//services/zipkin:9411/proxy/zipkin/ - ``` - Where `` is the namespace where Zipkin is deployed, for example, `knative-serving`. -1. Click "Find Traces" to see the latest traces. You can search for a trace ID - or look at traces of a specific application. Click on a trace to see a - detailed view of a specific call. - - - -## Jaeger - -In order to access request traces, you use the Jaeger visualization tool. - -1. To open the Jaeger UI, enter the following command: - - ```bash - kubectl proxy - ``` - - This command starts a local proxy of Jaeger on port 8001. For security - reasons, the Jaeger UI is exposed only within the cluster. - -1. Access the Jaeger UI at the following URL: - - ``` - http://localhost:8001/api/v1/namespaces//services/jaeger-query:16686/proxy/search/ - ``` - Where `` is the namespace where Jaeger is deployed, for example, `knative-serving`. - -1. Select the service of interest and click "Find Traces" to see the latest - traces. Click on a trace to see a detailed view of a specific call. - - diff --git a/docs/serving/observability/accessing-traces.md b/docs/serving/observability/accessing-traces.md new file mode 100644 index 0000000000..3c858035e9 --- /dev/null +++ b/docs/serving/observability/accessing-traces.md @@ -0,0 +1,66 @@ +--- +audience: administrator +components: + - serving +function: how-to +--- + +# Accessing request traces + +Traces give us the big picture of what happens when a request is made to an application. +Knative Serving is instrumented with [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) which can emit traces to a multitude of different backends. + +## Backends + +### Jaeger V2 + +Following [these instructions](https://github.com/jaegertracing/jaeger-operator?tab=readme-ov-file#jaeger-v2-operator) to setup Jaeger V2 on Kubernetes and access your traces. + +## Configuring Serving Tracing + +You can update the configuration for tracing in using the [`config-observability` ConfigMap](https://github.com/knative/serving/blob/main/config/core/configmaps/observability.yaml). + +**Example:** + +The following example `config-observability` ConfigMap samples 10% of all requests: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: config-observability + namespace: knative-serving +data: + tracing-protocol: "grpc" + tracing-endpoint: "http://jaeger-collector.observability:4318/v1/traces" + tracing-sampling: "0.1" +``` + +### Configuration options + +You can configure your `config-observability` with following options: + + * `tracing-protocol`: Valid values are `grpc` or `http/protobuf`. The default is `none`. + + * `tracing-endpoint`: Specifies the URL to the backend where you want to send the traces. + Must be set if backend is set to `grpc` or `http/protobuf`. + + * `tracing-sampling`: Specifies the sampling rate. Valid values are decimals from `0` to `1` + (interpreted as a float64), which indicate the probability that any given request is sampled. + An example value is `0.5`, which gives each request a 50% sampling probablity. + +### Viewing your `config-observability` ConfigMap + +To view your current configuration: + +```bash +kubectl -n knative-eventing get configmap config-observability -oyaml +``` + +### Editing and deploying your `config-observability` ConfigMap + +To edit and then immediately deploy changes to your ConfigMap, run the following command: + +```bash +kubectl -n knative-eventing edit configmap config-observability +```