stats/opentelemetry: Introduce Tracing API #7852

aranjans · 2024-11-18T05:20:32Z

Overview

This pull request implements the OpenTelemetry tracing support in the gRPC-Go library as outlined in proposal A72. The implementation provides a robust framework for tracing gRPC calls using OpenTelemetry, facilitating a smooth migration path from OpenCensus tracing.

Key Features

OpenTelemetry Tracing API: Introduces a new API for enabling and configuring OpenTelemetry tracing within gRPC. This includes the addition of TraceOptions in the Options struct to allow users to specify their TraceProvider.
Context Propagation: Implements context propagation between gRPC clients and servers using OpenTelemetry's TextMapPropagator. This ensures that trace context is correctly passed along with RPC calls.
Migration Path: Provides a clear migration path from OpenCensus to OpenTelemetry, allowing users to transition their tracing implementations without breaking existing functionality. This includes support for both cross-process and in-binary migration scenarios(this will be added in separate PR).
Tracing Information: Captures detailed tracing information during the RPC lifecycle, including events for outbound and inbound messages, retries, and load balancer delays. This information is essential for monitoring and debugging distributed systems.

RELEASE NOTES:

stats/opentelemetry: Added OpenTelemetry tracing support in gRPC-Go, enabling enhanced observability and a migration path from OpenCensus.

codecov · 2024-11-18T05:25:38Z

Codecov Report

Attention: Patch coverage is 84.19244% with 46 lines in your changes missing coverage. Please review.

Project coverage is 82.03%. Comparing base (c63aeef) to head (8e78478).
Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
stats/opentelemetry/trace.go	77.77%	13 Missing and 5 partials ⚠️
...entelemetry/internal/tracing/custom_map_carrier.go	68.29%	11 Missing and 2 partials ⚠️
stats/opentelemetry/client_metrics.go	86.48%	7 Missing and 3 partials ⚠️
stats/opentelemetry/grpc_trace_bin_propagator.go	87.50%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7852      +/-   ##
==========================================
+ Coverage   81.80%   82.03%   +0.22%     
==========================================
  Files         375      378       +3     
  Lines       37978    38233     +255     
==========================================
+ Hits        31068    31363     +295     
+ Misses       5609     5568      -41     
- Partials     1301     1302       +1

Files with missing lines	Coverage Δ
clientconn.go	`92.42% <100.00%> (+0.27%)`	⬆️
stats/opentelemetry/opentelemetry.go	`76.82% <100.00%> (+0.95%)`	⬆️
stats/opentelemetry/server_metrics.go	`90.81% <100.00%> (+1.43%)`	⬆️
stream.go	`81.47% <100.00%> (+0.03%)`	⬆️
stats/opentelemetry/grpc_trace_bin_propagator.go	`87.50% <87.50%> (ø)`
stats/opentelemetry/client_metrics.go	`86.34% <86.48%> (-1.59%)`	⬇️
...entelemetry/internal/tracing/custom_map_carrier.go	`68.29% <68.29%> (ø)`
stats/opentelemetry/trace.go	`77.77% <77.77%> (ø)`

... and 25 files with indirect coverage changes

---- 🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

purnesh42H · 2024-11-20T04:51:37Z

examples/features/opentelemetry/client/main.go

@@ -34,7 +34,10 @@ import (

 	"github.com/prometheus/client_golang/prometheus/promhttp"
 	"go.opentelemetry.io/otel/exporters/prometheus"
+	"go.opentelemetry.io/otel/propagation"


let's remove the examples from this PR

purnesh42H · 2024-11-20T04:54:30Z

gcp/observability/go.mod

@@ -1,6 +1,6 @@
 module google.golang.org/grpc/gcp/observability

-go 1.22
+go 1.22.7


why is this changed? I think the OpenTelemetry plugin is stabalised already stabilised now #7759

purnesh42H · 2024-11-20T04:55:41Z

clientconn.go

@@ -604,6 +605,9 @@ type ClientConn struct {
 	idlenessMgr         *idle.Manager
 	metricsRecorderList *stats.MetricsRecorderList

+	// Tracks if there was a delay in name resolution.


nit: To track

purnesh42H · 2024-11-20T04:56:32Z

interop/xds/go.sum

@@ -77,6 +77,8 @@ google.golang.org/genproto/googleapis/api v0.0.0-20241015192408-796eee8c2d53 h1:
 google.golang.org/genproto/googleapis/api v0.0.0-20241015192408-796eee8c2d53/go.mod h1:riSXTwQ4+nqmPGtobMFyW5FqVAmIs0St6VPp4Ug7CE4=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20241015192408-796eee8c2d53 h1:X58yt85/IXCx0Y3ZwN6sEIKZzQtDEYaBWrDvErdXrRE=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20241015192408-796eee8c2d53/go.mod h1:GX3210XPVPUjJbTUbvwI8f2IpZDMZuPJWDzDuebbviI=
+google.golang.org/grpc/stats/opentelemetry v0.0.0-20241028142157-ada6787961b3 h1:hUfOButuEtpc0UvYiaYRbNwxVYr0mQQOWq6X8beJ9Gc=


same. all these go.mod and go.sum files shouldn't need to change

purnesh42H · 2024-11-20T05:02:06Z

stats/opentelemetry/client_metrics.go

@@ -119,22 +134,46 @@ func (h *clientStatsHandler) streamInterceptor(ctx context.Context, desc *grpc.S
 	}

 	startTime := time.Now()
+	ctx, span := h.createCallTraceSpan(ctx, method)


i think we should just not call createCallTraceSpan if tracing is disabled

purnesh42H · 2024-11-20T05:05:18Z

stats/opentelemetry/client_metrics.go

+// in the context provided if created.
+func (h *clientStatsHandler) createCallTraceSpan(ctx context.Context, method string) (context.Context, trace.Span) {
+	var span trace.Span
+	if !isTracingDisabled(h.options.TraceOptions) {


we should just not call this when tracing is disabled. It should probably just panic if tracer is not there instead of silently supressing it because there will be failures eventually later in rpc lifecycle.

purnesh42H · 2024-11-20T05:06:08Z

stats/opentelemetry/client_metrics.go

+		if info.NameResolutionDelay {
+			callSpan.AddEvent("Delayed name resolution complete")
+		}
+		ctx = trace.ContextWithSpan(ctx, callSpan)


can be one line ctx, ti = h.traceTagRPC(trace.ContextWithSpan(ctx, callSpan), info)

purnesh42H · 2024-11-20T05:11:21Z

stats/opentelemetry/client_metrics.go

+		h.processRPCEvent(ctx, rs, ri.ai)
+	}
+	if !isTracingDisabled(h.options.TraceOptions) {
+		populateSpan(ctx, rs, ri.ai.ti)


may be we should make this also h.populateSpan

purnesh42H · 2024-11-20T05:15:27Z

stats/opentelemetry/trace.go

+// current span, message counters for sent and received messages (used for
+// generating message IDs), and the number of previous RPC attempts for the
+// associated call.
+type attemptTraceSpan struct {


It should be just attemptInfo similar to callInfo and it can have a field traceSpan

purnesh42H · 2024-11-20T05:19:46Z

stats/opentelemetry/opentelemetry.go

@@ -181,6 +205,8 @@ type attemptInfo struct {

 	pluginOptionLabels map[string]string // pluginOptionLabels to attach to metrics emitted
 	xdsLabels          map[string]string
+
+	ti *attemptTraceSpan


Same. Change to attemptInfo. We should restrict to traces. attemptInfo can have traces, metrics etc. together as separate fields within

purnesh42H · 2024-11-20T05:24:21Z

@aranjans please update this with latest main branch to get rid of all go.mod and go.sum changes

aranjans · 2024-11-22T01:18:19Z

@purnesh42H Thanks for your review, I have addressed all your comments and this PR is ready for another pass.

purnesh42H

Some more non-test comments. Will review tests in next pass

purnesh42H · 2024-11-22T05:08:43Z

clientconn.go

@@ -604,6 +605,9 @@ type ClientConn struct {
 	idlenessMgr         *idle.Manager
 	metricsRecorderList *stats.MetricsRecorderList

+	// Track if there was a delay in name resolution.


nit: To track

purnesh42H · 2024-11-22T05:13:03Z

stats/handlers.go

+	NameResolutionDelay bool
+	// IsTransparentRetry indicates whether the stream is undergoing a
+	// transparent retry.
+	IsTransparentRetry bool


this is also valid on client side only?

Also, it looks like its already being passed to HandleRPC https://github.com/grpc/grpc-go/blob/master/stream.go#L428. Do we have to track it still? We only need to add it while populating span which can be done directly under begin case

Yeah this was already handled earlier, but somehow while rebasing with my earlier commits, this came. In the populateSpan, we were already using from Begin handler here.

purnesh42H · 2024-11-22T05:15:31Z

stats/opentelemetry/client_metrics.go

+// in the context provided if created.
+func (h *clientStatsHandler) createCallTraceSpan(ctx context.Context, method string) (context.Context, trace.Span) {
+	if h.options.TraceOptions.TracerProvider == nil {
+		panic("tracing is required but the TracerProvider is not set. Ensure that tracing is enabled in the options.")


no, need to panic forcefully. You can just return ctx, nil and log an error

If you don't check at all, it will panic anyway

purnesh42H · 2024-11-22T05:17:05Z

stream.go

@@ -416,8 +417,10 @@ func (cs *clientStream) newAttemptLocked(isTransparent bool) (*csAttempt, error)
 	method := cs.callHdr.Method
 	var beginTime time.Time
 	shs := cs.cc.dopts.copts.StatsHandlers
+	nameResolutionDelayed := false


why do you have to assign twice. can assign cs.nameResolutionDelayed directly?

purnesh42H · 2024-11-22T05:19:10Z

stats/opentelemetry/client_metrics.go

@@ -119,22 +138,47 @@ func (h *clientStatsHandler) streamInterceptor(ctx context.Context, desc *grpc.S
 	}

 	startTime := time.Now()
-
+	var span trace.Span


this should be a pointer as it will be passed around a lot

purnesh42H · 2024-11-22T05:21:04Z

stats/opentelemetry/client_metrics.go

@@ -85,8 +100,12 @@ func (h *clientStatsHandler) unaryInterceptor(ctx context.Context, method string
 	}

 	startTime := time.Now()
+	var span trace.Span


same. should be pointer.

purnesh42H · 2024-11-22T05:22:00Z

stats/opentelemetry/client_metrics.go

-	))
-	h.clientMetrics.callDuration.Record(ctx, callLatency, attrs)
+func (h *clientStatsHandler) perCallTracesAndMetrics(ctx context.Context, err error, startTime time.Time, ci *callInfo, ts trace.Span) {
+	s := status.Convert(err)


this is only required for tracing? so should be inside tracing condition

purnesh42H · 2024-11-22T05:22:58Z

stats/opentelemetry/client_metrics.go

-		otelattribute.String("grpc.status", canonicalString(status.Code(err))),
-	))
-	h.clientMetrics.callDuration.Record(ctx, callLatency, attrs)
+func (h *clientStatsHandler) perCallTracesAndMetrics(ctx context.Context, err error, startTime time.Time, ci *callInfo, ts trace.Span) {


may be we should have a docstring for this method. Can be simple like "perCallTracesAndMetrics records per call trace spans and metrics."

purnesh42H · 2024-11-22T05:25:05Z

stats/opentelemetry/client_metrics.go

+	}
+}
+
+// createCallTraceSpan creates a call span if tracing is enabled, which will be put


creates a call span to put in the provided context using provided TraceProvider. If TraceProvider is nil, it returns context as is.

purnesh42H

Some more non-test comments. Will review tests in next pass

purnesh42H · 2024-11-22T05:36:44Z

@aranjans you should link the gRFC and concise your description

aranjans · 2024-11-23T16:05:05Z

@purnesh42H I have addressed all the comments, and updated the description to link the grfc proposal.
Feel free to close the thread which are resolved now.

purnesh42H · 2024-11-25T07:37:01Z

stats/opentelemetry/client_metrics.go

@@ -68,6 +74,15 @@ func (h *clientStatsHandler) initializeMetrics() {
 	rm.registerMetrics(metrics, meter)
 }

+func (h *clientStatsHandler) initializeTracing() {
+	if h.options.TraceOptions.TracerProvider == nil || h.options.TraceOptions.TextMapPropagator == nil {


we can just call isTracingDisabled here?

purnesh42H · 2024-11-25T07:39:47Z

stats/opentelemetry/client_metrics.go

@@ -85,8 +100,12 @@ func (h *clientStatsHandler) unaryInterceptor(ctx context.Context, method string
 	}

 	startTime := time.Now()
+	var span *trace.Span
+	if !isTracingDisabled(h.options.TraceOptions) {


how about we pass the method to perCallTracesAndMetrics and create the trace span there if tracing is not disabled? it is because we are already checking traces disable in perCallTracesAndMetrics

We need to have create call span here as we need to add event for "name resolution delay", and this info is only available in RPCTagInfo. One alternative way is to have a struct for nameResolutionDelay as a key of context metadata, but I don't think it'd be good idea to do that.

Based on offline discussion, going ahead with earlier approach.

purnesh42H · 2024-11-25T07:40:01Z

stats/opentelemetry/client_metrics.go

@@ -119,22 +138,50 @@ func (h *clientStatsHandler) streamInterceptor(ctx context.Context, desc *grpc.S
 	}

 	startTime := time.Now()
-
+	var span *trace.Span
+	if !isTracingDisabled(h.options.TraceOptions) {


same suggestion here

purnesh42H · 2024-11-25T07:42:08Z

stats/opentelemetry/client_metrics.go

+// provided TraceProvider. If TraceProvider is nil, it returns context as is.
+func (h *clientStatsHandler) createCallTraceSpan(ctx context.Context, method string) (context.Context, *trace.Span) {
+	if h.options.TraceOptions.TracerProvider == nil {
+		logger.Error("tracing is required but the TracerProvider is not set. Ensure that tracing is enabled in the options.")


error can be simply "TracerProvider is not supplied/provided in trace options"

purnesh42H · 2024-11-25T07:47:07Z

stats/opentelemetry/client_metrics.go

+		return setRPCInfo(ctx, &rpcInfo{
+			ai: &attemptInfo{
+				startTime:           time.Now(),
+				xdsLabels:           labels.TelemetryLabels,


i think its fine to populate startTime: time.Now(), xdsLabels: labels.TelemetryLabels, method: info.FullMethodName before and set trace stuff inside if so that you don't duplicate below

purnesh42H · 2024-11-25T07:49:28Z

stats/opentelemetry/opentelemetry.go

@@ -84,6 +88,16 @@ type MetricsOptions struct {
 	pluginOption otelinternal.PluginOption
 }

+// TraceOptions are the tracing options for OpenTelemetry instrumentation.
+type TraceOptions struct {
+	// TracerProvider provides Tracers that are used by instrumentation code to


// TracerProvider is the OpenTelemetry tracer which is required to record traces/trace spans for instrumentation

purnesh42H · 2024-11-25T08:23:10Z

stats/opentelemetry/server_metrics.go

+		ctx, ai = h.traceTagRPC(ctx, info)
+		return setRPCInfo(ctx, &rpcInfo{
+			ai: &attemptInfo{
+				startTime:           time.Now(),


same comment here regarding assigning common fields before and flip the condition

ai: attemptInfo{...}
if tracingDisabled(...) {
return setRPCInfo(ctx, ri)
}

purnesh42H · 2024-11-25T08:24:54Z

stats/opentelemetry/trace.go

+	"google.golang.org/grpc/status"
+)
+
+// traceTagRPC populates context with a new span, and serializes information


// traceTagRPC populates provided context with a new span using the TextMapPropagator supplied in trace options and internal itracing.carrier. It creates a new outgoing carrier which serializes information about this span into gRPC Metadata, if TextMapPropagator is provided in the trace options. if TextMapPropagator is not provided, it returns the context as is.

purnesh42H · 2024-11-25T08:29:41Z

stats/opentelemetry/trace.go

+	}
+}
+
+// traceTagRPC populates context with new span data, with a parent based on the


// traceTagRPC populates context with new span data using the TextMapPropagator supplied in trace options and internal itracing.Carrier. It creates a new incoming carrier which extracts an existing span context (if present) by deserializing from provided context. If valid span context is extracted, it is set as parent of the new span otherwise new span remains the root span. If TextMapPropagator is not provided in the trace options, it returns context as is.

aranjans · 2024-11-30T18:32:36Z

@purnesh42H I have addressed all your comments, and updated the PR. Kindly review the PR.

aranjans added this to the 1.69 Release milestone Nov 18, 2024

aranjans added Type: Feature New features or improvements in behavior Area: Observability Includes Stats, Tracing, Channelz, Healthz, Binlog, Reflection, Admin, GCP Observability labels Nov 18, 2024

aranjans requested a review from purnesh42H November 18, 2024 08:22

purnesh42H self-assigned this Nov 19, 2024

purnesh42H reviewed Nov 20, 2024

View reviewed changes

purnesh42H assigned aranjans and unassigned purnesh42H Nov 20, 2024

aranjans force-pushed the a72 branch from 559dd4c to 58eb051 Compare November 21, 2024 05:57

aranjans added 6 commits November 21, 2024 13:30

Implement A72: OpenTelemetry Tracing

cfb49fc

Add go.mod for stats/opentelemetry

3dff313

Update examples go.mod

4ae07dd

Rebase with Purnesh's PR

38eabb9

refactor and exclude otel from examples

30257cf

Revert go.mod files

0e01810

aranjans force-pushed the a72 branch from 58eb051 to 0e01810 Compare November 21, 2024 08:00

aranjans added 4 commits November 21, 2024 13:39

Rebase with grpc-go master

cea60d4

Rebase with grpc-go master

39f7693

remove attemptTraceInfo and use attemptInfo instead

2a26fc2

Use NewOutgoingCarrier to inject into GRPCTraceBinPropagator

8cb8222

aranjans force-pushed the a72 branch from 378ddd3 to 8cb8222 Compare November 21, 2024 17:44

aranjans added 2 commits November 21, 2024 23:31

rename commonHandler to statsHandler

b9f23ee

nits

729713b

aranjans assigned purnesh42H and unassigned aranjans Nov 22, 2024

purnesh42H reviewed Nov 22, 2024

View reviewed changes

make vet happy

382d053

purnesh42H reviewed Nov 22, 2024

View reviewed changes

purnesh42H assigned aranjans and unassigned purnesh42H Nov 22, 2024

purnesh42H changed the title ~~Implement A72: OpenTelemetry Tracing~~ stats/opentelemetry: Introduce Tracing API Nov 22, 2024

addressed nits and comments from Purnesh

8e78478

aranjans assigned purnesh42H and unassigned aranjans Nov 23, 2024

purnesh42H reviewed Nov 25, 2024

View reviewed changes

purnesh42H assigned aranjans and unassigned purnesh42H Nov 25, 2024

aranjans and others added 3 commits November 30, 2024 23:12

addressed comments

60bb987

Rebase with master

82bdbd7

Merge branch 'master' into a72

ce57921

aranjans force-pushed the a72 branch from f50a6b9 to ce57921 Compare November 30, 2024 18:29

aranjans assigned purnesh42H and unassigned aranjans Nov 30, 2024

stats/opentelemetry: Introduce Tracing API #7852

Are you sure you want to change the base?

stats/opentelemetry: Introduce Tracing API #7852

Conversation

aranjans commented Nov 18, 2024 • edited Loading

Overview

Key Features

codecov bot commented Nov 18, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aranjans Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

purnesh42H commented Nov 20, 2024

aranjans commented Nov 22, 2024

purnesh42H left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aranjans Nov 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

purnesh42H left a comment

Choose a reason for hiding this comment

purnesh42H commented Nov 22, 2024

aranjans commented Nov 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aranjans commented Nov 30, 2024

aranjans commented Nov 18, 2024 •

edited

Loading

codecov bot commented Nov 18, 2024 •

edited

Loading

aranjans Nov 20, 2024 •

edited

Loading

aranjans Nov 23, 2024 •

edited

Loading

aranjans commented Nov 23, 2024 •

edited

Loading