gRPC Context propagation using OTel agent

[kvajjala59]

I have a Spring Boot App with java 11 using GRPC extensively. We do not propagate the context in our code. But it looks like if OTel agent is enabled then it automatically propagates the context across thread pools.

Assuming that OTel agent does indeed propagate the context, I have a few questions

1. Does OTel agent modify the Java bytecode for CompletableFuture to ensure Context is propagated correctly to all chained tasks
2. Does adding manual propagation interfere with the OTel agent context propagation

[steverao]

1. Does OTel agent modify the Java bytecode for CompletableFuture to ensure Context is propagated correctly to all chained tasks

We have done something to support context propagation asynchronously. For instrumentations for some JDK thread pool, you can refer to https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/executors

2. Does adding manual propagation interfere with the OTel agent context propagation

Why do you want to add manual propagation based on OTel agent or why don't use OTel agent directly?

[kvajjala59]

Thanks for the response.

Typically we expect the application behavior not change because of OTel agent. as a gRPC service we would like to propagate the context across thread boundaries and we want that to be part of the application rather than the Agent. I understand that Agent does propagate the gRPC context as well (although indirectly) but I do not want to modify my application because the Agent is doing it.

The agent will not be aware of what the application is doing so when I enable the agent, the application propagation and agent propagation via the instrumentation both will work at the same time.

I wanted to understand if it will result in any interference. In some cases I see an exception

exception: java.lang.Exception: null
at io.grpc.Context.validateGeneration(Context.java:1110)
at io.grpc.Context.<init>(Context.java:202)
at io.grpc.Context.withValue(Context.java:345)
at i.o.j.s.i.g.v.i.ContextStorageBridge.doAttach(ContextStorageBridge.java:82)
at io.grpc.Context.attach(Context.java:427)
at io.grpc.Context$1.run(Context.java:564)
at i.m.c.i.composite.CompositeTimer.record(CompositeTimer.java:89)
at i.m.core.instrument.Timer.lambda$wrap$0(Timer.java:152)
at j.b.u.c.Executors$RunnableAdapter.call(Executors.java:515)
at j.b.util.concurrent.FutureTask.run(FutureTask.java:264)
at j.b.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at j.b.u.c.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at j.b.u.c.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

level: ERROR

logger: io.grpc.Context
span_id: 915c94ccd34a1597
trace_flags: 00
trace_id: fbfb2d8419cef0198b1f257e3467cd18
}

message: Context ancestry chain length is abnormally long. This suggests an error in application code. Length exceeded: 1000

I do not see this issue when Agent is not running

[laurit]

@kvajjala59 consider providing a sample application that reproduces the issue. We need to get a better understanding of the issue, I guess it is possible that we could add a switch that disables otel context propagation via the grpc context.

[steverao]

Context ancestry chain length is abnormally long. This suggests an error in application code. Length exceeded: 1000

It seems that is influenced by context propagation of OTel Java, there is a similar issue: #14239 cc @laurit