[Bug]: A2A HTTP protocol upgrade fails when using Otel auto instrumentation due Netty

Vấn đề nằm ở quá trình thương lượng nâng cấp HTTP/2 khi tính năng tự động instrument của OTEL được bật. Hãy xem mã giao tiếp HTTP của A2A client, có thể nằm trong io.a2a.client.http.A2ACardResolver hoặc các handler Netty/Vert.x liên quan. Tái hiện bằng cách chạy một ứng dụng với OTEL Java agent và debug JAVA_OPTS để xem quá trình nâng cấp thất bại. Bản sửa có thể cần điều chỉnh instrumentation của Netty hoặc cấu hình client để xử lý việc nâng cấp chính xác khi chạy dưới OTEL.

What happened?

Related to this issue: https://github.com/a2aproject/a2a-java/issues/502

When adding OTEL auto instrumentation to my application, by using the Java OTEL agent, the A2A client fails to negotiate an upgrade to HTTP 2. Look below for the logs in the application:

The reason behind the issue:

The OTel Java agent's Netty bytecode instrumentation breaks Vert.x's HTTP/2 cleartext (h2c) upgrade handling. When Java's  HttpClient (used by the A2A SDK) sends Upgrade: h2c, the instrumented server drops the connection instead of responding with 101 Switching Protocols.

To fix the issue:

Added OTEL_INSTRUMENTATION_NETTY_ENABLED=false to the cooking-agent and delivery-agent K8s manifests. This disables only the
  Netty-level instrumentation while keeping higher-level Quarkus/Vert.x observability intact.

If you add the following JAVA_OPTS to your application, you can see the requests being created by the A2A library and how it fails to upgrade the connection only when OTEL instrumentation is on.

- name: JAVA_OPTS
   value: -Djdk.httpclient.HttpClient.log="errors,requests,headers,frames:control:data:window,ssl,trace,channel"

Relevant log output

Application Error:

 [io.quarkus.vertx.http.runtime.QuarkusErrorHandler] (executor-thread-2) HTTP Request to /mgmt/processOrder failed, error id: 1cdeb479-c8fe-41f0-9802-3fe3e05490ec-3:                         
  jakarta.enterprise.inject.CreationException: Error creating synthetic bean [aLyk-Wj_mwEcG8EdDSX8kol41oQ]: java.lang.RuntimeException: io.a2a.spec.A2AClientError: Failed to obtain agent     
  card                                                                                                                                                                                         
          at com.pizzavibe.store.workflows.PizzaOrderWorkflow_aLyk-Wj_mwEcG8EdDSX8kol41oQ_Synthetic_Bean.create(Unknown Source)                                                                
          at com.pizzavibe.store.workflows.PizzaOrderWorkflow_aLyk-Wj_mwEcG8EdDSX8kol41oQ_Synthetic_Bean.create(Unknown Source)                                                                
          at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)                                                                                    
          at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)                                                                                                    
          at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)                                                                                                    
          at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.ch24(Unknown Source)                                                               
          at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source)                                                    
          at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)                                                                                                      
          at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:23)                                                                                             
          at com.pizzavibe.store.workflows.PizzaOrderWorkflow_aLyk-Wj_mwEcG8EdDSX8kol41oQ_Synthetic_ClientProxy.arc$delegate(Unknown Source)                                                   
          at com.pizzavibe.store.workflows.PizzaOrderWorkflow_aLyk-Wj_mwEcG8EdDSX8kol41oQ_Synthetic_ClientProxy.processPizzaOrder(Unknown Source)                                              
          at com.pizzavibe.store.StoreMgmtResource.processOrder(StoreMgmtResource.java:48)                                                                                                     
          at com.pizzavibe.store.StoreMgmtResource$quarkusrestinvoker$processOrder_0f393797cf8d9883eee70376b7ab7d4e45e97c49.invoke(Unknown Source)                                             
          at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)                                                                                   
          at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:190)                                   
          at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)                                                             
          at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:666)                                                                                            
          at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)                                                                                           
          at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)                                                                                                 
          at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)                                                                                            
          at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)                                                                                           
          at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)                                                                                                              
          at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)                                                                                          
          at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                                                                                             
          at java.base/java.lang.Thread.run(Thread.java:1583)                                                                                                                                  
  Caused by: java.lang.RuntimeException: io.a2a.spec.A2AClientError: Failed to obtain agent card                                                                                               
          at dev.langchain4j.agentic.a2a.DefaultA2AClientBuilder.agentCard(DefaultA2AClientBuilder.java:83)                                                                                    
          at dev.langchain4j.agentic.a2a.DefaultA2AClientBuilder.<init>(DefaultA2AClientBuilder.java:63)                                                                                       
          at dev.langchain4j.agentic.a2a.DefaultA2AService.a2aBuilder(DefaultA2AService.java:24)                                                                                               
          at dev.langchain4j.agentic.AgenticServices.a2aBuilder(AgenticServices.java:271)                                                                                                      
          at dev.langchain4j.agentic.AgenticServices.createA2AClientAgent(AgenticServices.java:786)                                                                                            
          at dev.langchain4j.agentic.AgenticServices.createBuiltInAgentExecutor(AgenticServices.java:764)                                                                                      
          at dev.langchain4j.agentic.AgenticServices.createSubagent(AgenticServices.java:700)                                                                                                  
          at dev.langchain4j.agentic.AgenticServices.lambda$createSubagents$0(AgenticServices.java:694)                                                                                        
          at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)                                                                                               
          at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)                                                                                        
          at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)                                                                                                   
          at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)                                                                                            
          at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)                                                                                                   
          at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)                                                                                        
          at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)                                                                                                  
          at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)                                                                                                  
          at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)                                                                                                   
          at dev.langchain4j.agentic.AgenticServices.createSubagents(AgenticServices.java:695)                                                                                                 
          at dev.langchain4j.agentic.AgenticServices.buildSequentialAgent(AgenticServices.java:394)                                                                                            
          at dev.langchain4j.agentic.AgenticServices.createComposedAgent(AgenticServices.java:355)                                                                                             
          at dev.langchain4j.agentic.AgenticServices.createAgenticSystem(AgenticServices.java:324)                                                                                             
          at io.quarkiverse.langchain4j.agentic.runtime.AgenticRecorder$1.apply(AgenticRecorder.java:52)                                                                                       
          at io.quarkiverse.langchain4j.agentic.runtime.AgenticRecorder$1.apply(AgenticRecorder.java:36)                                                                                       
          at com.pizzavibe.store.workflows.PizzaOrderWorkflow_aLyk-Wj_mwEcG8EdDSX8kol41oQ_Synthetic_Bean.createSynthetic(Unknown Source)                                                       
          ... 25 more                                                                                                                                                                          
  Caused by: io.a2a.spec.A2AClientError: Failed to obtain agent card                                                                                                                           
          at io.a2a.client.http.A2ACardResolver.getAgentCard(A2ACardResolver.java:105)                                                                                                         
          at io.a2a.A2A.getAgentCard(A2A.java:187)                                                                                                                                             
          at io.a2a.A2A.getAgentCard(A2A.java:155)                                                                                                                                             
          at io.a2a.A2A.getAgentCard(A2A.java:142)                                                                                                                                             
          at dev.langchain4j.agentic.a2a.DefaultA2AClientBuilder.agentCard(DefaultA2AClientBuilder.java:81)                                                                                    
          ... 48 more                                                                                                                                                                          
  Caused by: java.io.IOException: HTTP/1.1 header parser received no bytes                                                                                                                     
          at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:970)                                                                                                  
          at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)                                                                                              
          at io.a2a.client.http.JdkA2AHttpClient$JdkGetBuilder.get(JdkA2AHttpClient.java:212)                                                                                                  
          at io.a2a.client.http.A2ACardResolver.getAgentCard(A2ACardResolver.java:99)                                                                                                          
          ... 52 more                                                                                                                                                                          
  Caused by: java.io.IOException: HTTP/1.1 header parser received no bytes                                                                                                                     
          at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Utils.java:412)                                                                                              
          at java.net.http/jdk.internal.net.http.Http1Response$HeadersReader.onReadError(Http1Response.java:590)                                                                               
          at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Http1AsyncReceiver.java:302)                                                                                
          at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:268)                                                                                         
          at java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:182)                                                           
          at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149)                                                          
          at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:207)                                                                  
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)                                                                                         
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)                                                                                         
          ... 1 more                                                                                                                                                                           
  Caused by: java.io.EOFException: EOF reached while reading                                                                                                                                   
          at java.net.http/jdk.internal.net.http.Http1AsyncReceiver$Http1TubeSubscriber.onComplete(Http1AsyncReceiver.java:601)                                                                
          at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadSubscription.signalCompletion(SocketTube.java:648)                                                       
          at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:853)                                                           
          at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:181)                                                                                            
          at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:207)                                                                  
          at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:280)                                                                        
          at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:233)                                                                        
          at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:782)                                                 
          at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:965)                                                                   
          at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:253)                                                                                        
          at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:1477)                                                                          
          at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:1422)                                                                         
          at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)                                                                                                                        
          at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:1422)

Code of Conduct

• I agree to follow this project's Code of Conduct