Java VerifyError occurs when using com.google.cloud libraries-bom > 26.31.0 #10698

rnaval · 2024-04-15T09:25:45Z

Environment details

API tested: CloudFilestoreManagerClient.create(), SubscriptionAdminClient.create()
OS type and version: CentOS 7 docker
Java version: Java 17
Version(s):

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>libraries-bom</artifactId>
        <version>26.37.0</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-monitoring</artifactId>
    </dependency>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-compute</artifactId>
    </dependency>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-storage</artifactId>
    </dependency>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-filestore</artifactId>
    </dependency>
    <dependency>
      <groupId>com.google.protobuf</groupId>
      <artifactId>protobuf-java-util</artifactId>
      <version>4.26.1</version>
    </dependency>
  </dependencies>

Code example

  private CloudFilestoreManagerClient createClient() {
    try {
      return CloudFilestoreManagerClient.create();
    } catch (Exception e) {
      logger.get().error("{} Error when creating client for CloudFilestoreManagerClient : {}",
          this.getClass().getSimpleName(), e.getMessage());
    }
    return null;
  }

Stack trace

java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    com/google/cloud/filestore/v1/Instance.internalGetMapFieldReflection(I)Lcom/google/protobuf/MapFieldReflectionAccessor; @24: areturn
  Reason:
    Type 'com/google/protobuf/MapField' (current frame, stack[0]) is not assignable to 'com/google/protobuf/MapFieldReflectionAccessor' (from method signature)
  Current Frame:
    bci: @24
    flags: { }
    locals: { 'com/google/cloud/filestore/v1/Instance', integer }
    stack: { 'com/google/protobuf/MapField' }
  Bytecode:
    0000000: 1bab 0000 0000 0018 0000 0001 0000 0009
    0000010: 0000 0013 2ab7 0004 b0bb 001d 59bb 001e
    0000020: 59b7 001f 1220 b600 211b b600 22b6 0023
    0000030: b700 24bf                              
  Stackmap Table:
    same_frame(@20)
    same_frame(@25)

	at com.google.cloud.filestore.v1.stub.GrpcCloudFilestoreManagerStub.<clinit>(GrpcCloudFilestoreManagerStub.java:88)
	at com.google.cloud.filestore.v1.stub.CloudFilestoreManagerStubSettings.createStub(CloudFilestoreManagerStubSettings.java:494)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.<init>(CloudFilestoreManagerClient.java:517)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.create(CloudFilestoreManagerClient.java:499)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.create(CloudFilestoreManagerClient.java:490)

Another one:

java.lang.VerifyError: Bad return type
Exception Details:
Location:
com/google/pubsub/v1/Subscription.internalGetMapFieldReflection(I)Lcom/google/protobuf/MapFieldReflectionAccessor; @24: areturn
Reason:
Type 'com/google/protobuf/MapField' (current frame, stack[0]) is not assignable to 'com/google/protobuf/MapFieldReflectionAccessor' (from method signature)
Current Frame:
bci: @24
flags: { }
locals: { 'com/google/pubsub/v1/Subscription', integer }
stack: { 'com/google/protobuf/MapField' }
Bytecode:
0000000: 1bab 0000 0000 0018 0000 0001 0000 0009
0000010: 0000 0013 2ab7 0007 b0bb 0069 59bb 006b
0000020: 59b7 006d 126e b600 701b b600 74b6 0077
0000030: b700 7bbf
Stackmap Table:
same_frame(@20)
same_frame(@25)

    at com.google.cloud.pubsub.v1.stub.GrpcSubscriberStub.<clinit>(GrpcSubscriberStub.java:78)
    at com.google.cloud.pubsub.v1.stub.SubscriberStubSettings.createStub(SubscriberStubSettings.java:376)
    at com.google.cloud.pubsub.v1.SubscriptionAdminClient.<init>(SubscriptionAdminClient.java:544)
    at com.google.cloud.pubsub.v1.SubscriptionAdminClient.create(SubscriptionAdminClient.java:526)
    at com.google.cloud.pubsub.v1.SubscriptionAdminClient.create(SubscriptionAdminClient.java:517)

Any additional information below

Issue only occurs when we use 26.32.0 onwards. 26.31.0 works fine.

Unit tests seem to be working fine for all versions, but not sure what is happening in the docker image.

The text was updated successfully, but these errors were encountered:

diegomarquezp · 2024-04-16T18:44:11Z

Hi @rnaval, thanks for reporting this. Would you mind sharing the tag of the docker image you are using? Has this occurred when running your java application on your host machine? Thanks!

diegomarquezp · 2024-04-16T18:53:52Z

It looks like protobuf-java-util may be involved (thanks @suztomo).

diegomarquezp · 2024-04-16T19:04:39Z

@rnaval our libraries are not using protobuf 4.26 just yet. Would you mind trying to modify your dependency on java-protobuf-util to ~~3.25.3~~ 3.25.2 (the one our libraries use)?

From https://github.com/googleapis/java-cloud-bom/releases/tag/v26.37.0

rnaval · 2024-04-17T03:42:42Z

Thanks for the response.

Tried it out with the recommended protobuf library, issue still occurs.

java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    com/google/cloud/filestore/v1/Instance.internalGetMapFieldReflection(I)Lcom/google/protobuf/MapFieldReflectionAccessor; @24: areturn
  Reason:
    Type 'com/google/protobuf/MapField' (current frame, stack[0]) is not assignable to 'com/google/protobuf/MapFieldReflectionAccessor' (from method signature)
  Current Frame:
    bci: @24
    flags: { }
    locals: { 'com/google/cloud/filestore/v1/Instance', integer }
    stack: { 'com/google/protobuf/MapField' }
  Bytecode:
    0000000: 1bab 0000 0000 0018 0000 0001 0000 0009
    0000010: 0000 0013 2ab7 0004 b0bb 001d 59bb 001e
    0000020: 59b7 001f 1220 b600 211b b600 22b6 0023
    0000030: b700 24bf                              
  Stackmap Table:
    same_frame(@20)
    same_frame(@25)

	at com.google.cloud.filestore.v1.stub.GrpcCloudFilestoreManagerStub.<clinit>(GrpcCloudFilestoreManagerStub.java:88)
	at com.google.cloud.filestore.v1.stub.CloudFilestoreManagerStubSettings.createStub(CloudFilestoreManagerStubSettings.java:494)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.<init>(CloudFilestoreManagerClient.java:517)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.create(CloudFilestoreManagerClient.java:499)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.create(CloudFilestoreManagerClient.java:490)

Docker image is based on CentOS 7:

FROM centos:7
...

Here's a look at the dependency tree:

INFO] +- com.google.cloud:google-cloud-monitoring:jar:3.42.0:compile
[INFO] |  +- io.grpc:grpc-api:jar:1.62.2:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.26.1:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-protobuf:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-protobuf-lite:jar:1.62.2:runtime
[INFO] |  +- com.google.api:api-common:jar:2.29.1:compile
[INFO] |  +- com.google.auto.value:auto-value-annotations:jar:1.10.4:compile
[INFO] |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  +- com.google.j2objc:j2objc-annotations:jar:3.0.0:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:3.25.2:compile
[INFO] |  +- com.google.api.grpc:proto-google-common-protos:jar:2.37.1:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-monitoring-v3:jar:3.42.0:compile
[INFO] |  +- com.google.guava:guava:jar:33.1.0-jre:provided
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.2:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:3.42.0:compile
[INFO] |  +- com.google.api:gax:jar:2.46.1:compile
[INFO] |  +- com.google.auth:google-auth-library-credentials:jar:1.23.0:compile
[INFO] |  +- io.opencensus:opencensus-api:jar:0.31.1:compile
[INFO] |  +- io.grpc:grpc-context:jar:1.62.2:compile
[INFO] |  +- io.opentelemetry:opentelemetry-api:jar:1.36.0:compile
[INFO] |  +- io.opentelemetry:opentelemetry-context:jar:1.36.0:compile
[INFO] |  +- com.google.api:gax-grpc:jar:2.46.1:compile
[INFO] |  +- io.grpc:grpc-inprocess:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-core:jar:1.62.2:compile
[INFO] |  +- com.google.android:annotations:jar:4.1.1.4:runtime
[INFO] |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.23:runtime
[INFO] |  +- io.grpc:grpc-alts:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-grpclb:jar:1.62.2:compile
[INFO] |  +- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.2:compile
[INFO] |  +- io.grpc:grpc-auth:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-netty-shaded:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-util:jar:1.62.2:runtime
[INFO] |  +- io.perfmark:perfmark-api:jar:0.27.0:runtime
[INFO] |  +- io.grpc:grpc-googleapis:jar:1.62.2:runtime
[INFO] |  +- io.grpc:grpc-xds:jar:1.62.2:runtime
[INFO] |  +- io.opencensus:opencensus-proto:jar:0.2.0:runtime
[INFO] |  +- io.grpc:grpc-services:jar:1.62.2:runtime
[INFO] |  +- com.google.re2j:re2j:jar:1.7:runtime
[INFO] |  +- org.threeten:threetenbp:jar:1.6.8:compile
[INFO] |  +- com.google.auth:google-auth-library-oauth2-http:jar:1.23.0:compile
[INFO] |  +- com.google.http-client:google-http-client:jar:1.44.1:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.14:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.16.1:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.16:compile
[INFO] |  +- io.opencensus:opencensus-contrib-http-util:jar:0.31.1:compile
[INFO] |  +- com.google.http-client:google-http-client-gson:jar:1.44.1:compile
[INFO] |  \- com.google.code.gson:gson:jar:2.10.1:compile
[INFO] +- com.google.cloud:google-cloud-compute:jar:1.51.0:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-compute-v1:jar:1.51.0:compile
[INFO] |  \- com.google.api:gax-httpjson:jar:2.46.1:compile
[INFO] +- com.google.cloud:google-cloud-storage:jar:2.36.1:compile
[INFO] |  +- com.google.http-client:google-http-client-jackson2:jar:1.44.1:compile
[INFO] |  +- com.google.api-client:google-api-client:jar:2.4.0:compile
[INFO] |  +- com.google.oauth-client:google-oauth-client:jar:1.35.0:compile
[INFO] |  +- com.google.http-client:google-http-client-apache-v2:jar:1.44.1:compile
[INFO] |  +- com.google.apis:google-api-services-storage:jar:v1-rev20240311-2.0.0:compile
[INFO] |  +- com.google.cloud:google-cloud-core:jar:2.36.1:compile
[INFO] |  +- com.google.cloud:google-cloud-core-http:jar:2.36.1:compile
[INFO] |  +- com.google.http-client:google-http-client-appengine:jar:1.44.1:compile
[INFO] |  +- com.google.cloud:google-cloud-core-grpc:jar:2.36.1:compile
[INFO] |  +- com.google.api.grpc:proto-google-iam-v1:jar:1.32.1:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-storage-v2:jar:2.36.1-alpha:compile
[INFO] |  +- com.google.api.grpc:grpc-google-cloud-storage-v2:jar:2.36.1-alpha:compile
[INFO] |  +- com.google.api.grpc:gapic-google-cloud-storage-v2:jar:2.36.1-alpha:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.16.1:compile
[INFO] |  \- io.grpc:grpc-rls:jar:1.62.2:runtime
[INFO] +- com.google.cloud:google-cloud-filestore:jar:1.42.0:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-filestore-v1:jar:1.42.0:compile
[INFO] |  \- com.google.api.grpc:proto-google-cloud-filestore-v1beta1:jar:0.44.0:compile
[INFO] +- com.google.protobuf:protobuf-java-util:jar:3.25.2:compile

suztomo · 2024-04-17T14:14:43Z

@rnaval Do you observe the problem only happens in the CentOS docker container?

diegomarquezp · 2024-04-17T14:32:05Z

cc: @suztomo
Hi @rnaval, I created a reproducer based on our comments here.

I could run the main function without any issues in IntelliJ. +1 to @suztomo asking if the problem occurs in your host machine as well.

rnaval · 2024-04-29T06:59:01Z

Sorry for the late response. Based on testing results, we have verified that it works on:

MacOS host machine
Windows host machine

We also tried it on a Centos 7 VM - same error was encountered.

rnaval · 2024-05-17T05:15:27Z

@diegomarquezp @suztomo Also tried it out with Ubuntu 22.04 docker, seems to work.

diegomarquezp · 2024-05-17T05:23:19Z

What's the image tag of this CentOS image? Are you using a dockerfile with this image as base or are you running your code directly on it?
Maybe a small reproducer such as a bash script showing how you setup your code can help us find out what's missing in this CentOS image.

If no dockerfile is involved, then we should look at the jdk versions in each environment you tested, or more generally the differences in each execution environment.

mdetemad · 2024-06-17T16:14:26Z

I have the exact same issue after doing 3.25.1 --> 3.25.3.

edporras · 2024-07-17T09:42:47Z

The circleCI used to generate the docs for our project is failing due to this issue as well although the dependency is from Google's Beam SDK :

    [org.apache.beam/beam-sdks-java-io-google-cloud-platform "2.57.0"

       [com.google.protobuf/protobuf-java-util "3.25.3" :exclusions [[org.checkerframework/jdk8] [com.google.guava/guava-jdk5] [jdk.tools] [com.google.protobuf/protobuf-lite] [org.hamcrest/hamcrest-all]]]

https://app.circleci.com/pipelines/github/cljdoc/builder/53299/workflows/d2e886c4-683d-415a-a8dd-040a88745c5f/jobs/69674

hifly81 · 2024-09-29T07:54:01Z

same for me, from 3.23.4 to 3.25.4

hubert-da · 2024-12-12T12:35:59Z

Not sure if this is what is happening in google-cloud-java, but in our case there's been a different library (semanticdb-scalac) that sneaked into our runtime classpath. The library unpacks the protobuf-java library into its own classes under the original (com.google.protobuf) package, which is not a good practice. It should have shaded/unpacked the classes under a different package for its own use. The solution was to exclude the library from the runtime classpath.

diegomarquezp added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Apr 16, 2024

diegomarquezp added type: question Request for information or clarification. Not an issue. and removed type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Apr 16, 2024

suztomo mentioned this issue Jan 9, 2025

[secretmanager] Failed to instantiate SecretManagerServiceClient in Java client library #11373

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Java VerifyError occurs when using com.google.cloud libraries-bom > 26.31.0 #10698

Java VerifyError occurs when using com.google.cloud libraries-bom > 26.31.0 #10698

rnaval commented Apr 15, 2024 •

edited

Loading

diegomarquezp commented Apr 16, 2024 •

edited

Loading

diegomarquezp commented Apr 16, 2024

diegomarquezp commented Apr 16, 2024 •

edited

Loading

rnaval commented Apr 17, 2024

suztomo commented Apr 17, 2024

diegomarquezp commented Apr 17, 2024

rnaval commented Apr 29, 2024

rnaval commented May 17, 2024

diegomarquezp commented May 17, 2024

mdetemad commented Jun 17, 2024

edporras commented Jul 17, 2024 •

edited

Loading

hifly81 commented Sep 29, 2024

hubert-da commented Dec 12, 2024

Java VerifyError occurs when using com.google.cloud libraries-bom > 26.31.0 #10698

Java VerifyError occurs when using com.google.cloud libraries-bom > 26.31.0 #10698

Comments

rnaval commented Apr 15, 2024 • edited Loading

Environment details

Code example

Stack trace

Any additional information below

diegomarquezp commented Apr 16, 2024 • edited Loading

diegomarquezp commented Apr 16, 2024

diegomarquezp commented Apr 16, 2024 • edited Loading

rnaval commented Apr 17, 2024

suztomo commented Apr 17, 2024

diegomarquezp commented Apr 17, 2024

rnaval commented Apr 29, 2024

rnaval commented May 17, 2024

diegomarquezp commented May 17, 2024

mdetemad commented Jun 17, 2024

edporras commented Jul 17, 2024 • edited Loading

hifly81 commented Sep 29, 2024

hubert-da commented Dec 12, 2024

rnaval commented Apr 15, 2024 •

edited

Loading

diegomarquezp commented Apr 16, 2024 •

edited

Loading

diegomarquezp commented Apr 16, 2024 •

edited

Loading

edporras commented Jul 17, 2024 •

edited

Loading