Missing required property 'ShardHealthStats.unassignedPrimaryShards' when using client v 8.16 against elasticsearch 7.17.23

[karlney]

Java API client version

8.16.1

Java version

21.0.4

Elasticsearch Version

7.17.23

Problem description

When upgrading from v 8.15.2 of the client to 8.16.1 I start getting the following errors towards my elasticsearch cluster when fetching the cluster health

co.elastic.clients.transport.TransportException: node: http://localhost:38629/, status: 200, [es/cluster.health] Failed to decode response java.util.concurrent.CompletionException: co.elastic.clients.transport.TransportException: node: http://localhost:38629/, status: 200, [es/cluster.health] Failed to decode response at [email protected]/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) at [email protected]/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) at [email protected]/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) at [email protected]/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at [email protected]/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at app//co.elastic.clients.transport.ElasticsearchTransportBase.lambda$performRequestAsync$0(ElasticsearchTransportBase.java:218) at [email protected]/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) at [email protected]/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) at [email protected]/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at [email protected]/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) at app//co.elastic.clients.transport.rest_client.RestClientHttpClient$1.onSuccess(RestClientHttpClient.java:116) at app//org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:680) at app//org.elasticsearch.client.RestClient$1.completed(RestClient.java:403) at app//org.elasticsearch.client.RestClient$1.completed(RestClient.java:397) at app//org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) at app//org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182) at app//org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) at app//org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) at app//org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at app//org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87) at app//org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40) at app//org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) at app//org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at app//org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at app//org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at app//org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at app//org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at app//org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) at [email protected]/java.lang.Thread.run(Thread.java:1583) Caused by: co.elastic.clients.transport.TransportException: node: http://localhost:38629/, status: 200, [es/cluster.health] Failed to decode response at app//co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse(ElasticsearchTransportBase.java:409) at app//co.elastic.clients.transport.ElasticsearchTransportBase.getApiResponse(ElasticsearchTransportBase.java:367) at app//co.elastic.clients.transport.ElasticsearchTransportBase.lambda$performRequestAsync$0(ElasticsearchTransportBase.java:212) ... 23 more Caused by: co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch.cluster.health.IndexHealthStats: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'ShardHealthStats.unassignedPrimaryShards' (JSON path: indices['document.document-20241025'].shards['0']) (line no=1, column no=739, offset=-1) at app//co.elastic.clients.json.JsonpMappingException.from0(JsonpMappingException.java:134) at app//co.elastic.clients.json.JsonpMappingException.from(JsonpMappingException.java:121) at app//co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:353) at app//co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:333) at app//co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77) at app//co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78) at app//co.elastic.clients.json.ObjectDeserializer.deserializeInner(ObjectDeserializer.java:214) at app//co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:166) at app//co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:157) at app//co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77) at app//co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79) at app//co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) at app//co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:349) at app//co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:333) at app//co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77) at app//co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78) at app//co.elastic.clients.json.ObjectDeserializer.deserializeInner(ObjectDeserializer.java:214) at app//co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:166) at app//co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:157) at app//co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77) at app//co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79) at app//co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) at app//co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse(ElasticsearchTransportBase.java:404) ... 25 more Caused by: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'ShardHealthStats.unassignedPrimaryShards' at app//co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76) at app//co.elastic.clients.elasticsearch.cluster.health.ShardHealthStats.<init>(ShardHealthStats.java:89) at app//co.elastic.clients.elasticsearch.cluster.health.ShardHealthStats.<init>(ShardHealthStats.java:64) at app//co.elastic.clients.elasticsearch.cluster.health.ShardHealthStats$Builder.build(ShardHealthStats.java:276) at app//co.elastic.clients.elasticsearch.cluster.health.ShardHealthStats$Builder.build(ShardHealthStats.java:191) at app//co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:80) at app//co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) at app//co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:349) ... 45 more

Is there some workaround or do we need to wait on a patch?

[l-trotta]

Hello! Unfortunately this kind of things can happen when the client version and the server version start diverging, so this is probably a new field that is mandatory in new server versions, but that old server versions don't have. As the documentation states, the java client is forward compatible, meaning we guarantee its functionalities for versions of the server greater than those of the client, so there's won't be a patch for this.
What you can do as a workaround is disable the check just for that specific call, by performing it inside a try block configured like this:

try (ApiTypeHelper.DisabledChecksHandle h = ApiTypeHelper.DANGEROUS_disableRequiredPropertiesCheck(true)) {
    esClient.healthReport();
}

[karlney]

Thank you for the workaround suggestion. I will try it and report back here. If it works then this ticket can be closed as far as I am concerned.

[axeljohnsson]

I work with @karlney and we tried your suggestion. It would probably have worked if the instance variable was boxed. Now, with the check disabled, the Integer is cast to an int which fails when the value is null.

Thank you for your input! We will find another workaround.

[karlney]

Thanks for your input. I will close this ticket.