Failing to register S3 repository in ElasticSearch

[ikoniaris]

Hello, I'm testing the 2.4.1 version of the plugin with ElasticSearch 1.4.4 and Java 1.7.0_15.

When I try to add a new repository I get the following error back:

{
"error": "RepositoryException[[s3_repo] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.NoSuchMethodError: org.apache.http.conn.scheme.Scheme.(Ljava/lang/String;ILorg/apache/http/conn/scheme/SchemeSocketFactory;)V\n at org.elasticsearch.repositories.s3.S3Repository.(Unknown Source)\n while locating org.elasticsearch.repositories.s3.S3Repository\n while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: NoSuchMethodError[org.apache.http.conn.scheme.Scheme.(Ljava/lang/String;ILorg/apache/http/conn/scheme/SchemeSocketFactory;)V]; ",
"status": 500
}

Some Googling returns hits related to httpclient. I see in /usr/share/elasticsearch/plugins/cloud-aws that I have httpclient-4.2.jar.

[dadoonet]

What is your dependency tree ?

[ikoniaris]

@dadoonet I'm not Java proficient, sorry. Can you give an example on how to find this info? Just to clarify, I haven't written any code, I'm trying to take snapshosts from a web operations point of view. I've setup a bucket, brought up some instances with a proper IAM role, installed ElasticSearch and the plugin. Now when I try to curl -XPUT http://localhost:9200/_snapshot/s3_repo' -d '{ "type": "s3", "settings": { "bucket": "my-bucket", "region": "my-region" } }' I get the above error.

[dadoonet]

Sorry. I misunderstood your issue.

How do you create the repository? What does your elasticsearch.yml settings look like?

[ikoniaris]

HI @dadoonet, I have my bucket already created in S3. My ElasticSearch cluster nodes already have an instance profile with S3 permissions, I didn't add anything else in elasticsearch.yml. So, I'm trying to add the repository via the _snapshot API using curl (see my previous comment above for the command), and I'm getting back the error I've pasted in the original post.

[dadoonet]

Can you post your full stack trace or link to your full logs?

[ikoniaris]

Hello @dadoonet, here is what I see on my logs when I try to add the s3 repo:

[2015-05-21 12:19:32,103] [WARN] [my-machine] failed to create repository [s3_repo]
org.elasticsearch.repositories.RepositoryException: [s3_repo] failed to create repository
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:405)
        at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:356)
        at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:55)
        at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:110)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:352)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:184)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:154)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, java.lang.NoSuchMethodError: org.apache.http.conn.scheme.Scheme.<init>(Ljava/lang/String;ILorg/apache/http/conn/scheme/SchemeSocketFactory;)V
  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.s3.S3Repository
  while locating org.elasticsearch.repositories.Repository

1 error
        at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:344)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:131)
        at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:69)
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:395)
        ... 9 more
Caused by: java.lang.NoSuchMethodError: org.apache.http.conn.scheme.Scheme.<init>(Ljava/lang/String;ILorg/apache/http/conn/scheme/SchemeSocketFactory;)V
        at org.apache.http.impl.conn.SchemeRegistryFactory.createDefault(SchemeRegistryFactory.java:47)
        at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:93)
        at com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:26)
        at com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:97)
        at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:158)
        at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119)
        at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:405)
        at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:387)
        at org.elasticsearch.cloud.aws.InternalAwsS3Service.getClient(InternalAwsS3Service.java:137)
        at org.elasticsearch.cloud.aws.InternalAwsS3Service.client(InternalAwsS3Service.java:85)
        at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:130)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        ... 13 more

[danielpops]

@ikoniaris, FYI I had the same issue, and this ended up being caused by pollution of the classpath with some older version of the HttpCore library coming from a custom java module in the cluster used for some custom logging solution.

To come to this conclusion, I observed that the cloud-aws plugin could be installed and function correctly in a vanilla elasticsearch docker container, and then I compared the /usr/share/elasticsearch/lib folder (and any other paths included in the classpath) and found a jar that was present in the broken setup. Disassembling the jar file, I could confirm that there was a dependency on old HttpCore code.

I spent a bunch of time debugging this, so I sincerely hope this helps others out there who might run in to a similar issue!