Uploading failed, retrying. {:exception=>Seahorse::Client::NetworkingError, :message=>"Net::ReadTimeout"

[christianbeland]

I am fetching about 60mb of data from jdbc input plugin (mysql database) and trying to upload it to my minio server. I only get part of it.

For files that don't get uploaded, I receive the following error:

logstash_1  | [2020-04-23T19:24:28,978][ERROR][logstash.outputs.s3      ][main] Uploading failed, retrying. {:exception=>Seahorse::Client::NetworkingError, :message=>"Net::ReadTimeout", :path=>"/tmp/logstash/dc0fd933-fc1b-43bd-b081-9b1899bc9cd9/events/ls.s3.abd26f2c-a1d6-4588-a387-8e551bfdd9bb.2020-04-23T18.52.part2.txt", :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:181:in `rbuf_fill'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:157:in `readuntil'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:167:in `readline'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/response.rb:40:in `read_status_line'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/response.rb:29:in `read_new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/patches.rb:29:in `block in new_transport_request'", "org/jruby/RubyKernel.java:1193:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/patches.rb:26:in `new_transport_request'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1474:in `request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/connection_pool.rb:330:in `request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/handler.rb:72:in `block in transmit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/handler.rb:121:in `block in session'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/connection_pool.rb:96:in `session_for'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/handler.rb:119:in `session'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/handler.rb:71:in `transmit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/handler.rb:45:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/plugins/content_length.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_request_signer.rb:88:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_request_signer.rb:23:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_host_id.rb:14:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_request_signer.rb:65:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_redirects.rb:15:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:108:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_dualstack.rb:32:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_accelerate.rb:49:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_md5s.rb:31:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_iad_regional_endpoint.rb:31:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/rest/handler.rb:7:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/plugins/endpoint.rb:41:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/plugins/response_target.rb:21:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/request.rb:70:in `send_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/base.rb:207:in `block in define_operation_methods'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.451/lib/aws-sdk-resources/services/s3/file_uploader.rb:42:in `block in put_object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.451/lib/aws-sdk-resources/services/s3/file_uploader.rb:49:in `open_file'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.451/lib/aws-sdk-resources/services/s3/file_uploader.rb:41:in `put_object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.451/lib/aws-sdk-resources/services/s3/file_uploader.rb:34:in `upload'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.451/lib/aws-sdk-resources/services/s3/object.rb:252:in `upload_file'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.2.0/lib/logstash/outputs/s3/uploader.rb:38:in `upload'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.2.0/lib/logstash/outputs/s3/uploader.rb:29:in `block in upload_async'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in `run'"]}`

That is only if I use 200kb size_file. If I try to send bigger file, it usually ends up uploading no file to minio s3.

If I go on the logstash docker container filesystem, I see part###.txt files that were not sent in the /tmp/logstash folder.

• Version: logstash-output-s3 4.2.0 and 4.3.1, logstash 7.6.2
• Operating System: ubuntu 18 / docker
• Config File (if you have sensitive info, please remove it):

s3{
     endpoint => "${S3_HOST}"
     access_key_id => "${S3_ACCESS_KEY}"
     secret_access_key => "${S3_SECRET_KEY}"
     bucket => "twitter-monitoring"
     size_file => 204800
     time_file => 1
     codec => csv{
        columns => [ "uid", "id", "accessNumber", "authorHandle", "language", "excerpt", "location", "normalizedLocation"]
        charset => "UTF-8"
     }
     validate_credentials_on_root_bucket => false
     prefix => "events"
     additional_settings => {
         force_path_style => true
         follow_redirects => false
     }
   }