Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TLS Handshake Error: "wrong version number" with --tls #4526

Open
jahnavilatha589 opened this issue Jan 29, 2025 · 21 comments · May be fixed by #4539
Open

TLS Handshake Error: "wrong version number" with --tls #4526

jahnavilatha589 opened this issue Jan 29, 2025 · 21 comments · May be fixed by #4539
Assignees
Labels
bug Something isn't working

Comments

@jahnavilatha589
Copy link

I started Dragonfly with the flags --tls, --tls_key_file, and --tls_cert_file, but after the server starts, the following log message keeps appearing repeatedly.
W20250129 15:59:21.307173 1357310 dragonfly_connection.cc:696] Error handshaking error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Does DragonflyDB support specific TLS versions, and do I need to configure them explicitly?

@jahnavilatha589 jahnavilatha589 added the bug Something isn't working label Jan 29, 2025
@BagritsevichStepan
Copy link
Contributor

Hi @jahnavilatha589, thanks for reporting this. I will proceed with that and get back to you later

@BagritsevichStepan BagritsevichStepan self-assigned this Jan 29, 2025
@romange
Copy link
Collaborator

romange commented Jan 29, 2025

@jahnavilatha589 do you succeed to connect to the instance?

@jahnavilatha589
Copy link
Author

jahnavilatha589 commented Jan 29, 2025

@romange , Yes.able to connect to instance

@romange
Copy link
Collaborator

romange commented Jan 29, 2025

so maybe it's bots scanning your open ports, trying to connect? is your connection accessible from the internet?

@jahnavilatha589
Copy link
Author

@romange , No, Not accessible from the internet

@jahnavilatha589
Copy link
Author

Now it's working. Instance receiving requests from Monitoring tool. Thank you.

@BagritsevichStepan BagritsevichStepan removed their assignment Jan 30, 2025
@romange
Copy link
Collaborator

romange commented Jan 30, 2025

@BagritsevichStepan you are very quick :)
i think there i a lesson here that we can still learn from.
If I run dragonfly with tls mode but use plain protocol echo PING | nc localhost 6379, I will get a clear response from Dragonfly "-ERR Bad TLS header, double check if you enabled TLS for your client"

but the logs will contain:

W20250130 15:07:56.191462 86915 tls_engine.cc:243] SSL protocol error 0:167772427 wrong version number /home/roman/projects/dragonfly/helio/util/tls/tls_engine.cc : 136
I20250130 15:07:56.191491 86915 dragonfly_connection.cc:706] Error handshaking Software caused connection abort

which looks disturbing. We should tone down the logs or at least have better wording

@romange romange reopened this Jan 30, 2025
@BagritsevichStepan
Copy link
Contributor

It wasn't me who closed the ticket 😄
But okay, I will fix that.

@BagritsevichStepan BagritsevichStepan self-assigned this Jan 30, 2025
@jahnavilatha589
Copy link
Author

jahnavilatha589 commented Jan 30, 2025

Enabled TLS with flags --tls, --tls_key_file, --tls_cert_file, and tls_replication. Now replication stops working.

127.0.0.1:6379> REPLICAOF 10.12.34.53 6380
(error) ERR could not connect to master: Unknown error 337047686

LOGS in master :
W20250130 18:37:55.819727 3868529 dragonfly_connection.cc:688] Error handshaking Software caused connection abort
W20250130 18:37:56.320842 3868531 dragonfly_connection.cc:688] Error handshaking Software caused connection abort
W20250130 18:37:57.023748 3868532 dragonfly_connection.cc:688] Error handshaking Software caused connection abort
W20250130 18:37:57.068193 3868533 dragonfly_connection.cc:688] Error handshaking Software caused connection abort

Is --tls_ca_cert_file mandatory flag for replication.?

@romange
Copy link
Collaborator

romange commented Jan 30, 2025

@kostasrim can you help please?

@kostasrim
Copy link
Contributor

Enabled TLS with flags --tls, --tls_key_file, --tls_cert_file, and tls_replication. Now replication stops working.

127.0.0.1:6379> REPLICAOF 10.12.34.53 6380
(error) ERR could not connect to master: Unknown error 337047686

LOGS in master : W20250130 18:37:55.819727 3868529 dragonfly_connection.cc:688] Error handshaking Software caused connection abort W20250130 18:37:56.320842 3868531 dragonfly_connection.cc:688] Error handshaking Software caused connection abort W20250130 18:37:57.023748 3868532 dragonfly_connection.cc:688] Error handshaking Software caused connection abort W20250130 18:37:57.068193 3868533 dragonfly_connection.cc:688] Error handshaking Software caused connection abort

Is --tls_ca_cert_file mandatory flag for replication.?

how did you configure master and replica ? Plz place the exact commands you used to run both master and replica

@jahnavilatha589
Copy link
Author

Master : dragonfly.conf

--tiered_prefix=storage/flash/flash
--aclfile=aclfile.txt
--dir=storage/data
--version_check=false
--hz=1
--snapshot_cron=/30 * * *
--masteruser=repl
--masterauth=hypermemreplpwd
--proactor_threads=8

start command : ./dragonfly --flagfile=dragonfly.conf --tls --tls_key_file=/certs/hypermem.key --tls_cert_file=/certs/hypermem.pem --requirepass=test --port=6380 --tls_replication

Replica : dragonfly.conf

--tiered_prefix=storage/flash/flash
--aclfile=aclfile.txt
--dir=storage/data
--version_check=false
--hz=1
--snapshot_cron=/30 * * *
--masteruser=repl
--masterauth=hypermemreplpwd
--proactor_threads=8
--replicaof=10.12.32.78:6380

start command : ./dragonfly --flagfile=dragonfly.conf --tls --tls_key_file=/certs/hypermem.key --tls_cert_file=/certs/hypermem.pem --requirepass=test --port=6380 --tls_replication

@romange
Copy link
Collaborator

romange commented Jan 30, 2025

if --requirepass=test than your --masterauth= on replica should also be test.

@kostasrim
Copy link
Contributor

^^ this

@kostasrim
Copy link
Contributor

kostasrim commented Jan 30, 2025

@romange I wanted to add a configuration test (we only got replication with tls and ca_cert) and it failed. Basically, replication with tls enabled and without ca_cert is broken. Generally you want some form of authorization, which is offered by requirepass or by the certified authority. Should be a simple fix + I will add some tests such that we don't break it again.

@jahnavilatha589 for now plz use ca_cert and masterauth flag until the former is addressed.

@kostasrim kostasrim linked a pull request Jan 30, 2025 that will close this issue
@jahnavilatha589
Copy link
Author

@kostasrim , Any ETA for this fix?

@kostasrim
Copy link
Contributor

@jahnavilatha589 it's fixed already, see the linked PR. It needs merge + release (or you can get our weekly build once it's merged)

@jahnavilatha589
Copy link
Author

@kostasrim

dragonfly.conf

--tiered_prefix=storage/flash/flash
--aclfile=aclfile.txt
--dir=storage/data
--version_check=false
--hz=1
--snapshot_cron=/30 * * *
--masteruser=repl
--masterauth=hypermemreplpwd
--proactor_threads=8

start command : ./dragonfly --flagfile=dragonfly.conf --tls --tls_key_file=/certs/hypermem.key --tls_cert_file=/certs/hypermem.pem --tls_ca_cert_file=/certs/hypermem-trust.pem --requirepass=test --port=6380 --tls_replication

redis-cli --insecure --tls -> Not working
ERR : Error handshaking error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate

If i start dragonfly without tls_ca_cert_file then redis-cli --insecure --tls is working

Could you please check on this?

@kostasrim
Copy link
Contributor

Hi @jahnavilatha589 you are misusing the flags.

Why would this work ?

redis-cli --insecure --tls -> Not working
ERR : Error handshaking error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate

You are configuring dragonfly to verify client certificates with the ca_file specified in ca_cert_file and yet you are using a client without any certificates! --insecure disables the check of the server side (dragonfly) certificate on the client side (redis-cli in your case). In other words, --insecure does not effect the server but the client!

If i start dragonfly without tls_ca_cert_file then redis-cli --insecure --tls is working

Of course it's working, because now dragonfly is configured to not verify peer certificates which you do not supply to your client anyway.

All in all,

  1. Server (df) must authorize client connections somehow. Either via a password, a ca authority or both!
  2. Client (redis-cli) can authorize server connections via --cacert or not via --insecure.

@jahnavilatha589
Copy link
Author

Thank you @kostasrim . Now it's clear.

I used a CA cert to enable tls_replication, but I don't want to authorize client connections using the CA cert. Is there a way to disable client connection authorize on the server side (in DragonflyDB)?

@kostasrim
Copy link
Contributor

kostasrim commented Feb 3, 2025

a CA cert to enable tls_replication, but I don't wa

yes, once my PR is merged you can pull a docker weekly/daily build. Until the fix gets merged you do need ca_cert for now :(

client connection authorize on the server side (in DragonflyDB)?

You need some form of authorization, either with password or ca or both.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants