Skip to content

Commit

Permalink
Use CRC32 for the replication protocol
Browse files Browse the repository at this point in the history
It's about 7 times faster than SHA1, and has similar or better
characteristics for identifying file changes.
  • Loading branch information
carlhoerberg committed Jan 15, 2025
1 parent 4e0a719 commit 3ec1816
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 14 deletions.
3 changes: 2 additions & 1 deletion src/lavinmq/clustering.cr
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ require "digest/sha1"

module LavinMQ
module Clustering
Start = Bytes['R'.ord, 'E'.ord, 'P'.ord, 'L'.ord, 'I'.ord, 1, 0, 0]
Start100 = Bytes['R'.ord, 'E'.ord, 'P'.ord, 'L'.ord, 'I'.ord, 1, 0, 0]
Start = Bytes['R'.ord, 'E'.ord, 'P'.ord, 'L'.ord, 'I'.ord, 1, 0, 1]

class Error < Exception; end

Expand Down
12 changes: 6 additions & 6 deletions src/lavinmq/clustering/client.cr
Original file line number Diff line number Diff line change
Expand Up @@ -125,9 +125,9 @@ module LavinMQ

private def sync_files(socket, lz4)
Log.info { "Waiting for list of files" }
sha1 = Digest::SHA1.new
remote_hash = Bytes.new(sha1.digest_size)
local_hash = Bytes.new(sha1.digest_size)
crc32 = Digest::CRC32.new
remote_hash = Bytes.new(crc32.digest_size)
local_hash = Bytes.new(crc32.digest_size)
files_to_delete = ls_r(@data_dir)
missing_files = Array(String).new
loop do
Expand All @@ -139,9 +139,9 @@ module LavinMQ
path = File.join(@data_dir, filename)
files_to_delete.delete(path)
if File.exists? path
sha1.file(path)
sha1.final(local_hash)
sha1.reset
crc32.file(path)
crc32.final(local_hash)
crc32.reset
if local_hash != remote_hash
Log.info { "Mismatching hash: #{path}" }
move_to_backup path
Expand Down
14 changes: 7 additions & 7 deletions src/lavinmq/clustering/server.cr
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ module LavinMQ
# When a follower connects:
# It sends a static header (wrong header disconnects the client)
# It sends its password (servers closes the connection if the password is wrong)
# Server sends a list of files in its data directory and the sha1 hash of those files
# Server sends a list of files in its data directory and the crc32 hash of those files
# Client requests files that is missing or has mismatching checksums of
# In the meantime the server queues up changes (all publishes/consumes are paused)
# When client doesn't request more files starts to stream changes
Expand Down Expand Up @@ -72,12 +72,12 @@ module LavinMQ
end

def files_with_hash(& : Tuple(String, Bytes) -> Nil)
sha1 = Digest::SHA1.new
hash = Bytes.new(sha1.digest_size)
@files.each_key do |path|
sha1.file path
sha1.final hash
sha1.reset
crc32 = Digest::CRC32.new
hash = Bytes.new(crc32.digest_size)
@files.each do |path, mfile|
crc32.file path
crc32.final hash
crc32.reset
yield({path, hash})
end
end
Expand Down

0 comments on commit 3ec1816

Please sign in to comment.