Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: unlock GVL for simple compression and decompression #84

Merged
merged 6 commits into from
Apr 25, 2024

Conversation

SpringMT
Copy link
Owner

@SpringMT SpringMT commented Apr 23, 2024

Related to #79
In #79, GVL unlocking is limited to streaming only, so this PR unlocks GVL for simple compression and decompression.

This program demonstrates the difference:
(in benckmarks)

Simple Compression

$LOAD_PATH.unshift '../lib'
require 'zstd-ruby'
require 'thread'

GUESSES = (ENV['GUESSES'] || 1000).to_i
THREADS = (ENV['THREADS'] || 1).to_i

p GUESSES: GUESSES, THREADS: THREADS

sample_file_name = ARGV[0]
json_string = File.read("./samples/#{sample_file_name}")

queue = Queue.new
GUESSES.times { queue << json_string }
THREADS.times { queue << nil }
THREADS.times.map {
  Thread.new {
    while str = queue.pop
      Zstd.compress(json_string)
    end
  }
}.each(&:join)

Without this patch:

[springmt@MacBook-Pro] (main)✗ % time THREADS=4 bundle exec ruby multi_thread_comporess.rb city.json
{:GUESSES=>1000, :THREADS=>4}
THREADS=4 bundle exec ruby multi_thread_comporess.rb city.json  2.65s user 0.15s system 93% cpu 3.003 total

With the patch:

[springmt@MacBook-Pro] (main)✗ % time THREADS=4 bundle exec ruby multi_thread_comporess.rb city.json
{:GUESSES=>1000, :THREADS=>4}
THREADS=4 bundle exec ruby multi_thread_comporess.rb city.json  2.78s user 0.14s system 250% cpu 1.162 total

Simple Decompression

$LOAD_PATH.unshift '../lib'
require 'zstd-ruby'
require 'thread'

GUESSES = (ENV['GUESSES'] || 1000).to_i
THREADS = (ENV['THREADS'] || 1).to_i

p GUESSES: GUESSES, THREADS: THREADS

sample_file_name = ARGV[0]
json_string = File.read("./samples/#{sample_file_name}")
target = Zstd.compress(json_string)

queue = Queue.new
GUESSES.times { queue << target }
THREADS.times { queue << nil }
THREADS.times.map {
  Thread.new {
    while str = queue.pop
      Zstd.decompress(str)
    end
  }
}.each(&:join)

Without this patch:

[springmt@MacBook-Pro] (main)✗ % time THREADS=4 bundle exec ruby multi_thread_decomporess.rb city.json
{:GUESSES=>1000, :THREADS=>4}
THREADS=4 bundle exec ruby multi_thread_decomporess.rb city.json  0.98s user 0.17s system 86% cpu 1.323 total

With the patch:

[springmt@MacBook-Pro] (main)✗ % time THREADS=4 bundle exec ruby multi_thread_decomporess.rb city.json
{:GUESSES=>1000, :THREADS=>4}
THREADS=4 bundle exec ruby multi_thread_decomporess.rb city.json  1.03s user 0.17s system 180% cpu 0.666 total

@SpringMT SpringMT force-pushed the feature/unlock-gvl-simple-compress-decompress branch from d4692ff to 40d9544 Compare April 23, 2024 01:53
@SpringMT SpringMT force-pushed the feature/unlock-gvl-simple-compress-decompress branch from 3be3d8d to a61c625 Compare April 24, 2024 02:13
@SpringMT SpringMT merged commit 3eedc15 into main Apr 25, 2024
7 checks passed
@SpringMT SpringMT deleted the feature/unlock-gvl-simple-compress-decompress branch April 25, 2024 01:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant