chore: add benchmarks for a small cloud instance (#1264)

chore: add benchmarks showing single-threded performanc In addition addressed some questions raised during HN discussions. Signed-off-by: Roman Gershman <[email protected]>
dragonflydb · Dec 4, 2023 · f39eac5 · f39eac5
1 parent 24b1343
commit f39eac5
Show file tree

Hide file tree

Showing 2 changed files with 81 additions and 10 deletions.
diff --git a/README.md b/README.md
@@ -28,9 +28,50 @@ Fully compatible with Redis and Memcached APIs, Dragonfly requires no code chang
 
 ## <a name="benchmarks"><a/>Benchmarks
 
+We first compare Dragonfly with Redis on `m5.large` instance which is commonly used to run Redis
+due to its single-threaded architecture. The benchmark program runs from another
+load-test instance (c5n) in the same AZ using `memtier_benchmark  -c 20 --test-time 100 -t 4 -d 256 --distinct-client-seed`
+
+Dragonfly shows a comparable performance:
+
+1. SETs (`--ratio 1:0`):
+
+|  Redis                                   |      DF                                |
+| -----------------------------------------|----------------------------------------|
+| QPS: 159K, P99.9: 1.16ms, P99: 0.82ms    | QPS:173K, P99.9: 1.26ms, P99: 0.9ms    |
+|                                          |                                        |
+
+2. GETs (`--ratio 0:1`):
+
+|  Redis                                  |      DF                                |
+| ----------------------------------------|----------------------------------------|
+| QPS: 194K, P99.9: 0.8ms, P99: 0.65ms    | QPS: 191K, P99.9: 0.95ms, P99: 0.8ms   |
+
+The benchmark above shows that the algorithmic layer inside DF that allows it to scale vertically
+does not take a large toll when running single-threaded.
+
+However, if we take a bit stronger instance (m5.xlarge), the gap between DF and Redis starts growing.
+(`memtier_benchmark  -c 20 --test-time 100 -t 6 -d 256 --distinct-client-seed`):
+1. SETs (`--ratio 1:0`):
+
+|  Redis                                  |      DF                                |
+| ----------------------------------------|----------------------------------------|
+| QPS: 190K, P99.9: 2.45ms, P99: 0.97ms   |  QPS: 279K , P99.9: 1.95ms, P99: 1.48  |
+
+2. GETs (`--ratio 0:1`):
+
+|  Redis                                  |      DF                                |
+| ----------------------------------------|----------------------------------------|
+| QPS: 220K, P99.9: 0.98ms , P99: 0.8ms   |  QPS: 305K, P99.9: 1.03ms, P99: 0.87ms |
+
+
+Dragonfly throughput capacity continues to grow with instance size,
+while single-threaded Redis is bottlenecked on CPU and reaches local maxima in terms of performance.
+
 <img src="http://static.dragonflydb.io/repo-assets/aws-throughput.svg" width="80%" border="0"/>
 
-In benchmarks, Dragonfly showed a 25X increase in throughput compared to Redis, crossing 3.8M QPS on c6gn.16xlarge.
+If we compare Dragonfly and Redis on the most network-capable instance c6gn.16xlarge,
+Dragonfly showed a 25X increase in throughput compared to Redis single process, crossing 3.8M QPS.
 
 Dragonfly's 99th percentile latency metrics at its peak throughput:
 

diff --git a/docs/faq.md b/docs/faq.md
@@ -1,20 +1,50 @@
 # Dragonfly Frequently Asked Questions
 
-- [What is the license model of Dragonfly? Is it an open source?](#what-is-the-license-model-of-dragonfly-is-it-an-open-source)
-- [Can I use dragonfly in production?](#can-i-use-dragonfly-in-production)
-- [Dragonfly provides vertical scale, but we can achieve similar throughput with X nodes in a Redis cluster.](#dragonfly-provides-vertical-scale-but-we-can-achieve-similar-throughput-with-x-nodes-in-a-redis-cluster.)
-- [What if I have some commands that are missing?](#what_if_i_have_some_commands_that_are_missing)
+- [Dragonfly Frequently Asked Questions](#dragonfly-frequently-asked-questions)
+  - [What is the license model of Dragonfly? Is it an open source?](#what-is-the-license-model-of-dragonfly-is-it-an-open-source)
+  - [Can I use dragonfly in production?](#can-i-use-dragonfly-in-production)
+  - [We benchmarked Dragonfly and we have not reached 4M qps throughput as you advertised.](#we-benchmarked-dragonfly-and-we-have-not-reached-4m-qps-throughput-as-you-advertised)
+  - [Dragonfly provides vertical scale, but we can achieve similar throughput with X nodes in a Redis cluster.](#dragonfly-provides-vertical-scale-but-we-can-achieve-similar-throughput-with-x-nodes-in-a-redis-cluster)
+  - [If only Dragonfly had this command I would use it for sure](#if-only-dragonfly-had-this-command-i-would-use-it-for-sure)
 
 
 ## What is the license model of Dragonfly? Is it an open source?
-Dragonfly is released under [BSL 1.1](../LICENSE.md) (Business Source License). We believe that a BSL license is more permissive than AGPL-like licenses. In general terms, it means that dragonfly's code is free to use and free to change as long as you do not sell services directly related to dragonfly or in-memory datastores.
-We followed the trend of other technological companies like Elastic, Redis, MongoDB, Cockroach labs, Redpanda Data to protect our rights to provide service and support for the software we are building.
+Dragonfly is released under [BSL 1.1](../LICENSE.md) (Business Source License).
+BSL is considered to be "source available" license and it's not strictly open-source license.
+We believe that a BSL license is more permissive than licenses like AGPL, and it will allow us to
+provide a competitive commercial service using our technology. In general terms,
+it means that Dragonfly's code is free to use and free to change as long as you do not sell services directly related to
+Dragonfly or in-memory datastores.
+We followed the trend of other technological companies like Elastic, Redis, MongoDB, Cockroach labs,
+Redpanda Data to protect our rights to provide service and support for the software we are building.
 
 ## Can I use dragonfly in production?
-License wise you are free to use dragonfly in your production as long as you do not provide dragonfly as a managed service. From a code maturity point of view, Dragonfly's code is covered with unit testing. However as with any new software there are use cases that are hard to test and predict. We advise you to run your own particular use case on dragonfly for a few days before considering production usage.
+License wise you are free to use dragonfly in your production as long as you do not provide Dragonfly as a managed service.
+From a code maturity point of view, Dragonfly's code is covered with unit testing and the regression tests.
+However as with any new software there are use cases that are hard to test and predict.
+We advise you to run your own particular use case on dragonfly for a few days before considering production usage.
+
+## We benchmarked Dragonfly and we have not reached 4M qps throughput as you advertised.
+We conducted our experiments using a load-test generator called `memtier_benchmark`,
+and we run benchmarks on AWS network-enhanced instance `c6gn.16xlarge` on recent Linux kernel versions.
+Dragonfly might reach smaller throughput on other instances, but we would
+still expect to reach around 1M+ qps on instances with 16-32 vCPUs.
 
 ## Dragonfly provides vertical scale, but we can achieve similar throughput with X nodes in a Redis cluster.
-Dragonfly utilizes the underlying hardware in an optimal way. Meaning it can run on small 8GB instances and scale vertically to large 768GB machines with 64 cores. This versatility allows to drastically reduce complexity of running cluster workloads to a single node saving hardware resources and costs. More importantly, it reduces the complexity (total cost of ownership) of handling the multi-node cluster. In addition, Redis cluster-mode imposes some limitations on multi-key and transactional operations while Dragonfly provides the same semantics as single node Redis.
+Dragonfly optimizes the use of underlying hardware, allowing it to run efficiently on instances as small as 8GB,
+ and scale vertically to large 2TB machines with 128 cores. This versatility significantly
+ reduces the complexity of running cluster workloads on a single node, saving hardware resources and costs.
+ More importantly, it diminishes the total cost
+ of ownership associated with managing multi-node clusters. In contrast, Redis in cluster
+ mode imposes limitations on multi-key and transactional operations, whereas Dragonfly maintains
+ the same semantics as a single-node Redis system.
+ Furthermore, scaling out horizontally with small instances can lead to instability
+ in production environments.
+ We believe that large-scale deployments of in-memory stores require both vertical and horizontal scaling,
+ which is not efficiently achievable with an in-memory store like Redis.
 
 ## If only Dragonfly had this command I would use it for sure
-Dragonfly implements ~130 Redis commands which we think represent a good coverage of the market. However this is not based empirical data. Having said that, if you have commands that are not covered, please feel free to open an issue for that or vote for an existing issue. We will do our best to prioritise those commands according to their popularity.
+Dragonfly implements ~190 Redis commands which we think represent a good coverage of the market.
+However this is not based empirical data. Having said that, if you have commands that are not covered,
+please feel free to open an issue for that or vote for an existing issue.
+We will do our best to prioritise those commands according to their popularity.